---
title: Tokenization
slug: tokenization
qid: 
snapshot: enwiki-001304802084
content_hash: 
license: CC BY-SA 4.0
source: https://en.wikipedia.org/w/index.php?oldid=1304802084
tags: [23118434, 3564997, 3566086, 4410601]
---

# Tokenization

> [!note] Provenance
> Snapshot `enwiki-001304802084` · content root `…`
> Derived from <https://en.wikipedia.org/w/index.php?oldid=1304802084> · Licensed CC BY-SA 4.0

Tokenization may refer to: * Tokenization (lexical analysis) in language processing * Tokenization in large language models * Tokenization in search engine indexing * Tokenization (data security) * Asset tokenization in finance * Word segmentation * A procedure during the Transformer architecture

Tokenization may refer to: Tokenization (lexical analysis) in language processing Tokenization in large language models Tokenization in search engine indexing Tokenization (data security) Asset tokenization in finance Word segmentation A procedure during the Transformer architecture
