Tokenization in Simple Terms
Tokenization breaks text into units called tokens. A tokenizer for a language model also maps those units to numerical token IDs. Depending on its design, a token might represent a word, part of a word, punctuation, whitespace, or bytes that make up text.
Imagine receiving a sentence as a row of labeled building blocks. Tokenization decides how to divide that row and label each piece. The model then processes numerical representations of those pieces. Dividing text does not, by itself, interpret its meaning.
This lesson explains the difference between tokens, IDs, and embeddings; the main ways tokenizers split text; and why token counts matter. For the broader picture, see What Is Natural Language Processing (NLP)?
From Text to Tokens, IDs, and Embeddings
A tokenizer uses a vocabulary: a mapping between supported tokens and their IDs. An ID acts like an entry number. Two nearby ID numbers do not imply that their tokens have similar meanings, and a larger ID does not mean a more important token.
- 1. Text
AI helps. - 2. Tokenizer
Illustrative pieces:
"AI",[space],"helps","." - 3. Token IDs
Toy vocabulary entries:
17, 2, 41, 9 - 4. Embeddings
Look up a learned vector for each ID
- 5. Model
Process the sequence in context, then produce task output
The four pieces and IDs are invented for teaching. The label [space] represents one literal space. Joining the pieces, including the space and period, restores AI helps. exactly.
In a typical neural language model, an embedding layer looks up a learned vector, or list of numbers, for each ID. Later model layers combine information across the sequence to build context-sensitive representations. Position information also helps distinguish different word orders.
For example, the surrounding words help a model distinguish a river bank from a bank account. That interpretation happens downstream of tokenization. PyTorch’s embedding documentation describes the ID-to-vector lookup used in this kind of pipeline.
Word, Character, and Subword Tokens
Word-level tokenization uses word boundaries, often with additional rules for punctuation. A character-level approach uses smaller units, though “character” needs a precise definition when Unicode is involved. Subword approaches use reusable pieces learned from data; these pieces need not be dictionary words, syllables, or meaningful word roots.
Exact input: unbelievable
unbelievable
Keep this word together.
u · n · b · e · l · i · e · v · a · b · l · e
Use each letter separately.
un · believ · able
Use reusable pieces.
Dots show boundaries and are not part of the tokens. Joining each row without the dots or display spaces gives unbelievable. These are toy splits and counts, not the output of a named tokenizer.
A word-level vocabulary must account for many spellings and inflected forms. Smaller units can cover more combinations with fewer vocabulary entries, but may create longer sequences. Subword methods balance these concerns; the best balance depends on the data and model.
Sentence segmentation operates at another level: it divides a document into sentences. It can help prepare documents for analysis, but a sentence usually still needs word, subword, or other model tokens. A sentence boundary and a model-token boundary serve different purposes.
BPE, WordPiece, and Unigram
Three common subword methods use different rules to build a vocabulary or choose a split. Learning a tokenizer’s vocabulary is a preparation step; applying an already trained tokenizer to a new prompt is encoding.
Learn frequent adjacent-pair merges; apply the learned merge order when encoding.
Encode with the longest matching vocabulary piece from each position, using continuation rules.
Learn probabilities for candidate pieces; choose a high-probability segmentation.
The method, vocabulary, and preprocessing jointly determine the result. No method is best for every language or task.
Byte-pair encoding (BPE) grows a vocabulary by repeatedly merging frequent adjacent units in training text. Implementations may begin with characters or bytes. The Hugging Face BPE lesson walks through the merge process.
WordPiece, used with BERT-family models, commonly encodes by greedy longest-match lookup. Continuation markers such as ## are tokenizer notation, not literal characters to insert into the original word. See the WordPiece walkthrough.
Unigram starts with many candidate pieces, learns their probabilities, and prunes the vocabulary. It can select the most probable segmentation or sample alternatives when configured to do so. The Unigram lesson explains this distinction.
SentencePiece is a tokenizer toolkit that supports BPE and Unigram and can train from raw text without a separate word-splitting step. Its name does not mean “sentence segmentation.” Its official repository explains whitespace representation and model configuration.
Vocabulary and Unknown Tokens
A tokenizer’s vocabulary is finite. Some tokenizers use an unknown token, often written [UNK] or <unk>, when they cannot represent an input. This loses distinctions between inputs mapped to that same unknown entry.
Subword units reduce the need for a separate entry for every complete word. Byte-level designs or byte fallback can represent unfamiliar text through smaller byte units instead of replacing it with an unknown token. Coverage still does not guarantee that the model will handle that text well.
OpenAI’s tiktoken documentation describes a reversible, byte-based BPE implementation. Other tokenizer configurations may normalize text or discard information, so reversibility should be checked rather than assumed.
Vocabulary size also affects model storage. At the same vector width, an embedding table with more vocabulary entries contains more parameters. A larger vocabulary may shorten some inputs, but it does not automatically give a model better language ability.
Normalization, Spaces, and Special Tokens
There is no universal “tokenize first, clean afterward” pipeline. Normalization may happen before segmentation or inside the tokenizer. Depending on its configuration, it can lowercase text, standardize Unicode forms, or change whitespace. These choices can affect what survives decoding.
For example, a tokenizer that explicitly lowercases AI to ai has changed the input. Do not silently change capitalization in an example and present it as an exact reconstruction. The Hugging Face tokenizer pipeline separates normalization, pre-tokenization, model encoding, and post-processing.
Spaces and punctuation can be separate tokens, attached to nearby pieces, or represented with visible marker symbols. Models may also require special tokens for boundaries, padding, or message structure. A displayed list of ordinary words may therefore omit part of the actual model input.
Use the tokenizer and formatting expected by the chosen model. Avoid adding stemming, lemmatization, or punctuation removal simply because they appear in a general NLP checklist. SentencePiece’s normalization guide shows why configuration matters.
Tokenization Across Languages
Splitting at spaces is insufficient for many languages. Chinese, Japanese, and Thai, for example, do not consistently mark word boundaries with spaces. Languages with extensive inflection or long compounds can produce many related word forms.
Consider 你好, café, and text mixing English with another script. A tokenizer may split them at different boundaries, and visually similar Unicode text can have different underlying character sequences. Emojis may also contain several code points. One visible symbol is not necessarily one token.
These are reasons to inspect real inputs rather than rely on English word-count estimates. Test the languages, spellings, symbols, and code your application actually receives. Comparing token counts can reveal uneven segmentation, but it cannot establish translation quality, fairness, or overall task accuracy on its own.
Token Counts and Context Windows
There is no fixed number of tokens per word. Counts depend on the tokenizer, spelling, case, punctuation, whitespace, language, and message format. Use the exact model’s tokenizer or supported counting tool for a reliable estimate; a toy split cannot predict a ChatGPT request’s token usage.
A context window limits how much a model can process in a request. For a generative system, the budget may need room for instructions, conversation history, documents, tool results, and the response. Check the model and application’s input, output, and combined limits.
If a request is too long, an application may reject it, truncate it, summarize material, or retrieve selected passages. These are application and configuration choices. A different tokenizer can change how much text fits within a token budget; it does not change the model’s configured window size.
Stored conversation history or application-managed memory is separate from what is included in the current request. OpenAI’s conversation-state guide explains context management, while Hugging Face documents configurable truncation.
How Tokenization Can Affect Model Performance
Tokenization influences sequence length, vocabulary coverage, and the numerical inputs a model receives. Longer sequences can require more processing and memory, but the effect depends on the architecture, hardware, and workload.
For a service billed per token, token counts matter to price. Fewer tokens do not universally mean a cheaper or better result: model rates, output length, caching, and other service rules also matter. Do not swap tokenizers on a pretrained model just to obtain a smaller count; its learned vocabulary mapping must remain compatible.
When comparing designs, measure task quality as well as token count, latency, and memory on representative data. Include unfamiliar names, multilingual text, and long inputs. A tokenizer that compresses one benchmark well may behave differently on your application’s text.
Where Tokenization Is Used
Tokenization supplies units for many NLP workflows. The downstream system determines what those units are used to predict or generate.
| Task | What follows tokenization |
|---|---|
| Text classification | A model predicts document or message labels. |
| Entity extraction | A system identifies labeled spans in the original text. |
| Translation or chat | A model generates output tokens that are decoded into text. |
| Search | A system builds searchable terms or numerical representations. |
For extraction, retain alignment between tokens and the original text so that results point to the correct spans. Token boundaries can split a name into several pieces; the application must still return the intended complete mention.
Common Questions and Beginner Mistakes
Is tokenization the same as understanding?
No. Tokenization prepares a representation. Contextual interpretation and task predictions depend on later model processing. Even a correctly tokenized sentence can lead to an incorrect answer.
Does every word have one permanent token ID?
No. A word can be several tokens, and different tokenizers use different vocabularies and IDs. With the same fixed tokenizer and settings, repeated encoding is generally consistent; sampling-enabled tokenization is an exception.
Is tokenization the same as stemming or lemmatization?
No. Stemming reduces words using rules; lemmatization aims to find a dictionary base form. Tokenization identifies input units. Those transformations serve different purposes and are not mandatory additions to a pretrained transformer pipeline.
How can I check a tokenizer example?
Record the exact input, tokenizer name, version, and settings. Inspect the pieces, IDs, count, special tokens, and decoded result. Check whether normalization changed the text. For a reversible toy example, join the pieces and verify every character, including spaces, case, and punctuation.
Continue to Word Embeddings
The key distinction is simple: tokens divide text, IDs identify vocabulary entries, and embeddings provide learned numerical vectors for model processing. Tokenizer choices affect representation and resource use, so examples and performance claims need clear conditions.
Continue to Word Embeddings Explained to see how token IDs become vectors used by neural language models.