Tokenization

The model never sees letters

Ask a chatbot how many r’s are in “strawberry” and, for a long stretch of 2024, many of the best models in the world said two. The same models could write working code and pass law exams. The explanation is not a lack of intelligence. It is that the model never saw the letters at all.

What is a token?

A language model is a function from numbers to numbers. Before any text reaches it, a separate program called the tokenizer cuts the text into pieces from a fixed list, its vocabulary, and replaces each piece with that piece’s position in the list. Those integers are the only thing the network ever reads, and the only thing it ever writes: a model generates text by predicting the next token id, which the tokenizer turns back into characters.

Pieces are usually neither letters nor words but something in between. In GPT-4’s tokenizer, “ strawberry” with a leading space is a single token, while “strawberry” at the start of a line is three: str, aw, berry. Nowhere in either version does a token consist of a lone “r”.

How does it work in practice?

Tokenizers are trained once, before the model, on a large sample of text, and then frozen. The standard recipe, byte-pair encoding, simply notices which fragments of text are common and gives each one its own entry. The result is a vocabulary where frequent words (“ the”, “ and”, “ Alice”) are single tokens and rare words are assembled from pieces. For English prose, OpenAI’s rule of thumb is that one token is about four characters, or roughly three quarters of a word.

The demo below runs a real tokenizer trained in your browser on the first chapters of Alice in Wonderland, with 800 learned merges. It is much smaller than a production tokenizer, so it splits more, but it works exactly the same way.

Words the tokenizer saw often during training come out whole. Unusual words, names it never met, numbers and other scripts break into many small pieces. The row of integers at the bottom is, literally, the model’s entire view of your text.

Why does it matter?

Because tokens are the unit of almost everything you pay for or run up against when using a model:

• Price. APIs charge per million input and output tokens. The same request in a language the tokenizer handles badly can cost several times more.
• Context. A “200K context window” means 200,000 tokens, not words or characters. How much of a book fits depends on the tokenizer.
• Speed. Models generate one token per step. More tokens for the same answer means a slower answer.
• Failure modes. Spelling, letter counting, reversing strings, some arithmetic, and a handful of genuinely bizarre bugs all trace back to how text was cut up.

Tokenization is the least glamorous part of a language model and one of the most consequential. This lesson builds one from scratch, then uses it to explain those effects.

Key takeaways

• A tokenizer turns text into a sequence of integer ids from a fixed vocabulary; the model only ever sees and produces those ids.
• Common strings become single tokens and rare ones split into pieces; English averages roughly four characters per token.
• Prices, context limits and generation speed are all measured in tokens, and several well-known model failures come from tokenization.

Characters, words or pieces

There are three obvious ways to cut text into units, and two of them fail in instructive ways. Understanding why is the fastest route to understanding why every modern model uses the third.

What are the options?

How do the trade-offs play out?

Two costs pull in opposite directions. The first is sequence length. A transformer’s attention compares every token with every earlier token, so the work grows with the square of the length. Doubling the number of tokens for the same text roughly quadruples the attention cost and halves how much text fits in the context window. Characters are the worst case: about four times as many tokens as a subword tokenizer for English.

The second is vocabulary size. Every token needs a learned embedding vector at the input and a row in the output layer. With a vocabulary of 200,000 tokens and a model width of 4,096, the input embedding table alone is

Parameters in the input embedding table. The output layer often has another table the same size.

And each token must appear often enough in training for its vector to be learned well. A word-level vocabulary large enough to cover real text would be enormous and full of rarely seen entries. Worse, it still could not cover everything: new words, product names, usernames and misspellings would all map to a single [UNK] (unknown) token, and the model would be blind to them.

On familiar text the word vocabulary looks great: few tokens, all meaningful. The moment the text drifts from the training data, it breaks, and a single typo destroys a word entirely. Characters never break but always cost the most. Subwords degrade gracefully: an unfamiliar word costs a few extra tokens instead of being lost.

Why does it matter?

The subword compromise is why a model can read a brand-new product name, a misspelled search query or a line of code it has never seen. It also sets the vocabulary sizes you see in practice. GPT-2 used about 50,000 tokens. GPT-4’s tokenizer has about 100,000, GPT-4o’s about 200,000 according to OpenAI’s open-source tiktoken definitions, and Llama 3 uses 128,000, adding 28,000 tokens specifically for non-English languages. The trend is towards bigger vocabularies: as models grow, the embedding table becomes a small share of the parameters, and shorter sequences save compute on every single request.

Key takeaways

• Characters never fail but make sequences long; words keep sequences short but turn anything unfamiliar into [UNK].
• Subword tokenizers keep frequent words whole and build rare ones from pieces, so nothing is unknown and sequences stay short.
• Vocabulary size trades embedding parameters against sequence length; modern models use roughly 100,000 to 200,000 tokens.

Byte-pair encoding, step by step

The algorithm behind almost every modern tokenizer was invented in 1994 to compress files. It fits in five lines, and you can run it by hand.

What is byte-pair encoding?

Philip Gage described byte-pair encoding (BPE) as a compression scheme: find the most common pair of adjacent bytes in a file, replace every occurrence with a single unused byte, record the replacement, and repeat. Twenty-two years later, Sennrich, Haddow and Birch adapted it to build vocabularies for neural translation. Instead of compressing a file, they used the list of merges itself as the vocabulary: every merge creates a new token.

How does it work?

Start with every word split into single characters, and count how often each word occurs. Then repeat:

1. Count every pair of adjacent symbols across the corpus, weighted by word frequency.
2. Take the most frequent pair, say u + g, and add the joined symbol ug to the vocabulary.
3. Replace every adjacent u g in the corpus with ug.
4. Record the merge in an ordered list, and go back to step 1.

Stop when the vocabulary reaches the size you want. The final tokenizer is just two things: the base alphabet and the ordered list of merges. Here is a tiny corpus of five words (a classic teaching example from Hugging Face’s course) and how often each appears.

Notice how the counts change after each merge: once ug exists, the old pairs u + g and h + u vanish and new ones like h + ug appear. Merges build on earlier merges, which is how BPE assembles long tokens from short ones. A word like “bugs” that never appeared in the corpus still encodes cleanly as b + ug + s, because its fragments did.

Encoding new text

To tokenize a new string, split it into characters and apply the learned merges in the order they were learned: at each step, find the adjacent pair with the earliest merge in the list and join it, until no learned merge applies. Replaying the merges in order reproduces exactly the segmentation training would have produced. The merge list is not a dictionary of words; it is a small program.

Production tokenizers add one step first: pre-tokenization. A regular expression splits text into word-like chunks, and merges never cross chunk boundaries. GPT-2’s version keeps a leading space attached to each word, so ·the and The are different tokens, and stops letters, digits and punctuation from merging with each other (Radford et al., 2019).

Why does it matter?

Because BPE is driven purely by frequency, the vocabulary is a mirror of the training corpus. Whatever was common there becomes cheap to represent: English words if the corpus was mostly English, Python keywords and indentation if it contained code, and, as you will see, odd strings like Reddit usernames if those happened to repeat. That single fact explains the language costs and the glitch tokens later in this lesson.

Key takeaways

• BPE starts from single characters and repeatedly merges the most frequent adjacent pair, recording each merge in order.
• Encoding new text replays the merges in the order learned; the ordered merge list is the tokenizer.
• Because it is frequency-driven, the vocabulary reflects the training corpus: common strings become single tokens.

Bytes and Unicode

A character-level starting point still has an unknown-word problem: there are about 150,000 Unicode characters, and your training corpus will not contain all of them. The fix used by most large models is to start from bytes instead.

What is a byte-level tokenizer?

Computers store text as bytes using an encoding, almost always UTF-8. UTF-8 is variable length: the 128 ASCII characters (English letters, digits, common punctuation) take one byte each; accented Latin, Greek, Cyrillic, Hebrew and Arabic letters take two; most other scripts, including Chinese, Japanese and Hindi, take three; emoji take four.

A byte-level BPE tokenizer uses the 256 possible byte values as its base alphabet and learns merges on top of them. Since every possible string is some sequence of bytes, nothing can ever be unknown. GPT-2 introduced this design with 256 bytes, 50,000 merges and one special end-of-text token, for a vocabulary of 50,257 (Radford et al., 2019). The GPT-4, GPT-4o and Llama 3 tokenizers all follow the same pattern at larger sizes.

English is almost free: one byte per character. Hindi and Japanese pay three bytes per character before a single merge is learned, and some things that look like one character, like a waving hand with a skin tone, are several code points and eight bytes. A byte-level tokenizer only makes these cheap if its training data contained enough of them to earn merges.

How is a token decoded if it can hold half a character?

Tokens are byte strings, and the boundary between two tokens can fall in the middle of a multi-byte character. That is harmless while decoding a whole sequence, because the bytes are concatenated before being turned back into text. It becomes visible when you look at tokens one at a time: tools show a replacement symbol (�) or raw byte values for fragments that are not valid UTF-8 on their own. It also means a model streaming its answer token by token sometimes has to wait for the next token before it can display a character.

Why does it matter?

Byte-level BPE solved the unknown-token problem completely, which is why it became the default. But it moved the unfairness somewhere quieter: text in scripts that were rare in the tokenizer’s training data is never unknown, just expensive, because it stays as long runs of short byte-level tokens. That cost shows up in your bill and in how much fits in context, as the section on languages will show.

Key takeaways

• UTF-8 uses one byte for ASCII, two for many alphabets, three for most Asian scripts and four for emoji.
• Byte-level BPE starts from all 256 byte values, so any string can be tokenized and nothing is ever unknown.
• Unknown becomes expensive instead: text the tokenizer rarely saw stays as many short byte tokens.

Why models miscount letters

Some tasks that are trivial for a person reading letters are awkward for a model reading tokens. Once you know how the text was cut up, the pattern of failures stops looking random.

What goes wrong with spelling?

To count the r’s in “strawberry”, you look at the letters. The model receives a few ids. In GPT-4’s tokenizer the word is str, aw, berry at the start of a line, or the single token ·strawberry mid-sentence. The letters are not in the input at all. The model can still answer correctly, but only if it has learned from training data which letters each of those ids contains, and then counted across them. That is a memorised fact about each token, not something it can look at, and memorised facts are exactly where models are least reliable.

The same applies to reversing a word, finding words that rhyme, splitting a word into syllables, solving anagrams, or following an instruction like “reply without using the letter e”. All of them ask about units smaller than a token.

Spelling a word out with spaces or hyphens forces roughly one token per letter, putting every letter directly in front of the model. This is one reason models that reason step by step, writing out the letters before counting, do far better at these tasks. They are working around the tokenizer, one extra token at a time.

Numbers and arithmetic

Numbers are the other big casualty. How a tokenizer splits digits is a design choice. GPT-2 left it to the merges, so numbers were split into whatever chunks happened to be common. GPT-4’s tokenizer uses a rule: runs of up to three digits, taken from the left. The original LLaMA and PaLM split every digit into its own token instead (Singh and Strouse, 2024).

With left-to-right grouping, the chunk boundaries of numbers with different lengths rarely line up with ones, thousands and millions, so the model cannot add chunk by chunk the way you add column by column. Singh and Strouse tested exactly this: writing numbers with commas, which forces right-to-left groups of three, substantially improved GPT-3.5 and GPT-4 addition, and the errors under left-to-right grouping followed stereotyped patterns rather than looking random. Single-digit tokenization avoids the problem entirely at the cost of longer sequences, and larger models narrow the gap.

Why does it matter?

These are not signs that the model cannot reason about letters or numbers. They are signs that the information arrived in an inconvenient form. Knowing that lets you work around it: ask the model to spell words out, format numbers consistently (with separators, or digit by digit), or hand exact character-level and arithmetic work to a tool such as a code interpreter.

Key takeaways

• The model receives token ids, not letters, so counting, reversing or spelling requires memorised knowledge of what each token contains.
• Digit grouping is a tokenizer design choice; left-to-right chunks do not align with place value, which measurably hurts arithmetic.
• Workarounds follow from the mechanism: spell things out, format numbers consistently, or use a tool for exact work.

Tokens, money and languages

Model APIs charge per token and context windows are measured in tokens. So a tokenizer trained mostly on English quietly charges everyone else more for the same content.

What is the token tax?

Because BPE gives cheap, long tokens to whatever was common in its training corpus, the same meaning costs different numbers of tokens in different languages. Petrov and colleagues measured this across many tokenizers on parallel translations of the same text and found differences of up to 15 times in tokenized length between languages. Even tokenizers designed for many languages, and character or byte-level models, showed gaps of more than four times for some language pairs. Ahia and colleagues studied OpenAI’s API across 22 languages and found some needed up to five times more tokens than English, so speakers of those languages paid more for the same service and fitted less into the context window.

How big is the effect in real tokenizers?

We ran the same sentence, “The weather is nice today, so we are going to the park with the children,” translated into six languages, through three generations of OpenAI’s open-source tokenizers. The multiplier is relative to English under the same tokenizer.

Under GPT-2’s tokenizer the Hindi sentence cost five times as many tokens as the English one. GPT-4’s tokenizer all but closed the gap for the European languages but still charged Hindi and Greek more than three times as much. GPT-4o’s larger vocabulary nearly erased it for this sentence; OpenAI reported that its new tokenizer needs 2.9 to 4.4 times fewer tokens than before for several Indian languages and about 1.4 times fewer for Japanese. Bigger, more multilingual vocabularies are the main fix, which is also why Llama 3 added 28,000 tokens for non-English text.

A tokenizer trained only on English turns everything else into a mass of short pieces, and scripts it never saw into raw bytes. Production tokenizers are trained on multilingual data, which is why their gaps are smaller, but the principle is the same: the languages that dominate the tokenizer’s training data get the cheap tokens.

Estimating cost

Pricing is usually quoted per million tokens, with output tokens costing more than input tokens because they are generated one step at a time. A quick estimate for English prose:

Rule of thumb for English only. Code, numbers and other languages can differ a lot, so measure with the provider’s own token counter for anything important.

A 3,000-word English report is therefore about 4,000 tokens. The same report translated into a language that needs twice as many tokens is about 8,000, and costs twice as much to send.

Why does it matter?

Tokenization turns a technical choice into a question of fairness and access. The people who pay the token tax are disproportionately speakers of languages that were under-represented on the web, often in places where the same price is a larger share of income. When comparing models or budgeting a multilingual product, measure token counts in the languages you actually serve.

Key takeaways

• The same content can need several times more tokens in some languages; published studies found gaps of up to 15 times.
• More tokens means higher price, less text in the context window and slower generation for the same content.
• Larger multilingual vocabularies such as GPT-4o’s and Llama 3’s have shrunk the gap, but it depends on the tokenizer’s training data.

Glitch tokens

In early 2023, two researchers found that asking GPT-3 to repeat the phrase “SolidGoldMagikarp” made it say “distribute” instead. The cause was a token that existed in the vocabulary but that the model had barely ever seen.

What is a glitch token?

Jessica Rumbelow and Matthew Watkins were clustering the embedding vectors of GPT’s 50,257 tokens when they noticed a group of strange tokens sitting close to the centroid, the average of all the embeddings. The list included ·SolidGoldMagikarp, ·petertodd and several fragments that looked like usernames and pieces of web markup. When they prompted models to repeat these strings, the models evaded the request, misspelled the strings or substituted unrelated words.

How does a token end up untrained?

The tokenizer and the model are trained on different data. BPE gives a token to any string that is frequent in the tokenizer’s corpus. If a string was frequent there (for example, a username repeated thousands of times in one forum’s logs) but was filtered out or rare in the data used to train the model, the token gets an id and an embedding that is almost never updated. At inference time the model meets an input it has no learned meaning for, and its behaviour is unpredictable.

This is not a curiosity of one model. Land and Bartolo built automatic methods to find under-trained tokens using the tokenizer and the model’s weights, and found them across many open models. Tokenizer and training-data mismatches are common, and newer tokenizers fix specific cases rather than the cause: in GPT-4’s tokenizer, ·SolidGoldMagikarp is split into five ordinary tokens.

Why does it matter?

Glitch tokens are a clear example of a failure mode that sits entirely outside the model’s “knowledge”. They matter for reliability (strange outputs on innocent-looking input), for safety (unusual inputs are a natural place to hunt for jailbreaks) and for efficiency (vocabulary slots and parameters spent on tokens that do nothing). Checking for under-trained tokens is now a sensible step when releasing a model.

Key takeaways

• A glitch token is in the vocabulary but was rarely or never seen while training the model, so its embedding is essentially untrained.
• They arise because the tokenizer and the model are trained on different data; frequency in one does not guarantee exposure in the other.
• They cause erratic behaviour on specific strings and can be found automatically by inspecting the tokenizer and the model’s weights.

Beyond fixed vocabularies

If tokenization causes this many problems, why not remove it and let the model read bytes directly? Researchers have been trying, and in late 2024 it started to work at scale.

What would a tokenizer-free model look like?

The simplest version feeds the model raw UTF-8 bytes. ByT5 did this with a T5-style model and found it more robust to noise, misspellings and unusual text, and competitive in quality, but slower: byte sequences are several times longer than token sequences, and attention cost grows with length. Fixed-size chunks of bytes help with speed but reintroduce arbitrary boundaries.

How does the Byte Latent Transformer work?

Meta’s Byte Latent Transformer (BLT) groups bytes into patches of variable size. A small byte-level language model reads the text and, at each position, measures how uncertain it is about the next byte, its entropy. When the entropy spikes, a new patch begins. Predictable stretches (the rest of a common word) are folded into one long patch; surprising points (the start of a new word, a rare name, a number) get patches of their own. A large transformer then works on patches instead of tokens.

The effect is that compute goes where the text is hard to predict, instead of being fixed by a vocabulary chosen before training. In scaling experiments up to 8 billion parameters and 4 trillion training bytes, BLT matched the performance of tokenizer-based models, with better inference efficiency and more robustness on character-level tasks.

With a low threshold nearly every character is surprising enough to start a patch, and you are back to a character-level model. Raise it and patches grow: the ends of familiar words disappear into the patch that began them, while unfamiliar names keep breaking into short patches. The segmentation adapts to the text instead of being fixed in advance.

Why does it matter?

Tokenizer-free models promise to remove the language tax, the spelling blind spot and glitch tokens in one move. They are not yet the default: the open-weight model families most people run, such as Llama, Qwen and DeepSeek, have used byte-level BPE vocabularies, and every commercial API still bills by the token. For now, understanding tokenization remains part of understanding what a language model can and cannot see.

Key takeaways

• Byte-level models remove the tokenizer but face longer sequences; ByT5 was more robust but slower.
• The Byte Latent Transformer groups bytes into patches where a small model’s next-byte entropy spikes, spending compute where text is hard.
• BLT matched tokenizer-based models up to 8B parameters, but widely used models still rely on subword tokenizers.

Check your understanding

Seven situations you could meet when building with language models. Each one asks you to apply an idea from the lesson.

Your support chatbot handles English and Thai. Monthly API spend for Thai conversations is far higher than for English, even though the conversations are about the same length and topic. What is the most likely cause?

References

The papers and documentation cited in this lesson. The real tokenizer counts come from OpenAI’s open-source tiktoken library; every other number in the demos is computed live in your browser.

Sources

1. [1]

   What are tokens and how to count them?(opens in a new tab)

   OpenAI, 2024

   OpenAI help article. Rule of thumb for English: one token is about four characters, or three quarters of a word.

2. [2]

   tiktoken: a fast BPE tokeniser for use with OpenAI’s models(opens in a new tab)

   OpenAI, 2023

   Open-source implementation and definitions of the r50k, cl100k_base and o200k_base encodings.

3. [3]

   The Llama 3 Herd of Models(opens in a new tab)

   Grattafiori, A., Dubey, A., et al. (Llama Team, Meta), 2024

   Llama 3 technical report. Uses a 128K-token vocabulary: 100K tokens from tiktoken plus 28K added for non-English languages.

4. [4]

   A New Algorithm for Data Compression(opens in a new tab)

   Gage, P., 1994

   The C Users Journal 12(2). The original byte-pair encoding: repeatedly replace the most common pair of bytes with an unused byte.

5. [5]

   Neural Machine Translation of Rare Words with Subword Units(opens in a new tab)

   Sennrich, R., Haddow, B., Birch, A., 2016

   ACL 2016. Adapts BPE to learn a subword vocabulary, so translation models can handle rare and unseen words.

6. [6]

   Language Models are Unsupervised Multitask Learners(opens in a new tab)

   Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., 2019

   The GPT-2 paper. Introduces byte-level BPE with a 50,257-token vocabulary and pre-tokenization that stops merges across character categories.

7. [7]

   SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing(opens in a new tab)

   Kudo, T., Richardson, J., 2018

   EMNLP 2018 (demo track). A tokenizer library that treats text as a raw stream, including spaces, and supports BPE and unigram models.

8. [8]

   Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs(opens in a new tab)

   Singh, A. K., Strouse, D. J., 2024

   Shows that forcing right-to-left digit grouping (with commas) substantially improves GPT-3.5 and GPT-4 addition, with stereotyped errors otherwise.

9. [9]

   Language Model Tokenizers Introduce Unfairness Between Languages(opens in a new tab)

   Petrov, A., La Malfa, E., Torr, P. H. S., Bibi, A., 2023

   NeurIPS 2023. The same text translated into different languages can differ in token length by up to 15 times, affecting cost, latency and context.

10. [10]

    Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models(opens in a new tab)

    Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., Tsvetkov, Y., 2023

    EMNLP 2023. Measures OpenAI API costs across 22 languages; some need up to five times more tokens for the same content.

11. [11]

    Hello GPT-4o(opens in a new tab)

    OpenAI, 2024

    GPT-4o launch post. Its new tokenizer needs 2.9 to 4.4 times fewer tokens for several Indian languages and about 1.4 times fewer for Japanese.

12. [12]

    SolidGoldMagikarp (plus, prompt generation)(opens in a new tab)

    Rumbelow, J., Watkins, M., 2023

    LessWrong, February 2023. Discovers anomalous “glitch tokens” clustered near the centroid of GPT’s token embeddings.

13. [13]

    Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models(opens in a new tab)

    Land, S., Bartolo, M., 2024

    EMNLP 2024. Methods to find tokens that are in the vocabulary but barely trained; finds them across many open models.

14. [14]

    ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models(opens in a new tab)

    Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., Raffel, C., 2022

    TACL 2022. A T5 variant that reads raw UTF-8 bytes; more robust to noise and spelling, but slower because sequences are longer.

15. [15]

    Byte Latent Transformer: Patches Scale Better Than Tokens(opens in a new tab)

    Pagnoni, A., Pasunuru, R., Rodriguez, P., et al., 2024

    Meta. A byte-level model that groups bytes into dynamic patches by next-byte entropy; matches tokenizer-based models up to 8B parameters and 4T training bytes.