Tokenizer Lab

About

A language model never reads letters. Before any text reaches the network, a tokenizer chops it into pieces from a fixed vocabulary and replaces each piece with an integer id. Almost every modern model, from GPT-2 to GPT-4o and Llama 3, uses a variant of the same algorithm to build that vocabulary: byte-pair encoding (BPE).

This lab lets you train one. Pick a corpus, press Next merge, and watch the algorithm find the most common pair of neighbouring symbols, glue it into a new token, and go again. After a few dozen merges, fragments like ·t, he and ·the appear. After a few hundred, common whole words become single tokens while rare words stay in pieces.

The test string is re-tokenized after every merge, so you can see the consequences directly: how compression improves, why text in a language the tokenizer never saw costs many more tokens, and why a model can struggle to count the letters in strawberry.

The Tokenization lesson explains the ideas step by step; this lab is where you can break them.

How it works

1. Choose the starting alphabet

With Characters, the base vocabulary is every distinct character in the corpus. It is small and readable, but a character the corpus never contained (a Japanese kana, an emoji) has no id and becomes [UNK], an unknown token the model cannot read.

With Bytes, the base vocabulary is the 256 possible values of a byte, and text is first converted to UTF-8. Every string on Earth is some sequence of bytes, so nothing is ever unknown. This is the trick GPT-2 introduced and most large models now use. The cost: characters outside ASCII take two to four bytes, so unmerged non-English text becomes long runs of byte tokens like ‹e3›.

2. Pre-tokenize (optional)

Real tokenizers first split text with a regular expression into word-like chunks, and never merge across chunk boundaries. A word keeps its leading space, so ·the (mid-sentence) and The (sentence start) are different tokens. Turn off Split into words first and merges start swallowing spaces and punctuation, producing tokens like e· and ,·and.

Digits get special treatment. GPT-4’s tokenizer keeps runs of up to three digits together; the original LLaMA split every digit apart. Toggle One digit per chunk and train on the Numbers corpus to see the difference.

3. Count, merge, repeat

Each step counts how often every adjacent pair of symbols occurs across the corpus, weighting each chunk by how many times it appears. The winner becomes a new symbol and is recorded in the merge table with the next free id. Every merge makes the corpus shorter by exactly as many tokens as there were non-overlapping occurrences of the pair, which is what the compression stat tracks.

4. Encoding new text

To tokenize a new string, split it the same way, start from characters or bytes, and repeatedly apply whichever learned merge has the lowest rank (was learned earliest) until none applies. Order matters: the merge table is not a dictionary of words but an ordered program. That is why the test string changes as you scrub the slider.

Why it matters

Spelling and counting

Choose the Strawberry test string. The model does not receive the letters s-t-r-a-w-b-e-r-r-y; it receives a few ids. To count the r’s it has to have learned, from training data, which letters each token contains. Usually it has; sometimes the knowledge is shaky. This is one reason models historically miscounted letters, and why reasoning models that spell a word out one character per step do better.

Arithmetic

Train on Numbers with digit grouping on, then look at the Sum test string. 48213 splits as 482 + 13: grouped from the left, so the chunks do not line up with ones, tens and hundreds. Singh and Strouse (2024) showed that forcing right-to-left grouping (by adding commas) measurably improved GPT-3.5 and GPT-4 addition. Single-digit tokens avoid the problem at the cost of longer sequences.

The token tax

Train on English and switch the test string between languages. The same short sentence takes a handful of tokens in English, more in German and Spanish, and many more in Hindi or Japanese, which this corpus never saw. Petrov and colleagues (2023) measured the same effect in commercial tokenizers: some languages need up to 15 times as many tokens for the same content, so speakers pay more and get less context for the same money.

Glitch tokens

If a string is common in the tokenizer’s training data but rare in the model’s, it gets its own token whose embedding is barely trained. The famous example, ·SolidGoldMagikarp (a Reddit username), made GPT-3 era models behave erratically when asked to repeat it. Your tokenizer shows how this happens: whatever repeats in the corpus becomes a token, whether or not it means anything.

Try this

1. 01Watch words assembleEnglish corpus, Characters, Slow. Play the first 40 merges and note which fragments appear first. Spaces attach to the front of words.
2. 02Break the word boundaryTurn off Split into words first and retrain. Which merges now contain spaces or punctuation? Is compression better or worse after 300 merges?
3. 03Unknown versus bytesTest string Japanese with the English corpus. With Characters every symbol is [UNK]. Switch to Bytes: nothing is unknown, but count the tokens.
4. 04Home advantageTrain on German, then compare the English and German test strings at 300 merges. The language the tokenizer learned from is always the cheap one.
5. 05Digit groupingNumbers corpus. Tokenize the Sum string with and without One digit per chunk. Which version lines up digits by place value?
6. 06Code is its own languageTrain on Code. Watch indentation (runs of spaces) and keywords like ·return become single tokens early.