Text Playground

About

Computers do not read words. Before any language model, search engine or spam filter can do anything with text, the text has to become numbers. This playground shows four of the oldest and most important ways of doing that, each running for real on whatever you type:

• Tokens: cutting text into units. Includes a genuine byte-pair-encoding tokenizer, the same family of algorithm used by GPT-style models, trained in your browser.
• Sentiment: a rule-based scorer that follows the VADER algorithm, with every word's contribution visible.
• Keywords: TF-IDF, the weighting behind decades of search engines, comparing your text with five sample documents.
• N-grams: counting word and character sequences, and a bigram model that generates text from your own counts.

None of these understand language the way a modern model does, and each tab says where it breaks. They are still everywhere: every large language model starts with a tokenizer, TF-IDF-style scoring still powers keyword search alongside embeddings, and n-gram statistics are how the idea of a language model began. Continue with the Natural Language Processing lesson, the Embedding Explorer for meaning as vectors, and the LLM Sampling Lab for how models choose the next token.

How it works

What is simplified

• The BPE tokenizer is real, but tiny: trained on a 1065-word corpus written for this lab, with up to 700 merges. Production tokenizers learn tens or hundreds of thousands of merges from enormous corpora (GPT-2's vocabulary has 50,257 entries), and work on bytes rather than characters.
• The sentiment rules and constants follow VADER, but the lexicon is a hand-built list of 187 words with valences we assigned, not VADER's crowd-rated lexicon of about 7,500 entries.
• TF-IDF and n-gram counts are exact. The comparison corpus is just five short paragraphs, so idf values are coarse.

Why tokenization is not neutral

Token counts decide cost and context limits for language models, and they are not equal across languages or styles. Text that looks like the training corpus compresses into few tokens; unusual words, other languages, code and capital letters fragment. Try pasting a sentence in another language, or the same sentence in ALL CAPS, and watch the chars per token fall.

Tokenization

Whitespace splitting is the naive baseline: "model." and "model" become different tokens, and a vocabulary of whole words can never cover every word. Words and punctuation fixes the punctuation problem but still needs a fixed vocabulary, so unseen words become an unknown token. Characters never see an unknown word but make sequences long.

Byte-pair encoding sits in between (Sennrich, Haddow and Birch, 2016). Training starts from characters and repeatedly merges the most frequent adjacent pair into a new symbol. Early merges build common chunks like "th", "▁the" and "ing"; later ones build whole frequent words. To tokenize new text, the learned merges are replayed in the order they were learned. Frequent words become one token, rare words fall apart into pieces, and nothing is ever unknown.

Use the merges slider to watch this happen: at 0 merges BPE is a character tokenizer; as merges are added, tokens grow and the count drops. The "▁" marks a preceding space, the convention from SentencePiece (Kudo and Richardson, 2018). The number shown on hover is the token id a model would actually receive.

Sentiment rules

VADER (Hutto and Gilbert, 2014) scores text by looking words up in a lexicon of valences and adjusting them with a handful of rules drawn from how people actually write:

• A negation within the three previous words flips and dampens a word: multiply by −0.74. "not good" is mildly negative, not the mirror of "good".
• Boosters ("very", "extremely") add 0.293 in the direction of the word; dampeners ("slightly") subtract it.
• A word in ALL CAPS, in otherwise mixed-case text, gains 0.733.
• "But" shifts weight: words before it count half, words after it count one and a half times.
• Each exclamation mark (up to four) adds 0.292 to the total.

The summed score is squashed into a compound score between −1 and 1 with x / √(x² + 15). A common convention treats compound ≥ 0.05 as positive and ≤ −0.05 as negative.

Where it breaks: sarcasm ("great, another delay"), domain words ("unpredictable" is good for a thriller, bad for a car), and anything not in the lexicon. Modern sentiment systems fine-tune a transformer on labelled examples instead, which learns these patterns from data rather than rules.

TF-IDF

TF-IDF weighs a word by how often it appears in a document (term frequency) and how rare it is across the collection (inverse document frequency) (Spärck Jones, 1972). "The" is frequent everywhere, so it carries no information about any one document; "telescope" appears in one document and says a lot about it.

tfidf(w, d) = count(w, d) · (ln((1 + n) / (1 + df(w))) + 1), then each document vector is scaled to length 1

That is the scikit-learn default formula, used here exactly. Your text is document six. Turn stop-word removal off to see how much of a document's raw counts are function words, and how IDF pushes them down anyway, though not to zero with only six documents.

TF-IDF vectors are sparse and literal: "car" and "automobile" share nothing. Dense embeddings fix that, which is why modern search often combines the two. See the Embeddings and RAG lesson.

N-grams

An n-gram is a run of n consecutive units. Counting them is the oldest statistical model of language: Shannon used letter and word n-grams to generate increasingly English-looking text in 1948 (Shannon, 1948).

A bigram model predicts the next word from the previous word alone: P(next | word) = count(word, next) / count(word). Click a word in the explorer to see that distribution from your text, and generate a chain by sampling from it. The results are locally plausible and globally meaningless, because the model forgets everything beyond one word.

Notice how many n-grams occur exactly once (hapax legomena). As n grows, almost every n-gram is unique: the data sparsity problem that made n-gram models plateau, and that neural language models, which share statistical strength between similar contexts, were built to solve.