Next-Token Sampling Lab

About

Every large language model, from GPT-3 to today's reasoning models, writes by repeating one move: look at the text so far, produce a probability for every possible next token, pick one, append it, repeat. This lab lets you do that move by hand and watch each choice.

The model here is deliberately tiny: an n-gram model trained in your browser, in milliseconds, on a few thousand words of public-domain text. It is a real language model (it assigns a proper probability to every token and we measure it on text it never saw), just an old-fashioned one. The decoding controls, though, are exactly the ones you meet in LLM APIs: temperature, top-k and top-p. The maths that turns a distribution into a choice is identical whether the distribution came from counting words or from a trillion-parameter transformer.

What to look at

• The bar chart: the model's probabilities (dashed outline) versus the probabilities sampling actually uses (filled).
• The temperature chart: how each candidate's probability rises or falls as you turn the dial.
• The colour behind each generated token: how likely the model thought it was. Click one to see what else it could have said.
• Greedy vs sampled: why “always pick the most likely word” is a bad way to write.

How it works

Tokens

In Words mode the text is split into words and punctuation (the newline is a token too, shown as ↵). In Characters mode every character is a token, which gives a small vocabulary and lets the model spell words it never saw. Real LLMs sit in between: they use subword tokens learned by byte-pair encoding, so common words are one token and rare ones are a few.

Training is counting

An n-gram model assumes the next token depends only on the previous n−1 tokens. Training means counting, for every context seen in the text, which tokens followed it. The first 90% of the text is used for counting; the last 10% is held out.

Smoothing: never say never

Raw counts give zero probability to anything not seen in a context, and with a few thousand words almost everything is unseen. This model uses Witten–Bell interpolation: it blends the long-context estimate with the shorter-context one, trusting the long context less when many different tokens have followed it.

Here c(h) is how often the context h was seen, t(h) how many distinct tokens followed it and h′ is h without its oldest token. The recursion ends in a unigram model mixed with a uniform distribution, so every token in the vocabulary keeps a sliver of probability. That sliver is the long tail you unleash with a high temperature.

Perplexity

To score the model we feed it the held-out text and average the negative log-probability it gave each actual next token. Perplexity is the exponential of that average:

A perplexity of 50 means the model was, on average, as uncertain as if it were choosing uniformly among 50 tokens. The small bar chart in the control rail recomputes it for every order: longer contexts help at first, then hurt, because the counts become too sparse to trust. Large models escape that trap by sharing statistical strength across similar contexts instead of treating each one as a separate row in a table.

Temperature

A neural language model outputs a score (a logit) zᵢ for every token and turns scores into probabilities with the softmax. Temperature T divides the scores first:

Our n-gram model gives probabilities rather than logits, but log pᵢ plays exactly the same role (softmax of log p at T = 1 gives back p), so the lab computes softmax(log p / T). That is why the dashed line at T = 1 on the chart matches the model's own distribution.

A worked example

Say three candidates have probabilities 0.6, 0.3 and 0.1.

• T = 0.5 squares them before renormalising: 0.36, 0.09, 0.01 → 0.78, 0.20, 0.02
• T = 1 leaves them alone: 0.60, 0.30, 0.10
• T = 2 takes square roots: 0.77, 0.55, 0.32 → 0.47, 0.34, 0.19

As T approaches 0 all the mass piles onto the top token (greedy decoding); as T grows the distribution approaches uniform over the whole vocabulary. Temperature never changes the order of the candidates, only how far apart they are.

Top-k and top-p

Even a good model spreads a little probability over thousands of unlikely tokens. Each is individually rare, but together they are not: sample long enough and you will hit one, and one bad token can derail everything after it. Truncation methods cut that tail off before sampling.

Top-k

Keep the k most likely tokens, renormalise, sample. Simple, but k is fixed: after “the White” almost all the mass is on one word and k = 40 still lets 39 bad options in, while after “she said” the plausible set is large and k = 5 is too strict.

Top-p (nucleus sampling)

Sort tokens by probability and keep the smallest set whose total reaches p. The set shrinks when the model is confident and grows when it is not. Holtzman et al. (2020) introduced it after showing that both greedy/beam search (bland, repetitive) and pure sampling (incoherent) produce text that looks unlike human writing; nucleus sampling matched human text statistics much better.

Order of operations

This lab applies temperature, then top-k, then top-p, then renormalises and draws. That matches common open-source decoders, but providers differ, and some only expose a subset of the knobs. Newer truncation rules such as min-p (keep tokens whose probability is at least a fraction of the top one) follow the same idea.

Tiny vs real LLMs

The sampling step is the same. Almost everything that produces the distribution is different.

One consequence: an assistant model's distribution is often far sharper than this toy's, because post-training pushes it towards one preferred answer. That is why temperature matters less for many chat tasks than it does here, and why providers recommend specific settings for reasoning models. Read the full story in the Large Language Models lesson, or see a transformer compute its next-token distribution in the Transformer Explorer.

Experiments

1. Watch greedy loop. Choose Alice, press Compare. Greedy text quickly repeats itself (in our runs it gets stuck on “thought Alice”). Its perplexity is lower than the sampled text's, yet it reads worse. Likelihood is not quality.
2. Melt the model. Set temperature to 2.5 with top-p off. Rare tokens from the long tail flood in. Now set top-p to 0.9 and keep the high temperature: the nucleus keeps the text mostly on the rails.
3. Find the sweet spot for context. Switch to Characters on Alice and look at held-out perplexity by order. Generate with n = 2, then n = 5, then n = 8. Low orders babble; high orders start copying the training text verbatim, the n-gram version of memorisation.
4. Reproducibility. Generate 20 tokens, reset, generate again: identical. Change only the seed: different. Real APIs behave similarly, although batching and floating-point effects mean they rarely guarantee bit-exact repeats.
5. Top-k's blind spot. Set top-k to 5 and step through a sentence while watching “Candidates kept”. Compare with top-p 0.8: the number of kept tokens moves with the model's confidence.