Natural Language Processing

Why language is hard

You used natural language processing several times today. Your inbox filtered spam, your phone suggested the next word, a search engine understood a half-typed question, and perhaps a chatbot drafted an email. Each of those systems has to turn messy human language into numbers a computer can work with.

What is natural language processing?

Natural language processing (NLP) is the part of AI that reads, interprets and produces human language. Its tasks range from simple to astonishing: classifying a message as spam, pulling names and dates out of a contract, translating between languages, answering questions, summarising a report, and holding a conversation.

For most of its history NLP was a collection of separate systems, one per task. Today many of those tasks are handled by a single large language model. But the building blocks inside that model are the ones in this lesson: tokens, vectors, sequence models and attention. Understand those and the modern systems stop being magic.

Why is language so hard for computers?

Language is ambiguous at every level, and people resolve that ambiguity effortlessly using context and knowledge of the world. Computers get none of that for free.

The trophy example comes from the Winograd Schema Challenge, which was designed so that grammar alone can’t resolve the pronoun: you need to know that big things don’t fit into smaller things. Language is also wildly uneven. A handful of words (the, of, and) make up a large share of any text, while most words in a vocabulary are rare. A system trained on a few million sentences will still meet words it has never seen.

Why does it matter?

Language is how humans store and share almost everything they know: laws, science, medicine, code comments, customer complaints. A machine that can process it can search, summarise and act on that knowledge at a scale no team of people could. It is also where AI’s failures are most visible, from biased hiring filters to confident chatbot errors, so understanding how these systems represent language is a practical skill, not trivia.

Key takeaways

• NLP turns human language into numbers so machines can classify, extract, translate and generate it.
• Ambiguity in word meaning, structure, reference and intent is resolved by context and world knowledge, which machines must learn.
• Modern language models are built from the same pieces this lesson covers: tokens, embeddings, sequence modelling and attention.

Tokenization

A neural network cannot read letters. It reads lists of integers. Tokenization is the step that decides which chunks of text get an integer, and that quiet decision shapes what a model finds easy, what it finds hard, and how much it costs to use.

What is a token?

A token is one unit from a fixed vocabulary. The tokenizer splits text into those units and replaces each with its ID. The obvious choices both have problems:

Modern systems use subwords, the middle ground: frequent words get a single token, rarer words are assembled from pieces. The most common way to find those pieces is byte-pair encoding.

How does byte-pair encoding work?

Byte-pair encoding (BPE) started life as a data-compression trick and was adapted for language by Sennrich, Haddow and Birch in 2016. It learns a vocabulary from a corpus with a simple greedy loop:

1. Start with every word split into single characters.
2. Count every pair of adjacent symbols across the corpus.
3. Merge the most frequent pair into a new symbol, and record that merge rule.
4. Repeat until the vocabulary reaches the size you want.

To tokenize new text, split it into characters and replay the merge rules in the order they were learned. The number of merges is a dial: more merges means a bigger vocabulary and fewer, longer tokens per sentence. Production tokenizers run tens of thousands of merges, and most start from the 256 possible bytes rather than characters so that no input, in any script or even emoji, can ever be unknown.

At zero merges every character is its own token. The first merges glue together the most common pairs in the corpus (a+l, al+k, e+r), and whole fragments like walk, er and est soon become tokens because they repeat. Notice that BPE knows nothing about morphology. It may split “unkindness” into odd pieces, because it optimises for frequency in its training text, not for linguistic sense.

Why does tokenization matter?

Everything about using a language model is counted in tokens: the price of an API call, the size of the context window, the speed of generation. A commonly quoted rule of thumb is that one token is roughly three quarters of an English word. That ratio is much worse for languages the tokenizer saw less of during training. Petrov and colleagues found the same text can need up to 15 times more tokens in some languages than in English, which means higher cost, slower responses and less room in the context for speakers of those languages.

Tokenization also explains some famous LLM blind spots. Asking how many r’s are in “strawberry” is hard for a model that sees the word as one to three opaque token IDs rather than ten letters. Arithmetic on long numbers suffers when digits are chunked inconsistently. When a model stumbles on something character-level, suspect the tokenizer first.

Key takeaways

• Tokenizers map text to integer IDs; subword tokenizers balance vocabulary size against sequence length.
• BPE repeatedly merges the most frequent adjacent pair, then replays those merges to tokenize new text.
• Token counts drive cost and context length, and they are uneven across languages, which creates real unfairness.

Counting words: TF-IDF

The simplest way to turn a document into numbers is to count its words. It sounds crude, and it throws away grammar entirely, yet counting powered search engines for decades and is still inside the retrieval systems that feed today’s chatbots.

What is a bag of words?

A bag-of-words vector has one dimension for every word in the vocabulary. A document’s value in each dimension is how often that word appears. Word order is discarded, so “dog bites man” and “man bites dog” get identical vectors. That is the bag: the words are all there, jumbled together.

Raw counts have an obvious flaw: “the” and “and” dominate every document while telling you nothing about what it is about. The fix is to weight each word by how distinctive it is.

How does TF-IDF weighting work?

TF-IDF multiplies term frequency (how often a word appears in this document) by inverse document frequency (how rare the word is across all documents). The IDF idea came from Karen Spärck Jones in 1972: a word that appears in few documents is a better signal of what those documents are about.

N is the number of documents; df(w) is how many of them contain w.

With the five documents in the demo below, “the” appears in four of them, so its IDF is ln(5/4) ≈ 0.22: nearly worthless. “bank” appears in three, so ln(5/3) ≈ 0.51. “interest” appears in only one, so ln(5/1) ≈ 1.61. To rank documents against a query, we build the query’s vector the same way and compare directions with cosine similarity, which ignores document length:

1 means pointing the same way, 0 means no shared weighted words.

The query “bank interest rates” ranks D4 first by a wide margin because it contains the two rare words, “interest” and “rates”. D3 and D5 score only a little, through the moderately common word “bank”. Notice what TF-IDF cannot do: it has no idea that D5 means a river bank. It also fails completely on synonyms. A query for “automobile” scores zero against a document about cars, because different words are different dimensions.

Why does it matter?

TF-IDF and its refined descendant BM25 are still the default ranking in widely used search engines like Lucene and Elasticsearch, and they remain hard to beat for exact names, product codes and rare technical terms. Retrieval-augmented generation systems often combine this kind of keyword search with embedding search, getting exact matches from one and meaning from the other.

The synonym problem is the key limitation, and it motivates the next idea: vectors where similar words point in similar directions.

Key takeaways

• Bag of words represents a document by word counts and discards order.
• TF-IDF weights each word by how rare it is across documents, so distinctive words dominate the vector.
• Keyword methods are fast and exact but blind to synonyms and context, which is why embeddings were needed.

Word embeddings

In 1957 the linguist J. R. Firth wrote a line that became the motto of modern NLP: “You shall know a word by the company it keeps.” If two words keep turning up in the same kinds of sentences, they probably mean similar things. Turn that observation into arithmetic and you get word embeddings.

What is a word embedding?

An embedding is a short, dense list of numbers for each word, typically a few hundred, arranged so that words with similar meanings have similar vectors. Compare that with bag of words, where every word is its own axis and “cat” and “kitten” are exactly as unrelated as “cat” and “carburettor”.

The principle behind it is the distributional hypothesis, stated by Zellig Harris in 1954: words that occur in similar contexts tend to have similar meanings. You have used it yourself. If you read “I fed the wug and it purred”, you know roughly what a wug is without a dictionary.

How are embeddings learned?

There are two classic routes, and they turn out to be close relatives.

Counting co-occurrences

Slide a window over the text and count how often each word appears near each other word. Raw counts are dominated by frequent words, so reweight them with pointwise mutual information, which asks whether two words appear together more than chance would predict:

Negative values are set to zero (positive PMI). Each word’s row of PPMI values is its vector.

Then compress those long, sparse rows into short, dense vectors with a matrix factorisation such as the singular value decomposition.

Predicting context: word2vec

In 2013 Tomas Mikolov and colleagues at Google released word2vec, which learns vectors by prediction instead. The skip-gram variant takes each word and trains a tiny network to predict the words around it. The vectors that make those predictions work are the embeddings. It was fast enough to train on billions of words, and the results were striking. Later, Levy and Goldberg showed that skip-gram is implicitly factorising a shifted PMI matrix: the prediction route and the counting route find much the same structure.

Three clusters appear on their own: animals, food and vehicles. “cat” lands next to “kitten” and “dog” because they are all chased, fed, stroked and sleep on sofas; “bus” lands next to “train” because both are late, leave stations and go to work. The verbs join their clusters too, since “cooked” keeps the same company as “pasta”. Edit the corpus to teach it something new, such as a sentence where the cat drives to work, and watch the space shift.

Directions with meaning

Trained on billions of words, embeddings also encode relationships as directions. Mikolov, Yih and Zweig showed that vector arithmetic can answer analogies: the vector for king, minus man, plus woman lands closest to queen. The picture below is a sketch of that idea, not real data.

Why does it matter?

Embeddings turned meaning into geometry. Similarity became a distance, so search, recommendation, clustering and classification could all work on meaning rather than exact words. Every large language model starts by looking up an embedding for each token.

Static word vectors have one big limitation: each word gets exactly one vector. “Bank” gets a single blend of the river sense and the money sense. Fixing that required models that compute a fresh vector for each word in its context, which is where the transformer comes in.

Key takeaways

• Embeddings are dense vectors where words that appear in similar contexts end up close together.
• Counting co-occurrences with PMI and predicting context with word2vec are two routes to the same kind of structure.
• Static embeddings give one vector per word and absorb social biases from their training text.

Sentiment analysis

Companies read millions of reviews, support tickets and social posts they could never read by hand. Sentiment analysis, deciding whether a piece of text is positive or negative, is one of the oldest and most widely deployed NLP tasks, and a perfect place to watch a classifier think.

What is sentiment analysis?

It is text classification where the labels are opinions: positive or negative, sometimes neutral, sometimes a star rating. Real systems often go further and ask what the opinion is about (“great screen, awful battery”), which is called aspect-based sentiment.

How does a naive Bayes classifier decide?

Naive Bayes asks which label makes the observed words most probable. From labelled examples it estimates, for every word, how often it appears in positive versus negative text. The “naive” part is assuming each word is independent of the others, so the evidence simply multiplies, or in logs, adds up:

Add-one (Laplace) smoothing stops a single unseen word from making a probability zero.

Each word contributes log P(word | pos) − log P(word | neg): positive for words like “delicious”, negative for “rude”, near zero for words that appear equally in both. Pang, Lee and Vaithyanathan’s 2002 film-review study found that learned classifiers like this clearly beat lists of sentiment words chosen by people.

With only 50 training reviews the model is fragile, but its reasoning is fully visible. Without negation marking, “not good” counts as evidence for positive because “good” is a positive word. Marking negation turns it into a separate feature, NOT_good, which the training data associates with negative reviews. Sarcasm (“Oh great, another two hour wait. Just perfect.”) defeats it completely: every word looks positive, and only world knowledge tells you that a two hour wait is bad.

Why does it matter?

Brand monitoring, customer support triage, market research and content moderation all run on text classifiers. Today the most accurate options are fine-tuned transformers, or simply prompting an LLM with the review. Naive Bayes and logistic regression still earn their place: they train in seconds, run on anything, need no GPU, and every decision can be explained word by word, which matters when a regulator or a customer asks why.

The failures you just saw (negation, sarcasm, context) all come from ignoring word order. To fix that properly, models need to read text as a sequence.

Key takeaways

• Naive Bayes adds up per-word evidence learned from labelled examples; it is fast, simple and fully explainable.
• Treating words independently breaks on negation and sarcasm, where meaning depends on order and world knowledge.
• Simple classifiers are strong baselines; transformers and LLMs win when context matters.

Sequence models

“The dog chased the cat” and “the cat chased the dog” contain the same words and mean opposite things. To capture meaning, a model has to read in order, carrying forward what it has seen so far. That is what recurrent neural networks were built to do.

What is a recurrent neural network?

A recurrent neural network (RNN) reads one token at a time and keeps a hidden state, a vector that acts as its running memory. At each step it combines the new token with the previous state to make the next state:

The same weights W and U are reused at every step, so one small network can read a sequence of any length.

How is it trained, and what goes wrong?

Training uses backpropagation through time: unroll the network across the sequence, compute the error at the end, and send the gradient back step by step. Here is the catch. Each step back multiplies the gradient by the recurrent weights and the slope of the activation. Multiply by something less than one twenty times and almost nothing is left; multiply by something bigger than one and it explodes. Bengio, Simard and Frasconi showed in 1994 that this makes long-range dependencies very hard to learn with gradient descent.

The long short-term memory network (LSTM) of Hochreiter and Schmidhuber fixed this with a separate cell state that is updated by addition rather than repeated multiplication, and three learned gates: a forget gate that decides what to keep, an input gate that decides what to write, and an output gate that decides what to reveal. When the forget gate stays near 1, information and gradient can survive for many steps.

With w = 1, the RNN keeps 0.8²⁰ ≈ 0.012 of the signal after 20 words, about one percent, and a millionth after 60. Push w above 1.25 and the product grows instead, which is the exploding gradient. The LSTM with f = 0.97 still keeps 0.97²⁰ ≈ 0.54. In a real LSTM the gate is learned and can be different for every dimension and every word, so the network chooses what to remember.

Why does it matter?

LSTMs were the workhorse of NLP in the mid 2010s: speech recognition, keyboard prediction, handwriting recognition and the first neural translation systems deployed at scale all used them. Paired as an encoder and a decoder, they could read a sentence in one language and write it in another.

Two limits remained. The whole input still had to squeeze through one hidden state, and the sequential loop meant step 100 could not be computed until step 99 was done, so training could not use the full parallel power of GPUs. Both limits pointed to the same solution.

Key takeaways

• RNNs read tokens in order and carry a hidden state, so word order finally matters.
• Gradients shrink or explode as they travel back through many steps, so plain RNNs forget long-range context.
• LSTMs keep a gated, additive memory that preserves information longer, but they still process one step at a time.

The transformer turn

Between 2014 and 2018 NLP changed more than in the previous thirty years. One idea, attention, removed the memory bottleneck of recurrent networks, and a new habit, pretraining one big model on unlabelled text, removed the need to build a separate model for every task.

What is attention?

Attention lets a model, at each position, look directly at every other position and decide how much each one matters. Bahdanau, Cho and Bengio introduced it for translation in 2014: instead of compressing a whole sentence into one vector, the decoder could look back at the encoder’s state for every source word while writing each target word.

In 2017 Vaswani and colleagues proposed the transformer, which dropped recurrence entirely and used attention alone. Any word can reach any other word in a single step, so there is no long chain for gradients to vanish along, and every position is processed at once, so training parallelises beautifully on GPUs.

How did it change NLP?

Transformers produce contextual embeddings. Instead of one fixed vector for “bank”, each layer recomputes the word’s vector from its neighbours, so “river bank” and “bank loan” end up in different places. That fixed the core weakness of word2vec.

The second change was pretraining. BERT (2019) was pretrained on large amounts of unlabelled text by filling in masked words, then fine-tuned with a small labelled set for each task. It set new records across the GLUE benchmark of language-understanding tasks. Meanwhile GPT-style models were trained to predict the next token and simply scaled up. That line led to today’s large language models, where most NLP tasks are handled by describing them in a prompt.

Why does it matter?

Nearly every state-of-the-art language system today is a transformer. The classic NLP ideas did not disappear, though. Tokenizers are still BPE-style, the first layer is still an embedding table, retrieval systems still blend keyword and vector search, and the evaluation problems are sharper than ever. Knowing the pieces tells you where these systems are strong and where they break.

Key takeaways

• Attention lets every position consult every other position directly, removing the recurrent bottleneck.
• Transformers give each word a context-dependent vector and train in parallel, which made very large models practical.
• Pretraining on unlabelled text, then fine-tuning or prompting, replaced one-model-per-task NLP.

Measuring and using NLP

Language has no single right answer. There are many correct translations of a sentence and many good summaries of a report. That makes measuring an NLP system surprisingly hard, and mistakes in measurement lead to shipping the wrong model.

What do we measure?

It depends on the task:

• Classification (spam, sentiment, intent): accuracy, precision, recall and F1, measured on a held-out test set.
• Translation and summarisation: overlap metrics like BLEU and ROUGE against human references, increasingly supplemented by learned metrics and human or model judges.
• Language models: perplexity, the exponential of the average negative log-probability per token. A perplexity of 20 means the model is as uncertain as if it were choosing uniformly among 20 tokens at each step.

How does BLEU score a translation?

BLEU, introduced by Papineni and colleagues at IBM in 2002, counts how many of the system’s 1-, 2-, 3- and 4-word sequences also appear in a human reference, clipping repeats so that “the the the” can’t cheat. It takes the geometric mean of those four precisions and multiplies by a brevity penalty so that very short outputs can’t score well by saying little.

The paraphrase “a cat sits upon the rug” means the same thing as the reference and scores near the bottom, while the close match scores high for changing one word. The shuffled word salad keeps perfect 1-gram precision, and only the longer n-grams expose it. BLEU is useful averaged over thousands of sentences to track progress, and unreliable for judging any single output.

Why does it matter?

Benchmarks drive research, and they get used up. Models passed the human baseline on GLUE within about a year of its release, prompting harder successors. Large models are also trained on so much web text that test questions may have leaked into training, inflating scores. Good practice today combines automatic metrics, fresh held-out data, and careful human review of real outputs.

Where NLP is used

The hard problems left are not the ones that make headlines. Most of the world’s roughly 7,000 languages have little digital text, so models serve them poorly. Models reproduce social biases from their data. And fluent output is not the same as correct output, which is the subject of the next lessons.

Key takeaways

• Different NLP tasks need different metrics: F1 for classification, BLEU or learned metrics for generation, perplexity for language models.
• Overlap metrics like BLEU miss valid paraphrases; they are only meaningful averaged over large test sets.
• Benchmarks saturate and leak into training data, so fresh test sets and human review remain essential.

Check your understanding

Seven scenarios from real NLP work. Each explanation adds something the lesson only touched on.

Your support team wants to route incoming emails to the right department. You have 50,000 labelled historical emails and need something that runs in milliseconds on a cheap server and that you can explain to an auditor. What is the most sensible first model?

References

Papers and books cited in this lesson. For a single place to go deeper, Jurafsky and Martin’s free textbook covers everything here and much more.

Speech and Language Processing is the standard reference. Next, continue with Machine Translation or jump to Transformers and Attention.

Sources cited

1. [1]

   The Winograd Schema Challenge

   Levesque, Davis & Morgenstern, 2012

   KR 2012. Pronoun puzzles like the trophy and the suitcase that need world knowledge, not grammar, to resolve.

2. [2]

   Speech and Language Processing (3rd edition draft)(opens in a new tab)

   Jurafsky & Martin, 2025

   The standard NLP textbook, free online. Chapters on n-grams, naive Bayes, embeddings, RNNs and transformers.

3. [3]

   Neural Machine Translation of Rare Words with Subword Units(opens in a new tab)

   Sennrich, Haddow & Birch, 2016

   ACL 2016. Adapted byte-pair encoding, a 1994 compression trick, into the subword tokenizer used by most language models.

4. [4]

   Language Model Tokenizers Introduce Unfairness Between Languages(opens in a new tab)

   Petrov, La Malfa, Torr & Bibi, 2023

   NeurIPS 2023. The same text can need up to 15 times more tokens in one language than another.

5. [5]

   A statistical interpretation of term specificity and its application in retrieval(opens in a new tab)

   Spärck Jones, 1972

   Journal of Documentation. Introduced inverse document frequency: rare words are more informative.

6. [6]

   Distributional structure(opens in a new tab)

   Harris, 1954

   Word 10(2-3). Early statement that words with similar meanings occur in similar contexts.

7. [7]

   A synopsis of linguistic theory, 1930-1955

   Firth, 1957

   In Studies in Linguistic Analysis (Blackwell). Source of “You shall know a word by the company it keeps.”

8. [8]

   Efficient Estimation of Word Representations in Vector Space(opens in a new tab)

   Mikolov, Chen, Corrado & Dean, 2013

   The word2vec paper: skip-gram and CBOW learn word vectors from a 1.6 billion word corpus in under a day.

9. [9]

   Linguistic Regularities in Continuous Space Word Representations(opens in a new tab)

   Mikolov, Yih & Zweig, 2013

   NAACL 2013. Showed analogies like king − man + woman ≈ queen as vector arithmetic.

10. [10]

    Neural Word Embedding as Implicit Matrix Factorization(opens in a new tab)

    Levy & Goldberg, 2014

    NeurIPS 2014. Skip-gram with negative sampling implicitly factorises a shifted PMI matrix, linking word2vec to count methods.

11. [11]

    Visualizing Data using t-SNE(opens in a new tab)

    van der Maaten & Hinton, 2008

    Journal of Machine Learning Research 9. The standard method for drawing high-dimensional embeddings in 2D.

12. [12]

    Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings(opens in a new tab)

    Bolukbasi, Chang, Zou, Saligrama & Kalai, 2016

    NeurIPS 2016. Embeddings trained on news text absorb gender stereotypes.

13. [13]

    Thumbs up? Sentiment Classification using Machine Learning Techniques(opens in a new tab)

    Pang, Lee & Vaithyanathan, 2002

    EMNLP 2002. Naive Bayes, maximum entropy and SVMs on film reviews, with negation tagging.

14. [14]

    Learning long-term dependencies with gradient descent is difficult(opens in a new tab)

    Bengio, Simard & Frasconi, 1994

    IEEE Transactions on Neural Networks. The vanishing and exploding gradient problem in recurrent networks.

15. [15]

    Long Short-Term Memory(opens in a new tab)

    Hochreiter & Schmidhuber, 1997

    Neural Computation. Gated memory cells that let gradients flow across long sequences.

16. [16]

    Neural Machine Translation by Jointly Learning to Align and Translate(opens in a new tab)

    Bahdanau, Cho & Bengio, 2014

    Introduced attention for translation, letting the decoder look back at every input word.

17. [17]

    Attention Is All You Need(opens in a new tab)

    Vaswani et al., 2017

    NeurIPS 2017. The transformer: attention without recurrence, trained in parallel.

18. [18]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(opens in a new tab)

    Devlin, Chang, Lee & Toutanova, 2019

    NAACL 2019. Pretrain once on unlabelled text, fine-tune cheaply for each task.

19. [19]

    BLEU: a Method for Automatic Evaluation of Machine Translation(opens in a new tab)

    Papineni, Roukos, Ward & Zhu, 2002

    ACL 2002. Clipped n-gram precision with a brevity penalty.

20. [20]

    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding(opens in a new tab)

    Wang, Singh, Michael, Hill, Levy & Bowman, 2018

    Nine language-understanding tasks in one leaderboard. Models passed the human baseline within about a year.