Machine Translation

Why translation is hard

Every day, machine translation lets a nurse read a patient's note in another language, a small shop sell abroad, and a refugee fill in a form. It is one of the oldest goals in AI and one of the most used. It is also a problem that looks simple, swap the words, and turns out to be about as hard as understanding language itself.

What is it?

Machine translation (MT) automatically converts text in a source language into text in a target language that means the same thing and reads naturally. “Means the same thing” carries the difficulty. Languages do not just use different words; they divide up meaning differently, order it differently and leave different things unsaid.

How does it go wrong?

Linguists describe the correspondence between a sentence and its translation as an alignment: which words came from which. The explorer below shows hand-annotated alignments for five sentence pairs, each chosen to expose one reason word-for-word translation fails.

Five pairs, five different problems. Word order reverses whole phrases. Morphology and syntax split one word into two, far apart. Some words have no counterpart at all, while others must be invented (Japanese needs the particles は and を that English expresses by position). Idioms only align as whole phrases. And ambiguity means the right word depends on context that may be several words, or several sentences, away. Gender, politeness levels and tense systems add more of the same: the target language may demand information the source never gave.

Why does it matter?

These problems explain the whole history of the field. Each generation of MT, rules, statistics, phrases, neural networks, attention, large language models, can be read as a better answer to the question the explorer poses: how do we find the right correspondence, and how much context do we need to find it?

Key takeaways

• Translation must preserve meaning, not words: languages differ in word order, morphology, idiom and what they leave implicit.
• An alignment records which source words produced which target words; crossing links mean reordering.
• Ambiguity is resolved by context, sometimes far away, which is why better MT has always meant using more context.

From rules to statistics

The first public MT demonstration, in January 1954, translated Russian sentences into English on an IBM 701. Newspapers predicted the problem would be solved within a few years. It took rather longer.

Rule-based translation

What is it?

The Georgetown-IBM demonstration used a vocabulary of about 250 words and six grammar rules on carefully chosen sentences. Its successors, rule-based MT systems, scaled the idea up: bilingual dictionaries with hundreds of thousands of entries, and grammars written by linguists that analyse the source sentence and generate the target. Systems such as SYSTRAN were used for decades, including by the European Commission.

Rules were precise and predictable but brittle. Every idiom, exception and new domain needed another rule, and rules interacted in ways nobody could foresee. A 1966 US government review, the ALPAC report, concluded that MT was slower, less accurate and more expensive than human translation, and funding in the US largely dried up for a decade.

Statistical translation

How does it work?

In the late 1980s a group at IBM, trained in speech recognition rather than linguistics, tried something radical: ignore grammar and learn translation from data. The Canadian parliament publishes its proceedings in English and French, sentence by sentence, giving millions of parallel sentence pairs. Brown and colleagues (1993) framed translation as a probabilistic model and learned it from such text.

The catch is that parallel text tells you which sentences correspond, not which words. The alignments you explored above are hidden. The IBM models solve this with the expectation-maximisation (EM) algorithm. IBM Model 1, the simplest, assumes each English word was generated by some foreign word in the same sentence with probability t(e | f), and knows nothing else, not even word order.

1. Start with every t(e | f) equal: no idea which word means what.
2. E-step: for each English word, share one unit of “credit” among the foreign words of its sentence, in proportion to the current t(e | f).
3. M-step: add up the credit each foreign word received for each English word across the whole corpus and renormalise into new probabilities.
4. Repeat. Words that consistently co-occur pull probability towards each other.

After one iteration “das” already leans towards “the”, because it appears twice with “the”. That in turn explains away “the” in “das Haus / the house”, so “Haus” is left to claim “house”. This explaining-away cascades, and the log-likelihood rises at every step, as EM guarantees. No one told the model a single word pair. Try the Spanish corpus, where “verde” is learned from two sentences that share only it and “green”.

Why does it matter?

The statistical turn replaced hand-written knowledge with learning from data, the same move that later defined machine learning in general. Higher IBM models added word order, fertility (one word producing several) and more; the alignments they found became the raw material for the next generation of systems.

Key takeaways

• Rule-based MT encoded dictionaries and grammar by hand: precise but brittle and expensive to extend.
• Statistical MT learned translation probabilities from parallel text, treating word alignments as hidden variables.
• IBM Model 1 uses EM: soft-align with current probabilities, re-estimate from those alignments, repeat.

Phrase-based translation

Word-based models could never learn that “raining cats and dogs” is one unit. The fix, obvious in hindsight, was to translate chunks.

What is it?

Phrase-based statistical MT segments the source sentence into phrases (any contiguous word sequence, not necessarily a linguistic phrase), translates each phrase using a phrase table learned from word-aligned parallel text, and reorders the results. It dominated MT from the mid-2000s until 2016 and powered Google Translate and the open-source Moses toolkit.

How does it work?

The system scores candidate translations with a weighted combination of models. The core idea goes back to the IBM work and is called the noisy channel: pretend the foreign sentence f is a garbled version of an English sentence e, and search for the e that best explains it.

The translation model p(f | e) checks faithfulness; the language model p(e) checks fluency. Real systems added reordering, length and phrase-count features and tuned their weights on held-out data.

Splitting the problem this way is powerful because the two parts learn from different data. The translation model needs parallel text, which is scarce; the language model needs only target-language text, which is abundant. A decoder then searches the enormous space of segmentations, phrase choices and orderings with beam search, keeping only the most promising partial translations at each step.

Why does it matter?

Phrases captured local context, idioms and short reorderings, and phrase-based systems made MT genuinely useful to millions of people. But they had clear limits: phrases are translated largely independently, long-distance reordering (like the German verb at the end of a sentence) is hard, and the pipeline of separately trained components was complex. Output was often locally fluent but globally disjointed.

Key takeaways

• Phrase-based MT translates contiguous chunks using a phrase table extracted from word alignments, then reorders them.
• The noisy-channel view splits the job into faithfulness (translation model) and fluency (language model).
• It handled local context well but struggled with long-range reordering and whole-sentence coherence.

Neural encoder-decoders

In 2014 a single neural network, trained end to end on sentence pairs with no phrase tables, alignments or hand-built features, matched the best phrase-based systems. The pipeline collapsed into one model.

What is it?

A sequence-to-sequence (seq2seq) model has two recurrent networks. The encoder reads the source sentence word by word and updates a hidden state; its final state is a fixed-size vector, often called the context or “thought” vector. The decoder starts from that vector and generates the translation one word at a time, each step conditioned on the words produced so far.

How does it work?

Words enter as learned embeddings. The model is trained to maximise the probability of the reference translation, word by word, and at inference time beam search keeps the few most probable partial outputs. Sutskever, Vinyals and Le (2014) used deep LSTMs and reached 34.8 BLEU on WMT14 English-French. They also found a curious trick: feeding the source sentence in reverse made training much easier, because the first source words were then close to the first target words they had to produce.

Why does it matter?

Seq2seq showed that translation could be learned as one differentiable function, and the same recipe soon worked for summarisation, speech recognition and dialogue. But the reversal trick hinted at the weakness: squeezing a 50-word sentence into one vector of a few hundred numbers loses information, and quality fell off on long sentences. The fix would change the field.

Key takeaways

• Seq2seq uses an encoder RNN to compress the source and a decoder RNN to generate the target, trained end to end.
• It matched phrase-based systems without any hand-built components.
• The single fixed-size context vector is a bottleneck that hurts long sentences.

The attention breakthrough

A human translator does not memorise a whole sentence and then write the translation from memory. They glance back at the relevant part of the source as they go. Attention gives a neural network that same ability.

What is it?

Bahdanau, Cho and Bengio (2014) kept every encoder hidden state instead of only the last one. At each decoding step the decoder computes a fresh context vector as a weighted average of all encoder states. The weights, the attention, say which source words matter for the word being produced right now.

How does it work?

For decoder step i and source position j, a small network scores how well they match, giving a score e_ij. A softmax turns the scores for step i into weights α_ij that are positive and sum to 1. The context is the weighted sum of encoder states h_j.

Attention in three lines. The whole mechanism is differentiable, so the model learns where to look purely from the translation objective.

Look at the block in the middle. “European Economic Area” becomes “zone économique européenne”, and the bright cells run against the diagonal: the model has learned the adjective reordering you traced by hand in the first section. “a été” spreads its attention over “was” and “signed”, a soft many-to-one alignment that the IBM models would have had to force into a hard choice. Nobody supervised these alignments; they fall out of learning to translate.

Why does it matter?

Attention removed the bottleneck: quality no longer collapsed on long sentences. It also made models partly inspectable, since attention maps often look like alignments (though they are not guaranteed to be faithful explanations). Most importantly, it planted the idea that would take over all of AI three years later: if attention is this useful, perhaps it is all you need.

Key takeaways

• Attention builds a new context vector for every output word as a softmax-weighted average of all encoder states.
• The weights act as a learned soft alignment, discovered without any alignment supervision.
• It fixed the long-sentence bottleneck and led directly to the Transformer.

Transformers take over

The Transformer, the architecture behind every large language model today, was introduced in a paper about machine translation.

What is it?

Vaswani and colleagues (2017) dropped recurrence entirely. The encoder and decoder are stacks of attention layers: self-attention, where every word in a sentence attends to every other word in the same sentence, and cross-attention, where the decoder attends to the encoder, exactly as in Bahdanau's model. Positional encodings tell the model about word order, since attention alone does not see it.

How does it work (and why is it better)?

An RNN must process word 1 before word 2 before word 3. Self-attention compares all pairs of positions at once, which suits GPUs and lets training scale to far more data. It also shortens the path between distant words: the German verb at the end of a sentence can attend directly to its subject at the start. The original Transformer reached 28.4 BLEU on WMT14 English-German, better than all previous models, at a fraction of the training cost.

Two other ingredients made neural MT practical. Subword units such as byte-pair encoding split rare words into frequent pieces, so the model can translate names and compounds it has never seen whole. And multilingual training showed that one model can translate between many languages, sharing what it learns across them.

Why does it matter?

Neural MT was already reaching users: Google's 2016 production system, an LSTM with attention, cut translation errors by an average of 60% against the phrase-based system in human side-by-side comparisons. Transformers then became the default for MT, and the same architecture, scaled up and trained to predict the next word, became the large language model.

Key takeaways

• The Transformer replaced recurrence with self-attention and cross-attention, and was introduced for translation.
• Parallel computation and short paths between distant words made it faster to train and better at long-range structure.
• Subword tokenization and multilingual training made neural MT robust to rare words and able to share across languages.

Measuring translation: BLEU and beyond

There is rarely one correct translation. Ten professionals will produce ten different, equally good versions of the same paragraph. So how do you score a machine automatically, thousands of times a day, during development?

What is it?

BLEU (bilingual evaluation understudy, 2002) compares a system's output with one or more human reference translations by counting shared word sequences. It became the field's standard number because it is cheap, fast and, averaged over a whole test set, correlated reasonably with human judgement for the systems of its time.

How does it work?

1. For n = 1 to 4, count the hypothesis n-grams that also appear in a reference. Clip each count at the maximum number of times that n-gram appears in any single reference, so “the the the the” cannot score by repetition.
2. Divide by the total number of hypothesis n-grams to get the modified precisions p1 to p4.
3. Take their geometric mean, so all four orders must be non-zero.
4. Multiply by a brevity penalty, BP = exp(1 − r / c) when the output length c is shorter than the reference length r, because precision alone would reward saying very little.

The presets are designed to embarrass BLEU. The good paraphrase, which a human would accept, scores 0 because it shares no 3-gram with the reference (turn on smoothing and it gets about 26). The translation that flips the meaning with “not” scores 50, far higher than the correct paraphrase. The word salad has perfect unigram precision but nothing else. Plural forms lose most of their n-gram matches even though only two word endings changed; chrF, which compares character n-grams and weights recall more heavily, gives them substantial credit. Real evaluations use BLEU at the corpus level, where the zero problem mostly disappears, but the blindness to meaning does not.

Why does it matter?

What you measure is what you optimise. As systems improved, BLEU's correlation with human preference weakened. COMET and similar learned metrics, which embed the source, output and reference with a pretrained multilingual model, track human judgements much more closely, and a large study with professional translators found that crowd-sourced ratings could even rank human translations below machine output, while expert error annotation did not. Today's evaluation campaigns use expert human annotation as the final word and learned metrics for day-to-day development, with BLEU kept mainly for comparison with older work.

Key takeaways

• BLEU combines clipped n-gram precisions (n = 1 to 4) by geometric mean and multiplies by a brevity penalty.
• It rewards surface overlap, so it can score a meaning-flipping error above a correct paraphrase.
• chrF handles morphology better; learned metrics like COMET and expert MQM annotation track real quality far more closely.

LLMs, low-resource languages and bias

Translation has now been absorbed into general-purpose language models. The best systems handle whole documents and follow instructions about tone and terminology. Yet for most of the world's 7,000 or so languages, and for questions of fairness, the problem is far from solved.

What changed with LLMs?

Large language models learn translation as a by-product of training on multilingual text, then get refined with instruction tuning. At the WMT24 shared task, the main annual MT evaluation, organisers evaluated eight general LLMs alongside dedicated systems and titled their findings “The LLM era is here but MT is not solved yet”. By WMT25, the top systems did so well on ordinary text that the organisers deliberately built harder, document-level test sets.

LLMs bring real advantages: they use context across a whole document (consistent terminology, resolving “it” to the right noun), they can follow instructions (“formal register, keep product names in English”), and they are good at idioms. They also bring new failure modes: adding content that was never in the source, silently dropping sentences, or answering a question in the text instead of translating it.

Low-resource languages

How do we translate languages with little data?

Neural models are hungry for parallel text, and for most languages there is very little. Meta's No Language Left Behind project (2022) built one model for 200 languages by mining parallel sentences from the web, sharing parameters across related languages, and creating human-translated evaluation sets for languages that had none. It improved quality by 44% in relative BLEU over the previous state of the art. In June 2024 Google Translate added 110 languages, covering about 614 million speakers, with help from its PaLM 2 model.

Coverage is not the same as quality. For low-resource languages, test sets are small, automatic metrics are less reliable, and models are more likely to hallucinate or fall back into a related high-resource language. Speakers of those languages are also the people least able to check the output.

Gender bias and other failures

Why does it matter?

When the source language does not mark something the target language requires, the model must guess, and it guesses from training statistics. Turkish, Finnish and Hungarian use gender-neutral pronouns; translated into English, systems have historically produced “he is a doctor” and “she is a nurse”. A 2019 study built a challenge set of sentences where gender is determined by context and found every tested commercial and academic system significantly prone to gender-biased errors, performing much better when the correct gender matched the stereotype. In 2018 Google began showing both feminine and masculine translations for some ambiguous queries.

Key takeaways

• LLMs now rank among the strongest translators and bring document context and instructions, along with new errors like additions and omissions.
• Projects like NLLB-200 extend coverage to hundreds of languages, but quality and evaluation lag far behind for low-resource languages.
• When the source is ambiguous, models fall back on training statistics, reproducing gender and other stereotypes.

Check your understanding

Seven scenarios from real translation work. The explanations add detail the lesson only touched on.

A 1990s rule-based system translates "The spirit is willing but the flesh is weak" word by word and produces something about strong vodka and rotten meat. Which property of language is it failing to handle?

References

Three papers tell most of the neural story: Sutskever et al. for the encoder-decoder, Bahdanau et al. for attention, and Vaswani et al. for the Transformer. For the statistical era, Philipp Koehn's textbook Statistical Machine Translation (2010) remains the clearest account, and the NLP lesson covers the tokenization and embedding groundwork.

References

1. [1]

   The Georgetown-IBM Experiment Demonstrated in January 1954(opens in a new tab)

   Hutchins, W. J., 2004

   A history of the first public machine translation demonstration and the optimism it created.

2. [2]

   The Mathematics of Statistical Machine Translation: Parameter Estimation(opens in a new tab)

   Brown, P. F., Della Pietra, S. A., Della Pietra, V. J., Mercer, R. L., 1993

   Computational Linguistics 19(2). Introduces IBM Models 1 to 5, learning word alignments from sentence-aligned text with EM.

3. [3]

   Statistical Phrase-Based Translation(opens in a new tab)

   Koehn, P., Och, F. J., Marcu, D., 2003

   NAACL 2003. Shows that translating multi-word phrases extracted from word alignments beats word-based models.

4. [4]

   Sequence to Sequence Learning with Neural Networks(opens in a new tab)

   Sutskever, I., Vinyals, O., Le, Q. V., 2014

   Deep LSTM encoder-decoder; 34.8 BLEU on WMT14 English-French; reversing the source sentence helps.

5. [5]

   Neural Machine Translation by Jointly Learning to Align and Translate(opens in a new tab)

   Bahdanau, D., Cho, K., Bengio, Y., 2014

   Introduces attention: the decoder computes a fresh weighted average of encoder states at every step (ICLR 2015).

6. [6]

   Attention Is All You Need(opens in a new tab)

   Vaswani, A., Shazeer, N., Parmar, N., et al., 2017

   The Transformer, introduced as a translation model: 28.4 BLEU on WMT14 English-German.

7. [7]

   Neural Machine Translation of Rare Words with Subword Units(opens in a new tab)

   Sennrich, R., Haddow, B., Birch, A., 2016

   Byte-pair encoding for open-vocabulary translation.

8. [8]

   Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation(opens in a new tab)

   Wu, Y., Schuster, M., Chen, Z., et al., 2016

   Production neural MT at Google; reduces translation errors by an average of 60% compared with the phrase-based system in human side-by-side evaluations.

9. [9]

   BLEU: a Method for Automatic Evaluation of Machine Translation(opens in a new tab)

   Papineni, K., Roukos, S., Ward, T., Zhu, W.-J., 2002

   Clipped n-gram precision with a brevity penalty; the standard MT metric for two decades.

10. [10]

    chrF: character n-gram F-score for automatic MT evaluation(opens in a new tab)

    Popovic, M., 2015

    Character n-gram F-score; more forgiving of morphological variation than word-level BLEU.

11. [11]

    COMET: A Neural Framework for MT Evaluation(opens in a new tab)

    Rei, R., Stewart, C., Farinha, A. C., Lavie, A., 2020

    A learned metric trained to predict human quality judgements from source, hypothesis and reference.

12. [12]

    Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation(opens in a new tab)

    Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., Macherey, W., 2021

    Professional MQM error annotation; finds crowd ratings unreliable and learned embedding metrics better than BLEU.

13. [13]

    Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet(opens in a new tab)

    Kocmi, T., Avramidis, E., Bawden, R., et al., 2024

    11 language pairs; participating systems evaluated alongside 8 LLMs and 4 online providers by professional annotators.

14. [14]

    Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets(opens in a new tab)

    Kocmi, T., Artemova, E., Avramidis, E., et al., 2025

    30 language pairs, 60 systems; deliberately harder, document-level test sets.

15. [15]

    No Language Left Behind: Scaling Human-Centered Machine Translation(opens in a new tab)

    NLLB Team, Costa-jussa, M. R., Cross, J., et al., 2022

    A single model for 200 languages, with a 44% relative BLEU improvement over the previous state of the art on FLORES-101.

16. [16]

    Google Translate adds 110 languages in its biggest expansion yet(opens in a new tab)

    Google, 2024

    June 2024: 110 new languages, about 614 million speakers, added with help from the PaLM 2 model.

17. [17]

    Evaluating Gender Bias in Machine Translation(opens in a new tab)

    Stanovsky, G., Smith, N. A., Zettlemoyer, L., 2019

    WinoMT challenge set; all tested commercial and academic systems were significantly prone to gender-biased translation errors.

18. [18]

    Reducing gender bias in Google Translate(opens in a new tab)

    Kuczmarski, J., 2018

    Google begins showing both feminine and masculine translations for gender-neutral queries in some languages.