machine translation quality

Machine translation models the probability of a target-language sentence given a source-language sentence, and the entire architectural history of the field is about how to represent that source sentence so the model can draw on the right parts of it at the right moments while generating the translation word by word. Before 2014, the dominant approach was statistical machine translation, built from phrase tables and separately-trained language models stitched together with hand-tuned weights. Neural machine translation replaced that whole pipeline with a single network trained end to end, and within about three years had made statistical MT obsolete for nearly every major language pair.

Sequence-to-sequence with attention was the architecture that made neural translation work at all. Sutskever and colleagues (2014) showed that an encoder-decoder pair of LSTMs — one network reading the source sentence into a fixed-length vector, another generating the target sentence from it — could translate competitively, but compressing an entire sentence into one fixed-size vector was a hard bottleneck that degraded badly on longer sentences. Bahdanau and colleagues (2014) fixed this with attention: instead of relying on one compressed vector, the decoder computes, at every output step, a weighted combination over all of the encoder's hidden states, learning which source words matter most for generating the current target word — a soft, learned alignment that replaced the explicit word-alignment tables statistical MT had relied on.

The Transformer removed recurrence entirely and made translation models dramatically more parallelizable. Vaswani and colleagues (2017) replaced the encoder and decoder's recurrent LSTMs with self-attention layers, which let every position in a sequence attend directly to every other position in a single operation rather than passing information step by step through a chain of recurrent states. Because self-attention has no sequential dependency during training, all positions can be computed simultaneously on a GPU instead of one timestep at a time, and that parallelism — not raw accuracy alone — is what let translation models scale to far larger sizes and datasets than any RNN-based architecture had made practical, and the Transformer has been the dominant NMT architecture ever since.

Subword tokenization is what lets a translation model handle words it has never seen whole. Sennrich and colleagues (2016) applied byte-pair encoding to translation, splitting rare and unseen words into sequences of common subword pieces that the model has seen plenty of — "untranslatability" might become "un" + "translat" + "ability" — so the model never hits a true out-of-vocabulary word, only an unusual combination of familiar pieces. Sharing one subword vocabulary across the source and target language, when the two languages share enough script and morphology, also lets the model transfer statistical strength between related words across languages rather than treating every language's vocabulary as fully separate.

Back-translation turns an abundant resource — monolingual text — into scarce parallel training data. Also from Sennrich and colleagues (2016), the technique trains a translation model in the reverse direction (target language to source), uses that reverse model to translate large amounts of ordinary monolingual target-language text back into the source language, and then adds those synthetic source-target pairs to the forward model's training data. The synthetic source side is imperfect, but the target side is genuine, fluent, native text, which turns out to improve translation quality substantially — a particularly large win for low-resource language pairs where genuine parallel corpora are thin but monolingual text in the target language is comparatively abundant.

A single model can now translate between languages it never saw paired together during training. Google's multilingual NMT system (Johnson and colleagues, 2017) trained one shared encoder-decoder across many language pairs at once, simply prepending a target-language tag to the input, and found the model could translate directly between two languages it had only ever seen paired with a third — zero-shot translation, enabled by the shared representation space the multilingual training produces. Meta's No Language Left Behind (2022) pushed multilingual coverage much further, supporting around two hundred languages including many with very little digitized text, using language-specific routing within a mixture-of-experts architecture to give low-resource languages dedicated model capacity rather than drowning them out during training on high-resource pairs.

Read machine translation's history as a running answer to one question — how much of the source sentence should the model look at, and how, when generating each word of the target — and each major advance changed that answer: fixed-vector encoding looked at a compressed summary, attention looked at a learned weighted combination of every source position, and the Transformer made computing that combination for every position in parallel cheap enough to scale into the large multilingual systems translating hundreds of languages today.