next token prediction
Next-token prediction is the specific training objective behind GPT-style models: for every position in a training sequence, score how well the model's predicted probability distribution matched the actual token that came next, using cross-entropy loss, and average that score across the entire sequence. It needs no human labeling at all — any raw text already contains its own training signal, since the "label" for predicting a word is just whatever word actually follows it in the text — which is exactly how Radford and colleagues' GPT-2 (2019) and Brown and colleagues' GPT-3 (2020) trained on hundreds of billions of words scraped from the internet with no annotation effort beyond collecting the text itself. The autoregressive-model entry covers how this objective generates text one token at a time; this entry is about how training the model to do that actually works, and the two phases — training and generation — turn out to have a strikingly different computational shape.
Training is parallel and generation is sequential, purely because of what information is available at each point — and that asymmetry is the entire reason generating text is slower than training on it. Training on a 2,048-token sequence computes 2,048 separate next-token predictions and losses in a single forward pass, because the entire sequence — including every token the model is supposed to predict — is already sitting in memory and causal masking just prevents each position from peeking ahead; generating 2,048 new tokens, by contrast, requires 2,048 separate forward passes run one after another, because token 501 genuinely doesn't exist until token 500 has actually been produced and appended to the context. This single asymmetry is the root reason techniques like speculative decoding exist — they try to guess ahead and verify in parallel, specifically to claw back some of the parallelism that generation structurally lacks but training gets for free.
Cross-entropy loss, worked through with real small numbers, shows exactly what the model is being pushed to do at every single position. Suppose the true next word is "cat" in a toy 5-word vocabulary, and the model's softmax output assigns "cat" a probability of 0.4 — the cross-entropy loss for that one prediction is negative the natural log of 0.4, which comes out to about 0.916; if the model had instead assigned "cat" a probability of 0.9, the loss would drop to about 0.105, and if it had assigned "cat" only 0.01, the loss would spike to roughly 4.6. The training process is nothing more than adjusting every weight in the network, across every position in every training sequence, to push these per-token losses down — millions of tiny nudges toward "assign more probability to whatever token actually came next."
This single, simple objective — predict the next token, nothing more sophisticated than that — turned out to produce behavior nobody explicitly trained for once scaled up far enough. Brown and colleagues titled their 2020 GPT-3 paper "Language Models are Few-Shot Learners" precisely because a model trained on nothing but next-token prediction learned, as a side effect of scale, to perform new tasks from just a few examples given in its prompt, with no task-specific training at all; Wei and colleagues' 2022 paper "Emergent Abilities of Large Language Models" catalogued several such capabilities — multi-step arithmetic, certain reasoning patterns — that showed up only once model and training scale crossed certain thresholds, appearing as a discontinuous jump rather than a smooth improvement as scale increased.
Next-token prediction's real significance isn't the mechanism itself, which is almost embarrassingly simple — guess the next word, check if you were right, adjust — it's that this one objective, applied at the scale of GPT-3 and beyond, turned out to be a far richer training signal than its simplicity suggests, implicitly forcing the model to learn grammar, facts, and reasoning patterns as side effects of getting better and better at the one narrow thing it was actually asked to do.