positional encoding

Self-attention computes a weighted sum over every token's value vector based purely on query-key similarity — a computation that would produce the exact same result if the entire input sentence were shuffled into a random order, because nothing in the mechanism itself depends on which position a token occupies. Positional encoding is whatever gets added back to give the model that missing sense of order, and the field has moved through three genuinely different approaches to it. Vaswani and colleagues' original 2017 Transformer used fixed sinusoidal functions — sine and cosine waves of different frequencies, built from a base value of 10,000, added directly onto each token's embedding — a scheme that requires no learning at all and in principle, though imperfectly in practice, allows some generalization to sequence lengths longer than anything seen during training.

Token embedding computed
no notion of position yet
Positional signal computed
sinusoidal wave, learned vector, rotation angle, or distance penalty
Added to the embedding
sinusoidal (fixed) or learned positional embeddings
Injected into attention itself
RoPE rotates Q/K; ALiBi biases the attention score directly
Self-attention is now order-aware
shuffling the input would change the output

Learned positional embeddings, used in BERT and GPT-2, trade the sinusoidal method's theoretical extrapolation for a simpler, trainable lookup table — at the cost of a hard ceiling. Instead of a fixed mathematical function, each position from 1 up to some maximum gets its own trainable vector, learned the same way a token embedding is learned; this works cleanly and often performs well in practice, but a model trained with a learned table of, say, 1,024 position vectors has no vector at all for position 1,025, and simply cannot process a longer sequence without retraining or a separate extrapolation trick — a hard limit the sinusoidal and later approaches were specifically designed to avoid.

RoPE encodes position by rotating the query and key vectors themselves, rather than adding anything to the embedding at all. Su and colleagues' 2021 paper "RoFormer: Enhanced Transformer with Rotary Position Embedding" rotates each query and key vector by an angle proportional to its position before the attention dot product is computed, and because rotating two vectors by different amounts changes their dot product in a way that depends only on the difference between their two rotation angles, the resulting attention score naturally encodes relative distance between tokens rather than their absolute positions — a property that falls directly out of the rotation math rather than being engineered in separately. Llama, Mistral, and Qwen all use RoPE, which is now the most common choice in frontier open-weight models.

Three Places to Inject Position same goal, three different points in the pipeline Sinusoidal Vaswani et al., 2017 Added directly to the token embedding, before attention even runs Fixed, no training needed Weak extrapolation in practice Original Transformer RoPE Su et al., 2021 Rotates query and key vectors by position before their dot product is taken Relative position emerges naturally from the rotation Llama, Mistral, Qwen ALiBi Press et al., 2021 No embedding at all — subtracts a distance-based penalty from attention scores "Train short, test long" — best extrapolation of the three BLOOM, MPT Moving position further into the attention computation itself is the trend driving every newer method.

ALiBi skips embeddings entirely and solves the problem at the other end of the pipeline, directly inside the attention score. Press and colleagues' 2021 paper, titled plainly "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation," subtracts a penalty proportional to the distance between two tokens straight from their raw attention score before the softmax runs — no positional vector of any kind ever touches the token embeddings — and the paper's own headline claim held up in practice: models trained on comparatively short sequences using ALiBi extrapolated to meaningfully longer sequences at inference time better than sinusoidal or learned alternatives, which is exactly why BLOOM and MPT adopted it.

Extending RoPE to handle longer contexts than it was trained on has become its own small research area, because RoPE's relative-position math degrades gracefully but not perfectly past its training length. Simple linear scaling stretches the position values down before applying the rotation, NTK-aware scaling adjusts the underlying frequency base instead of the raw positions, and Peng and colleagues' 2023 YaRN method combines interpolation with an attention-scaling correction to extend context windows with less quality loss than either simpler technique alone — the existence of this entire sub-literature is itself evidence that no positional encoding scheme, including RoPE, fully solved length extrapolation on its own.

The three approaches disagree about where in the pipeline position should be injected — into the embedding, into the attention score's inputs, or directly into the attention score itself — but they're all answers to the identical problem self-attention creates by design: an operation with no built-in sense of order needs that sense supplied from somewhere, and which "somewhere" a given model chooses turns out to be one of the more consequential architectural decisions behind how well it generalizes to sequences longer than it was ever trained on.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account