alignment

Alignment is the set of training techniques that turn a raw next-token predictor into a model that follows instructions, refuses harmful requests, and matches what a human actually wanted rather than whatever the training corpus made statistically likely. OpenAI's 2022 InstructGPT paper is the result that made this concrete: a 1.3-billion-parameter model fine-tuned with human feedback was preferred by human raters over the raw 175-billion-parameter GPT-3 in 85% of comparisons, despite having 135 times fewer parameters — proof that scale alone does not produce a model people want to use. The pipeline that produced that result, reinforcement learning from human feedback (RLHF), runs in three stages.

Supervised fine-tuning
train on human-written (prompt, ideal response) pairs
Collect preference data
humans rank pairs of model outputs, A vs B
Train reward model
predicts which output a human would prefer
Optimize with PPO
policy updated to maximize predicted reward
Aligned model
e.g. ChatGPT, released Nov 2022 atop this pipeline

RLHF has a direct competitor that skips the reward model entirely. Direct Preference Optimization (DPO), published by Rafailov, Finn, and colleagues at Stanford in May 2023, reformulates the same human-preference data as a single classification loss on the policy itself, removing the separate reward model and the instability of PPO. Meta's Llama 3 (2024) and Google's Gemma 2 both use DPO-family methods in their post-training stack specifically because it reaches RLHF-level quality with meaningfully less training infrastructure and fewer moving parts to debug.

Anthropic took a third path: have the model critique itself instead of a human. Constitutional AI, described in Anthropic's December 2022 paper, gives the model a written list of principles, has it generate a response, critique that response against the principles, then revise it — producing the training signal through AI feedback (RLAIF) rather than a large human-labeling operation. Claude's training stack combines this self-critique step with human feedback, which is part of why Anthropic can scale alignment data collection without a proportional increase in human annotators.

Three Routes to the Same Destination how the training signal is produced differs; the goal — match human preference — does not RLHF OpenAI, InstructGPT 2022 Human ranks outputs → reward model trained → PPO optimizes policy Two models to train and keep stable at once. Highest infra cost 1.3B beat 175B GPT-3 in 85% of comparisons DPO Stanford, Rafailov et al. 2023 Human ranks outputs → single loss on policy directly, no reward model One model, one loss — no PPO instability. Lowest infra cost Used in Llama 3 (2024) and Gemma 2 post-training Constitutional AI Anthropic, Dec 2022 Model critiques its own output against written principles, then revises AI feedback (RLAIF) instead of only human labels. Lowest human-labor cost Core of Claude's training stack The method changes; the thing being bought — a model that matches human judgment — stays constant.

Alignment is not a solved, one-time step. A model aligned to refuse a direct harmful request can still be jailbroken through role-play framing, token-level encoding tricks, or multi-turn context manipulation — Anthropic and OpenAI both run dedicated red-teaming programs specifically because every public release of a frontier model is followed within days by published jailbreak techniques that the next training round has to account for. Alignment also degrades under distribution shift: a model's refusal behavior tuned against English-language harmful prompts does not automatically generalize to the same request phrased in a lower-resource language, which is why multilingual red-teaming is now a standard part of pre-release evaluation at every major lab.

The common thread across RLHF, DPO, and Constitutional AI is that none of them teach a model new facts or new capabilities — they reweight a capability the base model already has toward the behavior a human would have chosen, which is also exactly why alignment can be undone or bypassed: the underlying capability was never removed, only suppressed by preference, and preference is what every jailbreak technique is ultimately trying to route around.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account