constitutional ai alignment

Constitutional AI trains a model to critique and revise its own outputs against a written set of principles, replacing a large share of the human labeling that RLHF normally requires with AI feedback instead. Bai and colleagues at Anthropic (2022) built the method specifically to address a bottleneck in standard RLHF: training a good reward model needs a large volume of human preference comparisons, and having people repeatedly read harmful or borderline content to label it is itself costly and unpleasant — Constitutional AI replaces most of that human labeling with a model judging its own outputs against an explicit, written constitution instead.

The method runs in two stages, and the first stage is where the model teaches itself to self-correct. In the supervised stage, the model generates a response, is then prompted to critique that response against a specific principle from the constitution, and finally revises its own answer in light of that critique — producing a cleaned-up response through a process the model runs on itself rather than one a human reviewer runs on it. This self-critique-and-revise loop is repeated across many principles and many examples, and the resulting (prompt, revised-response) pairs become supervised fine-tuning data, giving the model early exposure to its own corrected behavior before any preference training even begins.

The second stage replaces human preference labeling with AI preference labeling, which is the step that gives the method its other name: RLAIF. Instead of having human raters compare pairs of model responses and mark which one is better — the expensive, slow step at the heart of ordinary RLHF — Constitutional AI has a separate instance of the model make that comparison, judging each pair against the same written constitution. Those AI-generated preference judgments train a reward model exactly the way human preference judgments would, and the policy is then optimized against that reward model with the same reinforcement learning machinery RLHF already uses — the architecture downstream of the preference data is unchanged; only the source of the preference labels is different.

The constitution being explicit and written down is the feature that makes the method auditable in a way RLHF's implicit preference signal isn't. Ordinary RLHF trains a reward model on thousands of individual human comparisons, and what that reward model has actually learned to value is only indirectly inspectable — you can probe its outputs on test cases, but there's no single document stating the principles it's supposed to be enforcing. Constitutional AI's constitution is a readable list of principles — drawing partly on sources like the UN Declaration of Human Rights, plus Anthropic's own judgment about helpful and harmless behavior — that anyone can read, critique, and propose changes to, which makes disagreements about the model's values a debate about specific written text rather than an attempt to reverse-engineer an opaque reward model's implicit judgments.

Constitutional AI measurably improved the standard RLHF tradeoff between harmlessness and evasiveness, rather than just making harmlessness training cheaper. A common failure mode of safety-trained models is evasiveness — refusing or deflecting borderline-but-legitimate questions because the safety training couldn't distinguish them from genuinely harmful ones, trading away helpfulness for an easy reduction in harm. Anthropic reported that Constitutional AI models were both more harmless and less evasive than RLHF models trained without it, which matters because it shows the self-critique process isn't just a cheaper way to reach the same harmlessness-helpfulness tradeoff — explicit principles the model can reason about explicitly, rather than only an implicit pattern learned from aggregated human preference, produced a better operating point on that tradeoff, not just a cheaper one.

The method still depends entirely on the quality of the constitution and the model's own judgment applying it, which is a different trust problem than RLHF's, not a strictly easier one. Replacing human labelers with an AI judge means the entire system is only as good as that judge's ability to correctly apply the written principles — if the model's self-critique step is itself flawed or can be fooled the same way its original output could be, the critique-and-revise loop can reinforce rather than correct the error. This is why Constitutional AI is typically described as reducing dependence on direct human labeling of harmful content, not eliminating human judgment from the alignment process altogether — a human still writes the constitution, still evaluates whether the resulting model actually behaves well, and still intervenes when the self-supervision loop produces an unsatisfying result.

Read Constitutional AI as moving the human judgment upstream rather than removing it: instead of a person reading and rating thousands of individual harmful-or-not outputs, a person writes and refines the principles once, and the model applies those principles to itself at a scale no team of human raters could match — trading a large amount of repetitive, unpleasant human labeling for a smaller amount of careful constitutional design, with the tradeoff that the entire system now inherits whatever gaps exist in both the written constitution and the model's ability to faithfully apply it.