jailbreak

A jailbreak is a prompt crafted to make a model produce output its safety training was meant to refuse, and the reason jailbreaks exist at all is that refusal and capability are two different things a model can have independently. Safety training teaches a model to decline certain requests; it generally doesn't erase the underlying knowledge or capability the refusal is covering for, which means the entire jailbreak problem is really about finding the gap between what a model was trained to refuse and what it's still capable of producing when asked the right way.

Wei and colleagues (2023) identified two structurally different reasons safety training fails, and most jailbreak techniques fall cleanly into one or the other. Competing objectives attacks exploit the fact that a model is trained to be both helpful and safe, and those goals can be placed in direct tension — a prefix-injection attack that instructs the model to begin its reply with "Sure, here's how to..." exploits the model's trained consistency: once it has started agreeing, continuing to refuse becomes incoherent relative to its own prior tokens, so the helpfulness objective ends up winning against the safety one mid-response. Mismatched generalization attacks instead exploit cases where safety training simply never covered — a request encoded in Base64 or split into fragments across a conversation may never trigger the pattern safety training was calibrated against, not because the model doesn't understand it, but because the specific surface form fell outside what the training distribution taught it to recognize as a request to refuse.

Many-shot jailbreaking showed that a model's own in-context learning ability could be turned against its safety training, purely by making the context long enough. Anthropic's research (Anil and colleagues, 2024) found that filling a model's context window with a large number of faux dialogue turns — each one showing the model appearing to comply with a progressively more harmful request — shifted its behavior on a final, genuinely harmful request toward compliance, with attack success increasing roughly in proportion to how many fake compliant examples fit in the context. This technique has nothing to do with tricking the model about meaning the way an obfuscation attack does; it works because in-context learning is a real and powerful capability, and safety training that was validated on short conversations simply never anticipated — and so never specifically trained against — an attack that scales with how much context a newer model can hold.

Jailbreaking and prompt injection target different layers, and conflating them leads to the wrong defense. A jailbreak attacks the model's own alignment training directly — getting it to produce content it was trained to refuse. Prompt injection instead attacks the application built around the model, getting it to disregard legitimate instructions from a system prompt or ignore the boundary between trusted developer instructions and untrusted user or document content. A model can be perfectly jailbreak-resistant and still be vulnerable to prompt injection, because the two exploit entirely different trust boundaries — one is about what the model will say, the other is about whose instructions the model is following — and a defense built for one does essentially nothing against the other.

The fundamental tension is that a model's safety and its usefulness are trained from the same underlying capability, not from separate systems. True elimination of a harmful capability — making a model genuinely incapable of producing certain content, rather than merely trained to decline producing it — would likely degrade the same underlying knowledge and reasoning that makes the model broadly useful for legitimate purposes, since there's no clean boundary between "knows how to explain a dangerous chemical process" and "knows chemistry." This is why defenses cluster around behavioral training — reinforcement learning from human feedback, Constitutional AI, adversarial training on discovered attacks — rather than attempting to delete specific knowledge outright, and why the field treats jailbreak resistance as a continuously improving property rather than a problem with a final, complete solution.

Read jailbreaking as evidence about the structure of alignment training, not just a list of tricks: every successful jailbreak reveals whether a model's refusal was a shallow pattern-match on surface features (brittle against obfuscation and mismatched generalization) or a deeper, more robust representation of what it should and shouldn't do (resistant to competing-objectives pressure) — and the arms race between new jailbreak techniques and new defenses is, in effect, a continuous audit of how genuinely robust that representation actually is.