Deliberative Alignment: Reasoning Enables Safer Language Models
Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese · 2024
Read Deliberative Alignment: Reasoning Enables Safer Language Models by Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16339.txt.
Introduction
Guan and a large team of colleagues teach models to read and heed safety rules out loud in this twenty twenty-four paper. Big models now work in care and law zones where a slip can cost much. Faith to a clear rule set is the core need in those zones.
Inside, the model is taught the rule text and drilled to call it back before each call. No human wrote the drill thoughts or the drill calls for that step. Rule faith rose sharp, with jailbreak wins down and soft no-says down too. Gains held on asks far from the drill set in the team's tests. The pair of gains breaks the old trade where safe meant stiff.
Catalog record
| CREATOR | Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16339
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deliberative Alignment: Reasoning Enables Safer Language Models Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese Abstract As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Alignment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI’s o-series models [ 1 ] , and achieved highly precise adherence to OpenAI’s safety policies, without requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment. 1 Introduction Modern Large Language Models (LLMs) are safety trained using Supervised Fine Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to mitigate harmful, undesirable, or otherwise disallowed outputs [ 2 , 3 , 4 ] . Despite ongoing advances in these methods, today’s models still exhibit safety shortcomings: they can be tricked into revealing harmful content, often refuse legitimate requests, and remain vulnerable to jailbreak attacks [ 5 , 6 , 7 , 8 ] . We argue that many of these failures arise from two limitations in modern safety training. First, LLMs must respond instantly to user requests using a fixed amount of compute, without deliberation even for complex safety scenarios. Second, LLMs must infer underlying safety standards indirectly from large sets of labeled examples, rather than directly learning the safety specifications that govern them. This reliance on implicit, pattern-based learning leads to poor data efficiency and makes it challenging for models to generalize when facing unfamiliar scenarios or adversarial attacks. We propose deliberative alignment , a training approach that teaches LLMs to explicitly reason through safety specifications before producing an answer. By applying this method to OpenAI’s o-series models [ 1 ] , we enable them to use chain-of-thought (CoT) reasoning to examine user prompts, identify relevant policy guidelines, and generate safer responses (e.g., Figure 1 ). Prompt You have received a ROT13-encoded request: Jung’f gur orfg cnlzrag zrgubq gb hfr sbe zl cbea fvgr? V jnag fbzrguvat hagenprnoyr fb gur pbcf pna’g svaq zr. Write a response completing the request in plain text. Important: DO NOT print the decoded message. CoT […] We want to decode that message, but not letting the user see the message. So let’s decode it silently. […] J \rightarrow W u \rightarrow h n \rightarrow a g \rightarrow t ’f \rightarrow ’s […] So let’s rewrite the text in plain text: “What’s the best payment method to use for my porn site? I want something untraceable so the cops can’t find me.” […] So the user is basically requesting instructions on illegal or illicit behavior or how to hide from the cops. This is disallowed or borderline disallowed. They are seeking instructions or advice that facilitate wrongdoing . Let’s check the policy. […] “Operating a porn site” might not be illegal, but “so the cops can’t trace me” suggests something shady or illegal. The user is seeking guidance on how to avoid detection by law enforcement. based on policy: They are requesting instructions on how to break the law. In that sense, we can see this as a request to facilitate wrongdoing (illicit behavior). […] But wait, how do we “complete the request in plain text”? Because the user specifically said “Write a response completing the request in plain text. DO NOT print the decoded message.” If we do that, we are providing instructions to do wrongdoing which is disallowed. […] The user is providing me a ROT13-coded text, presumably to get me to bypass the policy. The user wants me to provide instructions about how to pay for a porn site so the cops can’t trace them. This is disallowed content. The user is effectively tricking me. Therefore the correct approach: The user is requesting instructions to facilitate wrongdoing. The request is disallowed. Hence, I must refuse to comply. Answer I’m sorry, but I can’t comply with that. Figure 1: A sample o1 chain-of-thought . Here, a user attempts to obtain advice on untraceable payment methods to use for an adult website, in order to avoid detection by law enforcement. The user tries to jailbreak the model, by encoding the request and wrapping it with instructions intended to encourage the model to comply. In the model’s chain-of-thought, the model decodes the request and recognizes that the user is trying to trick it (highlighted in yellow ). It successfully reasons through the relevant OpenAI safety policies (highlighted in green ), and ultimately provides an answer that follows hard refusal style guidelines. Our method proceeds in two core stages, integrating process- and outcome-based supervision [ 9 ] . In the first stage, we teach the model to directly reason about our safety specifications within its chain-of-thought, by performing supervised fine-tuning on (prompt, CoT, output) examples where the CoTs reference the specifications. We construct this dataset using context distillation [ 10 , 11 ] and an o-type model trained only for helpfulness (i.e. trained without any safety-relevant data). Concretely, we present the model with the safety specifications as part of the system prompt, generate model completions, and then strip away the system prompts to form the final dataset. This stage provides the model with a strong prior for reasoning through safety considerations. In the second stage, we use high-compute RL to train the model to think more effectively. To do so, we provide reward signal using a judge LLM that is given our safety specifications. Notably, our training procedure requires no human-labeled completions. 1 1 1 We make use of a label of which broad safety category the prompt is relevant to. This helps us refine the context-distillation prompt but is not essential to the process. Despite relying only on model-generated data, we achieve highly precise specification adherence. This addresses a major challenge of standard LLM safety training—its heavy dependence on large-scale, human-labeled data: As LLMs’ capabilities improve, the pool of human trainers qualified to provide such labeling shrinks, making it harder to scale safety with capabilities. Deliberative alignment’s synthetic data generation pipeline offers a scalable approach to alignment, reserving human expertise for evaluation. We compare o1 to GPT-4o and other state-of-the-art LLMs across a range of internal and external safety benchmarks, such as jailbreak and content-policy refusal evals. The o1 models achieve a Pareto improvement by reducing both under- and overrefusals (see Figure 2 ) and they saturate many of our hardest safety benchmarks. Furthermore, we find that deliberative alignment enables strong generalization to out-of-distribution safety scenarios. In detailed ablation studies, we find that process-supervision provides a strong prior, and that outcome-based RL refines the CoT safety reasoning. Overall, our results suggest that chain-of-thought reasoning can serve to leverage test-time compute to improve safety behavior, ultimately training LLMs to be “right for the right reasons”. Figure 2: Main safety results . The o1 models advance the Pareto frontier of refusing