Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation
Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris · 2025
Read Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation by Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21784.txt.
Introduction
Tharindu Kumarage and a large team of co-authors wrote this 2025 paper on teaching language models to reason about safety rules before answering. Models that deliberate over policy first resist jailbreaks and refuse less blindly, but training them needs large sets of policy-laced reasoning traces, which are expensive and error-prone to write. Their recipe, AIDSAFE, manufactures the data with a society of agents.
Agents deliberate together over policies, expanding the reasoning of each other step by step, while a refiner stage prunes repetitive, redundant, or deceptive thoughts. A sister recipe uses belief augmentation to mint matched good and bad traces for preference training. Fine-tuning open models on the resulting data improved policy adherence and reasoning quality in their tests. The authors pitch agent-made deliberation as the scalable path to safer models.
Catalog record
| CREATOR | Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21784
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris Abstract Safety reasoning is a recent paradigm where LLMs reason over safety policies before generating responses, thereby mitigating limitations in existing safety measures such as over-refusal and jailbreak vulnerabilities. However, implementing this paradigm is challenging due to the resource-intensive process of creating high-quality policy-embedded chain-of-thought (CoT) datasets while ensuring reasoning remains accurate and free from hallucinations or policy conflicts. To tackle this, we propose AID safe : Agentic Iterative Deliberation for Safety Reasoning, a novel data generation recipe that leverages multi-agent deliberation to iteratively expand reasoning on safety policies. A data refiner stage in AID safe ensures high-quality outputs by eliminating repetitive, redundant, and deceptive thoughts. AID safe -generated CoTs provide a strong foundation for supervised fine-tuning (SFT)-based safety training. Additionally, to address the need of preference data in alignment stages, such as DPO training, we introduce a supplemental recipe that uses belief augmentation to create distinct selected and rejected CoT samples. Our evaluations demonstrate that AID safe -generated CoTs achieve superior policy adherence and reasoning quality. Consequently, we show that fine-tuning open-source LLMs on these CoTs can significantly improve safety generalization and jailbreak robustness while maintaining acceptable utility and over-refusal accuracy. AID safe -generated CoT datasets can be found here . 1 Introduction Recently, there has been a paradigm shift in LLM safety training towards “ safety reasoning. ”—an approach where models explicitly reason over safety policies before generating responses Jaech et al. 2024 ; Guan et al. 2024 . Safety reasoning, typically implemented through Chain-of-Thought (CoT) reasoning, has shown promise in improving jailbreak robustness and reducing over-refusals Guan et al. 2024 ; Zaremba et al. 2025 . However, adopting this paradigm presents significant data challenges. Effective safety reasoning requires high-quality CoTs that explicitly reason over a given set of safety policies. Specifically, for each prompt, we need well-reasoned CoT-response pairs. Obtaining such data through human annotations is prohibitively expensive due to the subjective nature of safety reasoning. Furthermore, generating CoTs that comprehensively cover multiple policies is time-consuming and cognitively demanding. As a result, an alternative approach is to leverage LLMs themselves to generate policy-embedded CoTs Guan et al. 2024 . However, this is a non-trivial task due to two key challenges. (1) Resource constraints : generating high-quality CoTs requires capable reasoning models Jaech et al. 2024 ; Guo et al. 2025 . However, training or acquiring such models is costly, making it inaccessible to most open-source initiatives. (2) Flawed reasoning : even with access to a powerful LLM, generated reasoning can be incorrect, deceptive, or misaligned with safety policies due to hallucinations. Additionally, safety policies are often inherently fuzzy or conflicting, further complicating the generation of reliable reasoning data. We address these data challenges by introducing a novel data generation recipe based on A gentic I terative D eliberation for SAFE ty reasoning ( AID safe ), designed to generate high-quality policy-embedded CoT datasets without requiring an expensive reasoning-capable generator . Our approach leverages collaborative reasoning and refinement in a multi-agent environment to generate high-quality thoughts that reason over safety policies (examples in Appendix E ). The deliberation stage in AID safe incorporates an iterative process where multiple agents collaboratively expand the thoughts over a defined set of policies to come up with the best response. This stage ends once the agents reach a consensus or exhaust a predefined deliberation budget. In the second, refiner stage, the output of the deliberation stage are post-processed to filter out redundant, deceptive, or policy-inconsistent thoughts , ensuring that the generated CoTs are of high quality and adhere to the specified policies. Our approach is inspired by related studies where multi-agent collaboration has been shown to reduce hallucinations and enhance reasoning reliability in tasks , such as mathematical reasoning, motivating our adoption of an agentic framework for CoT generation in the context of safety reasoning Du et al. 2023 . We evaluate AID safe outputs using two approaches. First, we assess the quality of generated CoTs through data quality metrics such as faithfulness to safety policies, completeness, relevance, and coherence. Second, we fine-tune open-source models, such as Mixtral Jiang et al. 2023 and Qwen Yang et al. 2024 , on AID safe -generated CoTs and assess their impact on models’ safety. We find that AID safe leads to improvements of the models both in safety generalization and jailbreak robustness while incurring minimal regression on their utility and over-refusal accuracy. Additionally, we introduce a supplemental recipe that leverages a belief augmentation model to generate diverse preference data that can be used in alignment stages such as Direct Policy Optimization (DPO) Rafailov et al. 2024 . We refer to this model as the “ear-whisperer” agent, whose role is to subtly influence the target LLM’s reasoning process. This approach ensures a controlled contrast in preference data, enhancing the effectiveness of safety alignment. Our ear-whisperer powered recipe achieves a significant distribution shift in preference data, as evidenced by our policy adherence metrics (see Section 5 ). We summarize our contributions as follows: • We introduce a multi-agent deliberation framework, AID safe for generating high-quality, policy-embedded CoT data. • We present extensive evaluation and experimentation to demonstrate the quality of this data and how it enhances the safety of models with minimal regression on utility. • We introduce a supplemental ear-whisperer agent-based recipe to generate diverse CoT-based preference data, facilitating effective DPO training for safety alignment. • We release a policy-embedded CoT dataset to support safety training, which can be utilized for supervised fine-tuning LLMs to improve safety. 2 AID safe Framework As shown in Figure 1 , we present a novel approach to generate CoTs that reason over safety policies while responding to user queries. This framework is designed to address key challenges in LLM safety reasoning, such as the application of nuanced safety policies and the avoidance of over-refusal. Below, we provide a detailed explanation of the safety policies we used, the initialization process, the deliberation stage, and the refinement stage. Figure 1: Proposed Multi-agent Deliberation Framework to Generate Safety-embedded CoTs 2.1 Safety Policies Our experiments incorporate five key safety policies derived from existing literature Qi et al. 2023 : Hate-Harass-Violence, Fraud and Deception, Physical Harm, Illegal Activity, and Helpfulness and Respectfulness Policy. Each policy is detailed using concise natural language descriptions to ensure that AID safe agents clearly understand their implications and applications. Hate-Harass-Violence Policy: “Must not generate content that is hateful, harassing, abusive, or threatening towards individuals or groups. Should avoid any insulting, discriminatory, violent, or manipulative language.” These descriptions, detailed in the Appendix A , serve as guidelines for reasoning and decision-making during the deliberation stage. While safety reasoning is the primary focus of our experiments in this paper, our framework can accommodate any type of policies, broadening its applicability to other domains requiring complex reasoning. 2.2 Initialization The initialization stage establishes the foundation for the deliberation process. It consists of two critical components, (1) intent decomposition and (2) seed CoT and response creation. Intent Decomposition This is an essential step for interpreting the user’s query comprehensively. An LLM agent identifies explicit