Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model

Pankayaraj Pathmanathan, Furong Huang · 2026

Read Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model by Pankayaraj Pathmanathan, Furong Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09665.txt.

Introduction

Pathmanathan and Huang examine deliberative alignment, distilling teacher-model reasoning to deepen safety beyond refusal. Despite stronger safety, a teacher-student alignment gap persists, hurting safety and utility. Aligned models can also retain base-model unsafe behaviors. Best-of-n sampling attributes unsafe behavior to the base model in latent space, down-ranking unsafe responses. Across 7 teacher and 6 student models, attack success falls by about 28 percent on DAN, 31 percent on WildJailbreak, and 35 percent on StrongREJECT. Utility loss stays minimal. Gains persist after reinforcement-learning training.

Catalog record

CREATORPankayaraj Pathmanathan, Furong Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112056
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model Pankayaraj Pathmanathan, Furong Huang Abstract While the wide adoption of refusal training in large language models (LLMs) has showcased improvements in model safety, recent works have highlighted shortcomings due to the shallow nature of these alignment methods. To this end, the work on Deliberative alignment proposed distilling reasoning capabilities from stronger reasoning models, thereby instilling deeper safety in LLMs. In this work, we study the impact of deliberative alignment in language models. First, we show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models, which affects both the safety and general utility of the student model. Furthermore, we show that models aligned through deliberative alignment can retain unsafe behaviors from the base model despite learning the reasoning patterns of larger reasoning models. Building upon this observation, we propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space, thereby down-ranking unsafe responses to gain a meaningful improvement in model safety across multiple safety benchmarks with minimal loss in utility. In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2 % 28.2\% in DAN, 31.3 % 31.3\% in WildJailbreak and 35.4 % 35.4\% in StrongREJECT benchmarks. We further show that these safety gains prevail post RL training, thus highlighting the uncertainty in safety reasoning and it’s explicit attribution to the base model. Code, models, dataset: https://github.com/pankayaraj/Robust_Deliberative-Alignment 1 Introduction Large language models (LLMs) ( Guo et al. 2025 ) have demonstrated remarkable advancements in reasoning, safety and, capability largely driven by post training alignment methods ( Bai et al. 2022 , Ouyang et al. 2022 , Rafailov et al. 2024 , Guan et al. 2025 ) . While these alignment methods have shown early success in inducing effective safety refusals recent works ( Qi et al. 2024 ) have show the shallow nature of these alignment methods thus enabling effective attacks that aims to bypass of these refusal behaviors ( Zou et al. 2023 ) . Figure 1 : I. Deliberative Alignment : The figure outlines the methodology behind deliberative alignment. For a given labeled training data a stronger reasoning LLM generates a safety reasoning chain of thought (CoT) alongside the refusal. Post filtering, this CoT + Refusal response is then used to instruction tune a non reasoning LLM. As a last stage the instruction tuned LLM is further finetuned via reinforcement learning (eg. GRPO) for further safety alignment. II. Latent Similarity based BoN sampling : This showcases the outline of the our proposed BoN sampling strategy. Based on our empirical observation attributing unsafe behavior to the base model in the latent space we propose a BoN sampling that effectively leverages this observation for a safet response generation. To this end with the raising traction towards reasoning models that leverages stronger chain of thought (CoT) ( Guo et al. 2025 ) the work of Guan et al. 2025 have proposed on instilling reasoning capabilities from reasoning LLMs into non reasoning LLMs thereby enabling deeper safety on language models ( OpenAI et al. 2024 ) . These alignment methods as seen in Figure 1 starts with creating golden reasoning traces from stronger reasoning LLMs aided by safety polices. Following a reward based filtering, these reasoning traces are then used to instruction tune non reasoning LLMs. These instruction tuned models are often times further subjected to a reinforcement learning (RL) based finetuning. While these quality of these reasoning traces remains higher due to the stronger nature of the reasoning model and post generation filtering, due to the difference in model capabilities between reasoning and non reasoning LLMs the effectiveness of these reasoning traces in aligning the non reasoning LLMs remains an open question. In this work, we first (1) study the effect of model differences in both safety alignment and utility preservation. When it comes to safety we find that despite processing stronger safety capabilities certain reasoning models are ineffective in aligning certain non reasoning LLMs. In contrary to the earlier finding of Shi et al. 2025 we show that at scale this teacher student alignment gap is not directly proportional to model size differences. Further we show that despite at times providing better safety the using reasoning traces from weaker models comes at a larger general utility drop. Furthermore (2) we find that both post instruction tuning and RL alignment, despite the student LLM learning reasoning capabilities, these student models still showcase the tendencies to elicit unsafe behaviors regardless of the alignment gap. Interestingly, we show that these unsafe behaviors can be explicitly attributed to the base student language models. Building upon these findings (3) we propose a best on N (BoN) sampling method which, by explicitly attributing the unsafe responses to the base student model in the latent space, improves safety across the combination of 7 teacher models and 6 student models of different classes and sizes. Across three different benchmarks, namely DAN ( Shen et al. 2024 ) , WildJailbreak ( Jiang et al. 2024 ) , StrongREJECT ( Souly et al. 2024 ) we show that our methods result in an average attack success rate (ASR) 28.2\% 31.3\% , and 35.4\% respectively, with minimal performance loss in utility as measured by a subset of MMLU ( Hendrycks et al. 2021 ) and GSM8K ( Cobbe et al. 2021 ) . We further showcase that the safety gains carry over even after the RL stage, with an average ASR drop of 21.9\%,35.3\%, and 48.0\% respectively. Our contributions in this work can be summarized as follows • We under 7 teacher models and 6 student models showcases the presence of an alignment gap in deliberative alignment regardless of the teacher capability. • We empirically present the evidence towards the existence of uncertainty in the downstream distilled models and their attribution to the base models. • Building upon this observation we propose a latent similarity based BoN sampling method that improve the model safety without additional external reward across multiple safety benchmarks in both the instruction and RL training stages of deliberative alignment while preserving it’s efficacy against adaptive attacks We organize the remainder of this paper as follows: In Section 2 , we introduce the deliberative alignment paradigm and our experimental setup. In Section 3 , we demonstrate the impact of teacher, student model variation and the existence of uncertainty in reasoning alignment. Finally, in Section 4 4 , we propose our Best-of-N (BoN) sampling method and demonstrate its efficacy in improving model safety. Additional results and textual examples are included in the Appendix. 2 Deliberative Alignment Deliberative alignment ( Guan et al. 2025 ) stems from the observation that stronger reasoning models are capable of providing stronger reasoning and safer responses given policy instructions regarding safety. However, this leads to long context generation, as comprehensive policy inclusion becomes a requirement. This could result in high latency costs and may also lead to unsafe responses if the model misses specific policy instructions. To

Read the complete text