SafeConstellations: Steering LLM Safety to Reduce Over-Refusals Through Task-Specific Trajectory

Utsav Maskey, Sumit Yadav, Mark Dras, Usman Naseem · 2025

Read SafeConstellations: Steering LLM Safety to Reduce Over-Refusals Through Task-Specific Trajectory by Utsav Maskey, Sumit Yadav, Mark Dras, Usman Naseem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11290.txt.

Introduction

Safety-tuned chatbots sometimes refuse harmless requests that merely sound dangerous, annoying users with false alarms. Utsav Maskey and co-authors traced how that happens inside the model. As text flows through the layers, its representation follows a path they call a constellation. Each task traces a consistent path. Refusals show up as predictable detours from the benign route.

Their fix, applied at inference time, nudges the representation back onto the task's non-refusal path. The steering applies only to task types prone to over-refusing, leaving other behavior alone. Tests showed fewer false refusals with little loss of usefulness. The authors offer the method as a principled, targeted alternative to retraining.

Catalog record

CREATORUtsav Maskey, Sumit Yadav, Mark Dras, Usman Naseem
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SafeConstellations: Steering LLM Safety to Reduce Over-Refusals Through Task-Specific Trajectory Utsav Maskey, Sumit Yadav, Mark Dras, Usman Naseem Abstract LLMs increasingly exhibit over-refusal behavior, where safety mechanisms cause models to reject benign instructions that superficially resemble harmful content. This phenomena diminishes utility in production applications that repeatedly rely on common prompt templates or applications that frequently rely on LLMs for specific tasks (e.g. sentiment analysis, language translation). Through comprehensive evaluation, we demonstrate that LLMs still tend to refuse responses to harmful instructions when those instructions are reframed to appear as benign tasks. Our mechanistic analysis reveal that LLMs follow distinct “constellation” patterns in embedding space as representations traverse layers, with each task maintaining consistent trajectories that shift predictably between refusal and non-refusal cases. We introduce SafeConstellations , an inference-time trajectory-shifting approach that tracks task-specific trajectory patterns and guides representations toward non-refusal pathways. By selectively guiding model behavior only on tasks prone to over-refusal, and by preserving general model behavior, our method reduces over-refusal rates by up to 73% with minimal impact on utility–offering a principled approach to mitigating over-refusals. 1 Introduction The rapid advancement of Large Language Models (LLMs) has revolutionized natural language processing capabilities. However, ensuring these models remain both safe and useful presents a significant challenge ( Röttger et al. 2024 ; Cui et al. 2025 ; Karaman et al. 2025 ; Zhang et al. 2025 ) . Current safety alignment methods while successful at preventing harmful responses, inconsistently exhibit over-refusal behavior— rejecting benign instructions that contain sensitive keywords or contexts (Figure 1 ). This becomes apparent in production applications where the context are extracted from external sources, user inputs, or retrieval augmented generation (RAG) ( Lewis et al. 2020 ) pipelines. For example, a customer support system that relies on sentiment analysis might refuse to process reviews containing metaphorical expressions like “killer good” or a translation service might reject historical documents that discuss sensitive topics of wars or revolutions due to perceived harmful content. Such refusals disrupt LLM helpfulness and diminish model utility. While over-refusal ( Cui et al. 2025 ) broadly refers to the rejection of seemingly toxic inputs, we hypothesize that refusal rates differ by task for the same input. We define the problem of task over-refusal as the rejection of benign instructions solely based on inputs context, ignoring the actual task intent, which is benign. Ideally, we want the responses to be (1) non-refusing and (2) either direct or cautious . Figure 1: Task Over-Refusal: LLMs fail to identify the intended benign task (sentiment analysis, translation, etc.) before refusing to respond. Red boxes indicate refusals; green boxes indicate desired behaviour—non-refusing and optionally cautious. Figure 2: LLMs follow distinct “constellation” patterns in embedding space as representation traverse layers, with each task maintaining consistent trajectories that shift only marginally (and predictably) between refusal and non-refusal cases. When we combine multiple tasks together (i.e. combine the datasets), the aggregate layer-wise non-refusal direction becomes ambiguous. To address safety concerns, recent defense mechanisms leverage internal state monitoring and intervention to promote safer LLM outputs ( Wei et al. 2024 ; Shen et al. 2025 ; Xie et al. 2024 ; Zhou et al. 2024 ) . Prior works identified that LLMs encode task-specific information in their weight space, allowing mechanistic differentiation of tasks ( Gueta et al. 2023 ) . Building on these advances, we investigate the diversity of embedding regions across LLM layers and extend the literature by identifying consistent propagation patterns for different task types (input categories). We also examine how these patterns relate to output behaviors such as refusal, direct answering, and cautious responses. This leads us to hypothesize that LLM layers encode task-specific information, enabling us to analytically distinguish original tasks—such as sentiment analysis, translation, paraphrasing, etc. And these task-specific “constellations patterns” are invariant to input content, and only reflect the task’s intent. Within each task constellation, finer variations correspond to behavioral outcomes, such as direct answering, refusal, or cautious response—appearing as localized shifts in the broader task trajectory (Figure 2 ). By mapping these task-dependent patterns, we design interventions that selectively adjust model behavior without compromising general utility. Our contributions are as follows: • We develop a benchmark dataset for measuring task over-refusal in NLP that analyzes how LLM safety goals vary by text type and task. • We propose a novel steering approach that guides representations layer-by-layer during inference to reduce over-refusal. Over-refusal are reduced without compromising general capabilities. • Experiments reveal that LLMs encode task identity via consistent internal trajectory patterns. Leveraging these patterns enables precise response adjustments, reducing over-refusals with negligible impact on utility. 2 Related Work LLM Safety Alignment and Defense: Ensuring the safe and reliable operation of LLMs is a critical research challenge ( Ouyang et al. 2022 ; Bai et al. 2022 ; Dai et al. 2023 ) . Safety alignment methods can be broadly categorized into instruction tuning, preference alignment, and inference-time defenses. Instruction tuning involves supervised fine-tuning on adversarial prompts and safety-oriented demonstrations ( Hsu et al. 2024 ; Choi, Du, and Li 2024 ) . Preference alignment methods, such as PPO ( Schulman et al. 2017 ) , DPO ( Rafailov et al. 2023 ) , and GRPO ( Shao et al. 2024 ) optimize model behaviour using human or synthetically generated preference data. Recent advances include reasoning-level preference optimization ( Mou et al. 2025 ) , which addresses safety vulnerabilities and over-refusals by reflecting on the model’s reasoning steps and aligning its preferences to spot harmful intent in the early steps. Furthermore, automated frameworks for generating high-quality preference data ( Huang et al. 2023 ) enhances alignment robustness. Inference-time defense mechanisms such as SafeDecoding ( Xu et al. 2024 ) analyze output logit differences between the aligned safer model and the unaligned harmful model to amplify the probabilities of safer disclaimers and attenuate harmful tokens, while RA-LLM ( Cao et al. 2024 ) and similar input mutating methods ( Kumar et al. 2024 ) identify jailbroken behavior by dropping a random sample of tokens and iteratively check for inconsistencies among the responses. Over-Refusal Benchmarks: Current literature highlights the trade-off between safety and utility in LLMs, where excessive alignment can lead to over-refusal—the erroneous rejection of benign but superficially sensitive queries ( Cui et al. 2025 ; Röttger et al. 2024 ) . XSTest ( Röttger et al. 2024 ) introduced a diagnostic suite of 250 hand-crafted safe prompts and 200 unsafe contrasts to evaluate exaggerated safety behaviors. Their findings indicate that lexical overfitting—where models rely on superficial keyword cues—is a primary cause of false refusals. OR-Bench ( Cui et al. 2025 ) introduces 80,000 synthetically generated over-refusal prompts, enabling large-scale assessment across tasks and models. Further work POROver ( Karaman et al. 2025 ) demonstrates that preference optimization can mitigate over-refusal in safety-aligned models. Mechanistic Interpretability and Internal State Steering: Recent research also analyzes mechanistic interpretability, exploring how internal model representations influence safety behaviors. Representation and latent space steering ( Xie et al. 2024 ; Shen et al. 2025 ; Zhou et al. 2024 ) demonstrate that targeted modifications to activation patterns can dynamically adjust model outputs without retraining or prompt modification. Notably, Jailbreak Antidote ( Shen et al. 2025 ) reveals that safety-critical

Read the complete text