Advancing LLM Safe Alignment with Safety Representation Ranking
Tianqi Du, Zeming Wei, Quan Chen, Chenheng Zhang, Yisen Wang · 2025
Read Advancing LLM Safe Alignment with Safety Representation Ranking by Tianqi Du, Zeming Wei, Quan Chen, Chenheng Zhang, Yisen Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15710.txt.
Introduction
Tianqi Du and co-authors published a research paper in 2025 on picking the safest answer from a language model by reading its inner states rather than its words. Safety checks usually grade the final text, yet the model's hidden layers carry richer signals about risk. Their framework ranks several draft answers using those hidden states.
Both the prompt and each draft are encoded from middle layers of the model. A light scorer then orders the drafts by likeness to safe patterns learned at the list level. Across several test sets, the method held up far better against trick prompts designed to jailbreak the model. The team plans to release the code on publication.
Catalog record
| CREATOR | Tianqi Du, Zeming Wei, Quan Chen, Chenheng Zhang, Yisen Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing LLM Safe Alignment with Safety Representation Ranking Tianqi Du, Zeming Wei, Quan Chen, Chenheng Zhang, Yisen Wang Abstract The rapid advancement of large language models (LLMs) has demonstrated milestone success in a variety of tasks, yet their potential for generating harmful content has raised significant safety concerns. Existing safety evaluation approaches typically operate directly on textual responses, overlooking the rich information embedded in the model’s internal representations. In this paper, we propose Safety Representation Ranking (SRR), a listwise ranking framework that selects safe responses using hidden states from the LLM itself. SRR encodes both instructions and candidate completions using intermediate transformer representations and ranks candidates via a lightweight similarity-based scorer. Our approach directly leverages internal model states and supervision at the list level to capture subtle safety signals. Experiments across multiple benchmarks show that SRR significantly improves robustness to adversarial prompts. Our code will be available upon publication. 1 Introduction Recent large language models (LLMs) have achieved remarkable capabilities across a wide range of tasks. However, this power comes with serious safety and alignment concerns [ 32 , 15 , 1 ] . By default, LLMs have the potential to generate biased, toxic, or harmful content, and adversarial jailbreak prompts can coax an LLM into violating its own content guidelines [ 20 , 33 , 50 ] . These vulnerabilities persist despite extensive alignment efforts during pre-training and post-training phases [ 2 , 7 , 17 ] . In practice, the potential for harmful outputs and the ability to bypass built-in safeguards raise significant concerns for deploying LLMs in real-world applications. To mitigate these safety risks, prior work has explored a variety of defense mechanisms. A common strategy is decoding-time intervention, which redirects the decoding logic of the LLM during inference, through token distributions [ 38 , 3 ] or safe prompts [ 37 , 35 , 47 ] . For example, SafeDecoding [ 38 ] adjusts the token distribution toward safe response distributions during decoding, while in-context defense [ 35 ] aligns the generation distributions to safe contexts with demonstrations. Such interventions can introduce a trade-off between safety and fluency: altering the decoding process may degrade the model’s natural performance on benign inputs or increase inference cost. Meanwhile, post-processing-based defenses apply judging LLMs to inspect the harmfulness of LLMs [ 14 , 21 ] . Unfortunately, recent studies have shown that LLM-based safety judges are often overcautious: they flag many benign prompts as unsafe (so-called over-refusal) [ 22 , 36 ] . This unreliability, i.e. , high false-positive rates, limits their practical use, as it can render the model unhelpful even on innocuous tasks. In this work, we propose an alternative paradigm (which we call Safety Representation Ranking, SRR) for LLM safety that avoids alteration of the base model’s generation logic and unreliable external judges. Our key idea is to generate multiple candidate responses to a given prompt and then rank them by safety using the model’s internal representations . This approach is similar to using a learned reward model to select outputs [ 10 , 4 , 42 ] , but there exists an important twist: Traditional reward models are trained on the final generated text, often focusing on general measures of quality or alignment. In contrast, our proposed SRR explicitly targets safety by learning directly from the LLM’s latent features. Existing external reward models may miss fine-grained safety cues embedded in the LLM’s state vectors. Moreover, relying solely on an LLM to judge its own outputs can be unreliable and costly. By delving into the model’s internal representation space, SRR can successfully detect subtle safety-critical representations [ 34 , 49 , 48 ] that an output-only classifier might overlook, and do so with a lightweight ranking step at inference time. The SRR framework works in two phases. First, we identify safety-sensitive representations through contrastive training. We construct safety contrastive groups: for each prompt, we use examples of both safe and harmful responses. We feed these paired responses through the LLM and extract their internal representations. Because the groups are semantically related but differ in safety, we can train a lightweight model (a single-layer Transformer) to distinguish safe vectors from unsafe ones. Through this process, SRR learns which features of the LLM’s latent space correlate with safe content. Then, at inference time, we use the learned safety signals to rank candidate responses. In effect, SRR filters among the model’s own outputs without changing how they were produced. Because it operates on the outputs after generation, SRR imposes almost no modification to the LLM’s decoding logic. Its only overhead is the additional cost of scoring a few extra responses with a small model, which is negligible compared to full decoding. We conduct comprehensive experiments to validate the effectiveness of the SRR model in identifying the safety responses across multiple datasets. Not only can SRR achieve a sufficiently high accuracy in unseen harmful prompts, but it can also generalize well across different safety evaluation datasets, demonstrating its prominent generalization ability in terms of safety ranking. Additionally, we extend our analysis in terms of other alignment perspectives like privacy and fairness, which validates the potential of SRR for diverse alignment considerations and broadens the applications of SRR. Grounded by these empirical analyses, we characterize the practicality of SRR for serving as a safeguard module in real-world deployments. First, we incorporate SRR into LLM generation to study how it strengthens their robustness against jailbreak attacks. Additionally, we compare the natural performance of SRR with vanilla generation and other defense paradigms. Because SRR only ranks among natural outputs, the quality and correctness on benign queries remain essentially unchanged. Overall, our empirical results suggest that SRR is both a practical and effective module for LLM alignment. Our contributions in this paper can be summarized as follows: 1. We introduce a novel paradigm, Safety Representation Ranking (SRR), which uses LLM internal representations to rank candidate responses by safety, without altering the model’s decoding logic. 2. We demonstrate that SRR accurately selects safe outputs across diverse safety benchmarks, generalizes to novel prompts, and can be adapted to other alignment perspectives like privacy and fairness. 3. We show that integrating SRR into LLM inference significantly reduces harmful outputs under attack, with negligible impact on normal task performance. 2 Related Work LLM Safe Alignment . The issue of ensuring safe alignment in LLMs has become a longstanding challenge critical to their trustworthy deployment [ 1 , 15 , 41 ] . Specifically, LLMs have shown a tendency to generate harmful responses when confronted with malicious requests. While current alignment techniques have improved at mitigating these risks to some extent, they still tend to be superficial and inadequate [ 24 ] . Additionally, inference-time defenses can reduce the success rate of these attacks, but they often struggle with a significant drawback of rejecting benign inputs, leading to over-refusal issues. The underlying mechanism of such issues is that these distribution-based or prompt-based defenses commonly change the decoding strategies of LLMs, making their generation distributions favor refusals. Thus, ensuring safe alignment whilst maintaining the generation distribution stands for a viable solution for these risks. Safety Representations of LLMs . Building on the representation engineering techniques of LLMs [ 49 ,