Leveraging RAG for Training-Free Alignment of LLMs
John Halloran · 2026
Read Leveraging RAG for Training-Free Alignment of LLMs by John Halloran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11217.txt.
Introduction
John Halloran presents RAG-Pref, retrieval-augmented generation for preference alignment, a training-free alignment method for large language models. Standard algorithms post-train over ranked pairs. But they demand heavy compute while proving far less capable of adding refusal guardrails against recent agentic attacks.
RAG-Pref conditions on preferred and dispreferred samples to use contrastive information during inference. It is online and works with off-the-shelf kits. Paired with offline training, it enables more than an average three point seven factor gain in agentic attack refusal cases across five widely used LLMs. That beats two point nine for other online methods and one point five for offline alone. It also improves general human-preference alignment without much raising compute. Builders get refusal guardrails that work at run time.
Catalog record
| CREATOR | John Halloran |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47111 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11217
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging RAG for Training-Free Alignment of LLMs John Halloran Abstract Large language model (LLM) alignment algorithms typically consist of post-training over preference pairs. While such algorithms are widely used to enable safety guardrails and align LLMs with general human preferences, we show that state-of-the-art alignment algorithms require significant computational resources while being far less capable of enabling refusal guardrails for recent agentic attacks. Thus, to improve refusal guardrails against such attacks without drastically increasing computational overhead, we introduce R etrieval A ugmented G eneration for Pref erence alignment (RAG-Pref), a simple RAG-based alignment algorithm which conditions on preferred and dispreferred samples to leverage contrastive information during inference. RAG-Pref is online (training-free), compatible with off-the-shelf packages, and, when combined with offline (training-based) alignment algorithms, enables more than an average 3.7 factor improvement in agentic attack refusals across five widely used LLMs, compared to 2.9 for other online alignment algorithms and 1.5 for offline alignment alone. We conclude by showing that, in stark contrast to other online alignment methods, RAG-Pref similarly increases performance on general human-preference alignment tasks and does not drastically increase overall computational requirements. 1 Introduction Alignment has become a critical step towards ensuring large language model (LLM) responses align with general human preferences . Currently, alignment algorithms are dominated by reinforcement learning-based schemes–such as RLHF ( Ouyang et al. 2022 ) and the more efficient direct preference optimization (DPO) ( Rafailov et al. 2023 ) –wherein models are post-trained over pairs of preferred and dispreferred responses for each input query. The responses from resulting models are thus aligned with desirable behaviors present in preferred training data, while the undesirable behaviors present in dispreferred training data are avoided. Such alignment-tuning –i.e., alignment via RLHF, DPO, or derivative preference optimization post-training algorithms–has proven pivotal towards producing LLM assistants whose responses are accurate as well as helpful ( Ouyang et al. 2022 ; Zheng et al. 2023 ; Dubois et al. 2024 ; Zhong et al. 2025 ) . With the success of alignment-tuning and growing evidence that LLMs are highly susceptible to adversarial attacks ( Mehrotra et al. 2024 ; Liu et al. 2025 ; Chao et al. 2025 ; Liu et al. 2023b ; Shen et al. 2024 ; Zhang et al. 2023 ) , significant works have sought to produce LLM assistants whose responses are not only helpful, but also aligned with harmless behaviors ( Bai et al. 2022 ; Ji et al. 2023 ; Tian et al. 2024 ) . Such works have shown that safety alignment-tuning (SAT) may be achieved by directly including harmless and harmful preference pairs during alignment-tuning ( Ji et al. 2023 ; Team et al. 2023 ; Tian et al. 2024 ) . SAT thus enables refusal guardrails , whereby models learn to refuse malicious instructions containing harmful language, while complying with benign requests. Furthermore, recent works have shown that such refusal guardrails may be further strengthened when additional safety training labels are available via specialized SAT algorithms ( Dai et al. 2024 ; Kim et al. 2026 ) . While SAT has become a ubiquitous and critical step when deploying frontier models ( Hurst et al. 2024 ; Grattafiori et al. 2024 ; Yang et al. 2025 ; DeepSeek-AI 2025 ; Anthropic 2025 ) , three major drawbacks have emerged: 1) training requires significant computational overhead, (2) SAT is offline , i.e., retraining is required if newly catalogued adversarial attacks are to be guarded against, and–most importantly–(3) SAT frontier models have proven to be highly susceptible to recent agentic attacks which lack common refusal triggers (i.e., harmful language). The latter has emerged with the recent advent of the model context protocol (MCP), a widely adapted universal agentic protocol ( Anthropic 2025b ) . In particular, despite extensive SAT ( Grattafiori et al. 2024 ; Anthropic 2025 ) , MCP-enabled LLMs have proven to be highly susceptible to adversarial attacks which induce malicious tool use yet lack standard refusal trigger phrases ( Radosevich & Halloran 2025 ) . Such agentic attacks are referred to herein as falsely benign attacks (FBAs). Most concerningly, we show that the effectiveness of FBAs are not simply due to the recency of the MCP and the lack of such attack data during SAT. Collecting a high-quality dataset of FBAs and truly benign (TB) samples, we safety-tune five popular LLMs using state-of-the-art (SOTA) SAT algorithms DPO and SafeDPO ( Kim et al. 2026 ) . Across all models, DPO and SafeDPO display limited ability to enable FBA refusal guardrails, improving baseline refusal rates by only an average factor of 1.4 and 1.6, respectively. Alarmingly, no safety-tuned model achieves an FBA refusal rate greater than 48%. Thus, to further improve refusal guardrails in the face of agentic attacks, we introduce R etrieval A ugmented G eneration for Pref erence alignment (RAG-Pref), a new RAG-based alignment algorithm which conditions on both preferred and dispreferred examples during inference. RAG-Pref is online (training-free), easily implementable using off-the-shelf packages/components, and significantly improves LLM refusal guardrails compared to both SOTA offline and online alignment algorithms. In particular, RAG-Pref increases baseline refusal rates by an average 3-fold improvement–2.1 and 1.9 times greater than DPO and SafeDPO, respectively, and 1.8 times greater than recent online alignment strategies ( Zhu et al. 2025 ) . Moreover, we show that RAG-Pref further improves performance when augmenting generation with samples from the LLM’s closed-book (parametric) knowledge; when combined with DPO and SafeDPO aligned models, RAG-Pref further increases baseline refusal rates by an average 3.5- and 3.9-fold improvements, respectively. In addition to significantly improving agentic guardrails, we demonstrate that RAG-Pref similarly improves performance for general human-preference alignment tasks; across five SOTA alignment-tuned models, RAG-Pref increases performance over alternative online alignment algorithms by an average 26.2% and 6.4% on standard benchmarks AlpacaEval 2 and MT-Bench, respectively. Furthermore, we demonstrate the computational advantages of RAG-Pref over offline and other online alignment algorithms; RAG-Pref requires three-orders of magnitude less preprocessing time and two-orders of magnitude less GPU memory compared to offline safety-tuning a 14B parameter model, while only incurring an inference slowdown of 20% compared to 372% for SOTA decoding-based online alignment ( Zhu et al. 2025 ) . Finally, we theoretically show that RAG-Pref is guaranteed to further reduce standard RAG’s expected uncertainty by a nonnegative amount, and verify this empirically. 2 Background and Related Work 2.1 LLM Alignment-Tuning LLM alignment most commonly consists of fine-tuning given queries with accompanying preferred and dispreferred response pairs (which reflect human preferences). Initial alignment approaches, such as RLHF ( Ouyang et al. 2022 ) and RLAIF ( Lee et al. 2024 ) , first trained reward models on human preference data, then utilized reinforcement learning (RL) fine-tuning to learn a policy which maximized the reward signal. To address training instability in RL fine-tuning, subsequent work reparameterized the RL objective, allowing direct learning of the optimal policy with a simple closed form objective ( Rafailov et al. 2023 ) . The resulting algorithm (i.e., DPO), was shown to provide significantly better training stability than RL-based alignment. For general human-preference tasks, DPO and its many follow-up variants ( Melnyk et al. 2024 ; D’Oosterlinck et al. 2025 ; Jung et al. 2024 ; Ji et al. 2024 ;