CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models
Zhehao Tan, Yihan Jiao, Dan Yang, Junjie Wang, Duolin Sun, Jie Feng, Xidong Wang, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu · 2026
Read CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models by Zhehao Tan, Yihan Jiao, Dan Yang, Junjie Wang, Duolin Sun, Jie Feng, Xidong Wang, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04406.txt.
Introduction
Tan and colleagues train retrieval models to stay faithful to context with CTRL-RAG. Old reinforcement rewards judge faithfulness poorly and miss similar answers. No self reward exists, and self judgment without objective feedback breeds hallucinations and collapse. Contrastive Likelihood Reward optimizes the likelihood gap between answers with and without evidence. That push extracts evidence and raises confidence only when grounded, doing well alone or with external rewards.
Catalog record
| CREATOR | Zhehao Tan, Yihan Jiao, Dan Yang, Junjie Wang, Duolin Sun, Jie Feng, Xidong Wang, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37530 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04406
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models Zhehao Tan, Yihan Jiao, Dan Yang, Junjie Wang, Duolin Sun, Jie Feng, Xidong Wang, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu Abstract With the growing use of Retrieval-Augmented Generation (RAG), training large language models (LLMs) for context-sensitive reasoning and faithfulness is increasingly important. Existing RAG-oriented reinforcement learning (RL) methods rely on external rewards that often fail to evaluate document faithfulness , and may misjudge similar answers in open-domain settings. In addition, there is no RAG-based self-reward mechanism. Moreover, although such a mechanism could in principle estimate answer confidence given documents, the absence of objective feedback in a self-judgment can cause “hallucination accumulation” and eventual model collapse. To tackle these issues, we propose a novel "internal–external" hybrid reward framework centered on a Contrastive Likelihood Reward (CLR) . CLR directly optimizes the log-likelihood gap between responses conditioned on prompts with and without supporting evidence. This encourages the model to extract relevant evidence and increases its confidence when grounded in a specific context. Experiments show that our method (used alone or combined with external correctness rewards) achieves strong performance on single-hop, multi-hop, vertical-domain, and faithfulness benchmarks. Our training code and models are coming soon. † † affiliationnotice: Equal contribution 1 Introduction With the burgeoning prominence of Retrieval-Augmented Generation (RAG), the enhancement of contextual capabilities has become a focal point in the post-training phase of Large Language Models (LLMs). Figure 1 : The comparison between the traditional RAG RL methods (external judge signals) and our Contrastive Likelihood Rewards (CLR). All rollouts are generated using the same input; however, outcomes may vary depending on the extent to which the model utilizes the retrieved documents. A higher positive score indicates a greater degree of document utilization by the model, whereas a larger negative score indicates that the documents pose a greater burden on the model. Current approaches largely employ Supervised Fine-Tuning (SFT) to enhance these capabilities ( 25 ; 7 ) or utilize Reinforcement Learning (RL) guided by external signals (e.g., accuracy, citation quality, format compliance) ( 6 ; 20 ) . However, these methods face some limitations: Deficiencies in External Evaluation to RAG Scenarios. The external reward system is the mainstream approach for reinforcement learning in RAG-related scenarios, but it often acts as an imperfect judge of signals. For example, it fails to judge contextual faithfulness. Moreover, accuracy-oriented methods may yield false positives/negatives or fail to rank similar candidate answers effectively. Additionally, citation-based rewards are highly susceptible to noise from formatting errors or poor instruction adherence, thereby confounding the assessment of core reasoning proficiency as shown in Figure 1 . Insufficient Adaptation and Attention of existing internal-rewarding methods to RAG Scenarios. Existing uncertainty quantification and control methods based on entropy ( 24 ) or perplexity ( 13 ) are predominantly designed and evaluated in the context of general language modeling or open-ended generation tasks, and there is a lack of dedicated research targeting RAG, where answering strongly depends on external context. In RAG settings, models perform conditional generation given retrieved evidence, and both the sources of uncertainty and the way contextual information is exploited differ substantially from generation without such external inputs. However, current work has largely overlooked this scenario, lacking fine-grained characterization and systematic analysis of model answering behavior under contextual constraints. Moreover, Sole reliance on internal-rewards without objective external feedback, can lead to “hallucination accumulation” ( 22 ) in complex reasoning tasks, potentially causing model collapse. To address these challenges, we propose a novel reward mechanism that integrates intrinsic and extrinsic signals via contrastive likelihood estimation. In this framework, the intrinsic signal is derived from the log-probabilities of the generated answers, while the extrinsic supervision signal is sourced from pre-annotated supporting documents. This reward is specifically designed to bolster the model’s accuracy and reasoning capabilities given a query and retrieved context. Formally, we define the reward as the differential contrastive likelihood, calculated as the gap between the answer probability conditioned on both supporting and noisy documents versus the probability conditioned solely on noisy documents. This signal can be used independently or superimposed with external correctness rewards. Our objective is twofold: (1) to enable the model to effectively identify the supporting document amidst noise. (2) To ensure that once the supporting document is located, the model exhibits increased confidence in generating the answer based strictly on that context. Our contributions are summarized as follows: • Novel RAG-Specific RL Framework. We propose CTRL-RAG , the first reinforcement learning approach specifically designed to optimize contextual faithfulness and contextual reasoning in RAG scenarios using Contrastive Likelihood Rewards (CLR). By integrating intrinsic log-probabilities with extrinsic document supervision, CLR mitigates reward sparsity and ensures answers are grounded in retrieved evidence rather than parametric memory. • Robustness Across Architectures. Extensive experiments on Dense and MoE models demonstrate significant performance gains, proving the effectiveness and generalizability of our hybrid reward mechanism. 2 Related Work Post-training for RAG Generator . While RAG and DeepSearch systems enhance LLM knowledge and mitigate hallucinations ( 26 ; 3 ; 11 ) , they demand generators with high faithfulness and context-aware reasoning for complex generation tasks. SFT-based methods like Self-RAG ( 1 ) and HIRAG ( 7 ) introduce specialized tokens or hierarchical data to improve evidence evaluation and information extraction. Meanwhile, RL approaches such as RAG-RL ( 6 ) and PA-RAG ( 20 ) optimize models for citation accuracy and document preference. To evaluate these capabilities, benchmarks such as MuSiQue ( 19 ) focus on reasoning, while PRGB ( 18 ) and FaithfulEval ( 15 ) specifically measure how well models prioritize external documents over internal priors through counterfactual or isolated testing. External and Internal Reward Signals . Reinforcement Learning with Verifiable Rewards (RLVR) has excelled in deterministic domains like math and code ( 2 ; 16 ; 5 ; 10 ) , yet applying it to general RAG tasks remains challenging. Rule-based rewards in RAG often lead to reward hacking, where models mimic formats without improving reasoning, while LLM-as-a-judge or Reward Models incur high costs and provide sparse, noisy signals ( 2 ; 23 ; 12 ) . Conversely, internal signals—such as entropy ( 24 ; 13 ) and self-certainty ( 27 ) or self-rewarding mechanisms—leverage the model’s own confidence to improve accuracy ( 22 ) . However, internal signals alone cannot expand a model’s knowledge boundaries or ensure grounding in retrieved context. This gap suggests that a hybrid approach, combining external verification with internal confidence signals, is necessary to achieve robust faithfulness and reasoning. 3 Preliminaries In this section, we establish the theoretical framework for our reward design. We first formalize the generation process and the current rule-based reward mechanisms in Retrieval-Augmented Generation (RAG), and subsequently introduce the Group Relative Policy Optimization (GRPO) algorithm. 3.1 Rewards for RAG Generator The RAG generator aims to produce a reliable and accurate response y conditioned on a user query q and a set of retrieved documents \mathcal{D}=\{d_{1},d_{2},\dots,d_{k}\} . Formally, the generator, parameterized by \theta , defines a policy \pi_{\theta} that models the conditional probability: P(y|q,\mathcal{D})=\prod_{t=1}^{T}P(y_{t}|y_{<t},q,\mathcal{D}) (1) where T denotes the sequence length. In many open-domain QA scenarios, the quality of a generated response is typically evaluated using the following rule-based metrics: • Citation Reward