KARL: Mitigating Hallucinations in LLMs via K nowledge-Boundary- A ware R einforcement L earning
Cheng Gao, Cheng Huang, Kangyang Luo, Ziqing Qiao · 2026
Read KARL: Mitigating Hallucinations in LLMs via K nowledge-Boundary- A ware R einforcement L earning by Cheng Gao, Cheng Huang, Kangyang Luo, Ziqing Qiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22779.txt.
Introduction
Gao and colleagues teach language models to say no when a question lies past their edge, without falling into too much caution. Their KARL plan gauges the edge on the fly from group scores, backing right answers or guided holding back as the edge shifts. A two-stage training plan first maps the edge, then turns past-edge errors into saying no while keeping accuracy high.
Catalog record
| CREATOR | Cheng Gao, Cheng Huang, Kangyang Luo, Ziqing Qiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54891 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22779
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KARL: Mitigating Hallucinations in LLMs via K nowledge-Boundary- A ware R einforcement L earning Cheng Gao, Cheng Huang, Kangyang Luo, Ziqing Qiao Abstract Enabling large language models (LLMs) to appropriately abstain from answering questions beyond their knowledge is crucial for mitigating hallucinations. While existing reinforcement learning methods foster autonomous abstention, they often compromise answer accuracy because their static reward mechanisms, agnostic to models’ knowledge boundaries, drive models toward excessive caution. In this work, we propose KARL , a novel framework that continuously aligns an LLM’s abstention behavior with its evolving knowledge boundary. KARL introduces two core innovations: a Knowledge-Boundary-Aware Reward that performs online knowledge boundary estimation using within-group response statistics, dynamically rewarding correct answers or guided abstention; and a Two-Stage RL Training Strategy that first explores the knowledge boundary and bypasses the “ abstention trap ”, and subsequently converts incorrect answers beyond the knowledge boundary into abstentions without sacrificing accuracy. Extensive experiments on multiple benchmarks demonstrate that KARL achieves a superior accuracy-hallucination trade-off, effectively suppressing hallucinations while maintaining high accuracy across both in-distribution and out-of-distribution scenarios. Figure 1: Hallucination rate vs. accuracy on NaturalQuestions (NQ) ( 19 ) . KARL achieves a better trade-off by increasing accuracy while reducing hallucination compared to existing non-abstention and abstention-aware methods. Dashed isocontours represent constant Rely scores. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in open-domain, knowledge-intensive tasks ( 26 ; 11 ; 13 ) . However, their reliability is often undermined by hallucination—the tendency to generate plausible but factually incorrect responses with confidence, rather than admitting they do not know the answer ( 24 ; 14 ) . A key cause of this issue is that conventional training typically penalizes abstention as an incorrect prediction, encouraging models to guess blindly in order to maximize training accuracy even when they lack sufficient knowledge ( 17 ) . Such behavior poses serious risks in high-stakes domains like medicine and law, where minor factual errors can lead to harmful consequences ( 43 ; 7 ; 23 ; 25 ) . Therefore, enabling LLMs to abstain appropriately has emerged as a fundamental objective in safety alignment ( 41 ) . Recently, Reinforcement Learning (RL) has gained significant attention for mitigating hallucination. Instead of forcing models to mimic fixed abstention responses, RL methods guide them to discover optimal policies through feedback signals ( 4 ; 27 ; 28 ) . For example, early efforts employed PPO or DPO to align models with abstention preferences ( 1 ; 16 ; 44 ) . More recently, Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm due to its effectiveness and data efficiency ( 31 ; 8 ; 13 ) . Building upon this, notable works such as TruthRL leverage GRPO ( 31 ) with a ternary reward to jointly optimize correct answers, abstentions, and incorrect responses ( 40 ) . While these methods allow models to autonomously learn when to abstain, they still suffer from substantial degradation in answer accuracy. The underlying issue is that existing methods often rely on static reward structures that are agnostic to the model’s knowledge boundary, which drives policies to converge rapidly toward abstention. As a result, models tend to become overly cautious, suppressing both hallucinations and correct answers. Thus, a central question arises: how to design a training framework that reduces hallucinations while maintaining answer accuracy, i.e., ensuring LLMs abstain only when exceeding their knowledge boundary? To tackle the above issues, we propose KARL , a framework that continuously aligns the model’s abstention behavior with its evolving knowledge boundary. KARL rests on two key innovations. First, we devise a Knowledge-Boundary-Aware Reward ( KAR ) mechanism that uses an online knowledge boundary estimator to assess the model’s competence on each query. While this estimator can be implemented in different ways, we adopt a simple and efficient sampling-based approach that dynamically adjusts the reward strength based on whether correct answers appear in each sampled group, thereby motivating the model to confidently explore questions within its competence while steering it toward abstention when knowledge is genuinely lacking. Second, to structurally decouple knowledge boundary exploration from abstention calibration, we design a two-stage RL training strategy . An initial knowledge boundary exploration stage employs a mixed-reward strategy that maximizes accuracy and bypasses the “ abstention trap ” (See Fig. 2 ); a subsequent abstention calibration stage then converts persistent errors into appropriate abstentions without erasing prior accuracy gains. Together, these components enable LLMs to abstain only when exceeding their knowledge boundaries, thereby reducing hallucinations while maintaining high accuracy. Extensive experiments on multiple benchmarks demonstrate that KARL achieves a superior accuracy-hallucination trade-off (Figure 1 ). Unlike prior methods that often sacrifice accuracy, KARL significantly slashes hallucination while maintaining accuracy, even achieving improvements on certain benchmarks. For instance, reducing hallucinations by 38.6% on NQ while improving accuracy by 4.0%. This capability to suppress hallucinations while maintaining high answer accuracy generalizes robustly across out-of-distribution inputs. 2 Pilot Study Models often struggle to balance factual accuracy with appropriate abstention. In this section, we first formalize the abstention task and reward structures. We then investigate the underlying trade-off between answering and abstaining, focusing on two core aspects: First, we assess whether binary rewards improve pass@1 in short-form open-domain QA, extending prior successes in reasoning tasks ( 47 ; 49 ) . Second, we identify a critical failure mode of static ternary rewards: excessive early-stage abstention that leads to irreversible policy collapse. By tracing the evolution of response distributions, we uncover the mechanistic drift into this “ abstention trap ”. Figure 2: Training dynamics of response distributions for Llama-3.1-8B-Instruct on NQ under different reward schemes. The plots show the proportions of Correct (green), Abstention (yellow), and Incorrect (red) responses during early training for (a) Binary Reward, (b) Static Ternary Reward, and (c) our method. The red dashed region in (b) highlights a rapid increase in abstention accompanied by declining accuracy. Task Formulation Given a factual query q , a language model \pi must decide whether to generate an answer a or to abstain from answering (e.g., output “I don’t know”). The goal is to train a model that abstains only when it is likely to produce an incorrect answer, thereby improving reliability while preserving answer accuracy. Formally, we consider a policy model \pi_{\theta} which, given q , generates a response y . The response space is \mathcal{Y}=\mathcal{A}\cup\{\varnothing\} , where \mathcal{A} is the set of possible answer texts and \varnothing denotes abstention. Each query q is associated with a ground-truth answer a^{\star} , and the outcome of a response is evaluated with a ternary reward r based on its factual correctness: \displaystyle r(y,a^{\star})=\begin{cases}r_{\text{pos}},&\text{if }y\text{ is correct},\\ r_{\text{neg}},&\text{if }y\text{ is incorrect},\\ r_{\text{abs}},&\text{if }y=\varnothing,\end{cases} (1) where r_{\text{pos}}>r_{\text{abs}}\geq r_{\text{neg}} . The objective is to optimize the policy \pi_{\theta}(y|q) to maximize the expected reward \mathbb{E}_{q,y\sim\pi_{\theta}}[r(y,a^{\star})] , thus learning when to answer and when to abstain. Binary Rewards for Knowledge Boundary Exploration. Prior work shows RLVR bolsters model confidence in correct reasoning paths, improving pass@1 ( 47 ; 49 ) . To investigate whether this benefit extends to short-form open-domain QA, we adopt GRPO ( 31 ) as our optimization backbone (formalized in Appendix B ) and train models on NQ with a binary reward (1