Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models
Junyi Li, Hwee Tou Ng · 2025
Read Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models by Junyi Li, Hwee Tou Ng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24630.txt.
Introduction
Junyi Li and Hwee Tou Ng show that training language models to reason better also makes them hallucinate more, in a 2025 paper. Reinforcement learning has sharpened models on hard benchmarks, but the authors' measurements reveal a cost: models tuned for reasoning produce more false statements. They trace the problem to noisy training gradients, randomness from entropy, and traps in the reward landscape.
Their fix, FSPO, checks factuality at each reasoning step during training. An automated verifier scores each step against the given evidence, and the training signal is adjusted token by token to reward correctness. Across math and hallucination benchmarks on two model families, the method cut hallucinations while improving reasoning accuracy.
Catalog record
| CREATOR | Junyi Li, Hwee Tou Ng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24630
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models Junyi Li, Hwee Tou Ng Abstract Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization ( FSPO ), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance. https://github.com/nusnlp/FSPO 1 Introduction Large language models (LLMs) have recently achieved remarkable breakthroughs in natural language generation and complex reasoning tasks [ 9 , 15 ] . This progress is fueled by new training paradigms that imbue LLMs with reasoning capabilities, including fine-tuning on long chain-of-thought (CoT) solutions [ 43 ] and reinforcement learning (RL) based optimization [ 9 ] . By leveraging these approaches, state-of-the-art models can decompose complex problems into reasoning steps and iteratively refine their answers, yielding impressive results on challenging benchmarks in mathematics [ 10 ] , multi-hop question answering [ 41 ] , coding [ 2 ] , and decision-making [ 32 ] . Along with improved reasoning ability, a critical question has emerged: To what extent do reasoning models maintain factuality after RL fine-tuning? In this work, our empirical observations indicate that when LLMs are trained with outcome-driven RL, they tend to produce more factually incorrect or fabricated statements (i.e., hallucinations [ 13 , 46 ] ) during their reasoning process. We find that these hallucinations usually stem from incorrect intermediate reasoning, i.e., the model may generate unsupported or false claims in the chain-of-thought, even if its final answer is occasionally correct. Indeed, RL-aligned models can be highly fluent and confident in their step-by-step explanations while subtly introducing factual errors [ 40 ] . This issue undermines the reliability of RL-trained reasoning models, as a flawed or untruthful answer is problematic for trust and interpretability. To understand why outcome-based RL fine-tuning exacerbates hallucinations, we conduct a theoretical analysis of the RL training dynamics for reasoning tasks. Our analysis identifies three primary factors. First, optimizing solely for correct final answers leads to extremely high variance in the policy gradient when such answers are rare, yielding unstable training updates. Second, the need to discover rewarding outputs forces the policy to maintain a high level of entropy in its predictions, heightening the risk of hallucinations. Third, standard RL objectives admit spurious local optima whereby the model can converge to a confidently held wrong answer that yields zero reward. These three factors, i.e., high-variance gradient, entropy-induced randomness, and spurious local optima, together explain why a purely outcome-based RL approach can cause a reasoning model to hallucinate and struggle to learn faithful reasoning patterns. To mitigate these issues, we propose F actuality-aware S tep-wise P olicy O ptimization ( FSPO ), a novel RL fine-tuning algorithm that explicitly incorporates factuality feedback at each reasoning step. The key idea is to adjust the advantage values of individual tokens using a step-wise factuality reward signal, encouraging LLMs to generate more factual content. Specially, we employ an automated verifier to check whether each generated reasoning sentence is entailed by the given evidence, yielding step-wise factuality scores. These scores are then integrated into the overall reward signal to adjust the advantages of tokens during optimization, rewarding factually correct tokens while penalizing incorrect tokens. This factuality-aware reward shaping provides denser and more informative feedback to the policy, addressing the sparse-signal problem and reducing training instability. FSPO steers the policy toward solutions that are not only correct but also generated via faithful, verifiable chains of reasoning, thereby directly mitigating hallucinations in the RL training process. We conduct extensive experiments to evaluate the effectiveness of FSPO on both mathematical reasoning and hallucination benchmarks using Qwen2.5-7B-Base/Instruct and Llama-3.1-8B-Instruct. The results show that our method achieves significantly improved mathematical reasoning performance and reduced hallucinations. Further detailed analyses indicate that our method can enhance the factuality of reasoning steps without compromising the quality of generated content. Figure 1: Hallucination rate (%) for six groups of twelve LLMs across three benchmarks. 2 Preliminary Experiments To explore the effect of RL on hallucinations, we conduct preliminary experiments based on several LLMs and hallucination benchmarks. Models and Datasets. For comparative analysis, we select several groups of LLMs without and with large-scale RL training: (1) DeepSeek-V3 [ 21 ] and DeepSeek-R1 [ 9 ] ; (2) Qwen2.5-32B-Instruct [ 39 ] and QwQ-32B [ 28 ] ; (3) Qwen2.5-14B/7B/1.5B-Instruct [ 39 ] and R1-Distill-Qwen-14B/7B/1.5B [ 9 ] ; (4) Llama3.1-8B-Instruct [ 8 ] and R1-Distill-Llama-8B [ 9 ] . These distilled models are fine-tuned on the synthesized long-CoT data from DeepSeek-R1, thus exhibiting similar thinking modes to RL-tuned models. For the hallucination datasets, we select three benchmarks, namely TruthfulQA [ 20 ] , HaluEval [ 19 ] , and HalluQA [ 4 ] . In TruthfulQA, we adopt the generation task, where the model is prompted to generate an answer to a question, and then utilize a fine-tuned GPT-3 to predict the truthfulness of the answer. The hallucination rate is equal to the ratio of untruthful answers. HalluQA follows the same method as TruthfulQA, except that it employs GPT-4 as the evaluator. For HaluEval, we adopt the QA subset, where the LLM is provided with a question and two candidate answers and it needs to assess which answer is factually correct. The hallucination rate refers to the ratio of false judgments made by the model. Figure 2: Distribution of hallucination causes over 200 samples from HaluEval-QA. Results and Analysis. The results are shown in Figure 1 . We can clearly observe that after large-scale training with RL or long CoT data, models demonstrate dramatically higher degree of hallucinations across the three benchmarks. To further investigate the underlying causes of this phenomenon, we randomly select 200 samples from HaluEval-QA that DeepSeek-V3 answers correctly while DeepSeek-R1 does not. We manually examine these cases and classify the source of errors (refer to Appendix E for more details). In Figure 2 , we can see that most errors stem from incorrect reasoning steps. We hypothesize that this may be attributed to the outcome reward modeling, where rewards and penalties are assigned solely based on the correctness of the final answer, while ignoring signals from intermediate reasoning steps. In the following sections, we will provide a theoretical analysis of how outcome-based reward modeling in RL could lead to hallucinations, and then propose the factuality-aware step-wise RL algorithm to mitigate this issue. 3 Background Reinforcement Learning. In reinforcement learning, language model generation can be formulated as a token-level Markov decision process, denoted by \mathcal{M}=\{\mathcal{S},\mathcal{A},\mathcal{R},\pi_{\theta}\} . At the t -th generation step, the state s_{t}\in\mathcal{S} is the concatenation of the input prompt x and the generated sequence so