The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination
Chenlong Yin, Zeyang Sha, Shiwen Cui, Changhua Meng, [2pt] The Pennsylvania State University · 2025
Read The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination by Chenlong Yin, Zeyang Sha, Shiwen Cui, Changhua Meng, [2pt] The Pennsylvania State University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22977.txt.
Introduction
Chenlong Yin and four co-authors ask whether stronger reasoning makes language model agents invent tools that do not exist. Builders often train agents to think first and then act, yet newer models seem to hallucinate more as they reason better. No prior study had tested that link in a controlled way for tool use.
The team built SimpleToolHalluBench, which checks two failure modes: no tool is present, and only decoy tools are present. Their tests support the worry that reasoning gains can raise tool hallucination. The benchmark gives builders a way to measure the trade-off before agents act in real settings.
Catalog record
| CREATOR | Chenlong Yin, Zeyang Sha, Shiwen Cui, Changhua Meng, [2pt] The Pennsylvania State University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22977
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination Chenlong Yin, Zeyang Sha, Shiwen Cui, Changhua Meng, [2pt] The Pennsylvania State University Abstract Enhancing the reasoning capabilities of Large Language Models (LLMs) is a key strategy for building Agents that “think then act.” However, recent observations, like OpenAI’s o3, suggest a paradox: stronger reasoning often coincides with increased hallucination, yet no prior work has systematically examined whether reasoning enhancement itself causes tool hallucination . To address this gap, we pose the central question: Does strengthening reasoning increase tool hallucination? To answer this, we introduce SimpleToolHalluBench , a diagnostic benchmark measuring tool hallucination in two failure modes: (i) no tool available, and (ii) only distractor tools available. Through controlled experiments, we establish three key findings. First, we demonstrate a causal relationship: progressively enhancing reasoning through RL increases tool hallucination proportionally with task performance gains. Second, this effect transcends overfitting—training on non-tool tasks (e.g., mathematics) still amplifies subsequent tool hallucination. Third, the effect is method-agnostic, appearing when reasoning is instilled via supervised fine-tuning and when it is merely elicited at inference by switching from direct answers to step-by-step thinking. We also evaluate mitigation strategies including Prompt Engineering and Direct Preference Optimization (DPO), revealing a fundamental reliability–capability trade-off : reducing hallucination consistently degrades utility. Mechanistically, Reasoning RL disproportionately collapses tool-reliability-related representations, and hallucinations surface as amplified divergences concentrated in late-layer residual streams. These findings reveal that current reasoning enhancement methods inherently amplify tool hallucination , highlighting the need for new training objectives that jointly optimize for capability and reliability. 1 Introduction The evolution of Large Language Models (LLMs) from text generators into Agents that interleave internal deliberation with external tool calls has marked a significant step towards accomplishing diverse real-world tasks ( Wei et al. 2022 ; Trivedi et al. 2023 ; Yao et al. 2023 ; Schick et al. 2024 ; Sha et al. 2025a ) . This agentic shift is largely propelled by advanced reinforcement learning techniques, which have moved beyond merely enriching knowledge to enhancing core capabilities like reasoning and tool use. Collectively, state-of-the-art reinforcement learning algorithms are gradually enabling models to acquire more advanced intelligence. However, while substantial evidence shows that reasoning abilities enhanced by reinforcement learning can increase hallucinations in model responses ( OpenAI 2025 ; Chowdhury et al. 2025 ) , its influence on models’ hallucination tendencies during tool use remains largely unexplored. We term this phenomenon tool hallucination , which occurs when models either fabricate non-existent tools or misappropriate available but irrelevant tools, leading to unfounded claims that compromise agent reliability ( Patil et al. 2024 ) . These hallucinations pose significant risks: they can directly impact real-world systems, generate misleading outputs that are difficult to verify, and critically undermine user trust by producing seemingly plausible but fundamentally flawed tool-based responses. To systematically investigate this emerging challenge, our work is guided by three central research questions: • RQ1 - Does enhancing reasoning amplify tool hallucination? We first investigate whether a causal link exists between reasoning enhancement and increased tool hallucination. This question explores if the phenomenon is a general side-effect of strengthening reasoning itself, rather than an artifact of specific training data or methods. • RQ2 - What are the underlying mechanistic drivers? Assuming such a link exists, we seek to understand its root cause. This pushes beyond behavioral observation to ask: how does reinforcing reasoning alter the model’s internal representations and processing pathways to make it more prone to fabricating tools? • RQ3 - To what extent can tool hallucination be effectively mitigated? This question examines whether existing alignment techniques, such as simple prompting or preference optimization, can effectively curb tool hallucination without compromising the very reasoning capabilities that were initially enhanced. Figure 1: Overview of our key findings. Left: Reinforcement learning for reasoning enhancement increases tool hallucination rates alongside task performance gains. Middle: Mechanistic analysis reveals that reasoning RL destabilizes tool-reliability-related representations in the model’s internal layers. Right: Mitigation strategies expose a fundamental trade-off—reducing hallucination consistently degrades utility, with no free lunch for achieving both reliability and capability. Our investigation, summarized in Figure 1 , systematically addresses these questions. To answer RQ1, we propose SimpleToolHalluBench , a lightweight diagnostic that isolates tool fidelity via two settings: No-Tool-Available task and Distractor-Tool task. Using this diagnostic, we first show that tool-specific Reasoning RL increases hallucination rates in tandem with task reward. To test whether this stems from overfitting, we then apply non-agentic Reasoning RL on a math dataset (GSM8K) with no tools. Strikingly, hallucination still rises, establishing that reasoning reinforcement itself is the driver. To address RQ2, we perform a mechanistic analysis of the model’s dynamics. Layer-wise representation similarity reveals a sharp asymmetry: while in-distribution reasoning pathways remain stable, tool-related representations collapse in early and middle layers. Further, activation probes show pronounced divergence in deep residual streams, pinpointing them as the locus where subtle inconsistencies accumulate into overt tool hallucinations. Furthermore, we evaluate mitigation strategies to address RQ3. Our experiments demonstrate that prompt-based instructions yield only marginal gains, indicating surface-level interventions are insufficient. While Direct Preference Optimization (DPO) meaningfully reduces hallucination, it introduces a substantial utility drop, underscoring the difficulty of reconciling reliability and capability under current training paradigms. Our results provide experimental and mechanistic evidence that reinforcement learning inherently biases models toward overconfident ”think-then-act” behaviors, amplifying tool hallucination. In summary, our contributions are three-fold: (1) Introducing a lightweight diagnostic benchmark, SimpleToolHalluBench , for measuring tool hallucination under controlled conditions. (2) Providing the first experimental and mechanistic evidence that reasoning-focused reinforcement learning inherently amplifies tool hallucination across different training methods and model families. (3) Demonstrating a fundamental reliability-capability trade-off, showing that current mitigation strategies reduce tool hallucination at the direct expense of reasoning performance, underscoring the need for novel training objectives that explicitly encode abstention and calibrate confidence. 2 Related Work LLMs as Tool-Using Agents Chain-of-Thought (CoT) prompting unlocked multi-step reasoning by guiding models to “think step by step” ( Wei et al. 2022 ) , enabling more agentic behaviors beyond single-turn prediction. Building on this, systems interleave internal reasoning with calls to external knowledge sources (e.g., search) for knowledge-intensive tasks, improving evidence acquisition and factual grounding ( Trivedi et al. 2023 ) . ReAct ( Yao et al. 2023 ) unifies reasoning and acting via interleaved thought traces and tool actions, creating a reason–act–observe loop that supports iterative planning, error correction, and information gathering. Complementarily, Toolformer ( Schick et al. 2024 ) demonstrates self-supervised tool learning, where LLMs synthesize supervision to learn when and how to call APIs across diverse functions. Collectively, these lines of work establish LLM agents that couple structured reasoning with purposeful tool use, enabling situated problem solving in open environments. Reinforcement Learning for Reasoning Early PPO-style approaches often incorporate process-aware supervision—e.g., token/step-level shaping that rewards correct intermediate reasoning—to stabilize training and encourage coherent multi-step plans ( Stiennon et al. 2020 ) . By contrast, newer algorithms such as Group Relative Policy Optimization (GRPO) optimize predominantly outcome-level signals by comparing groups of complete responses, yielding stronger and more stable credit assignment from result quality alone ( Shao et al. 2024 ; Guo et al. 2025 ) . This outcome-centric recipe delivers state-of-the-art results in mathematics (e.g., DeepSeek-Math). ( Shao et al.