AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization
Zhaiyu Fang, Ruipeng Sun · 2026
Read AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization by Zhaiyu Fang, Ruipeng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14696.txt.
Introduction
Zhaiyu Fang and Ruipeng Sun study tool use by language models in this 2026 paper. Tool-Integrated Reasoning has significantly enhanced the capabilities of large language models. Current agents, though, tend toward cognitive offloading. They redundantly invoke external tools even for simple tasks. Each call costs time and money, and can add errors.
The authors argue that true agentic intelligence requires adaptive wisdom: knowing when to use a tool and when to answer directly. Their method, AdaTIR, uses difficulty-aware policy optimization. The agent learns to match tool use to task difficulty. Readers get a training scheme for agents that reach for tools only when the problem earns it.
Catalog record
| CREATOR | Zhaiyu Fang, Ruipeng Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14696
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization Zhaiyu Fang, Ruipeng Sun Abstract Tool-Integrated Reasoning (TIR) has significantly enhanced the capabilities of Large Language Models (LLMs), yet current agents tend to exhibit cognitive offloading, redundantly invoking external tools even for simple tasks. In this paper, we suggest that true agentic intelligence requires not just tool invocation, but the adaptive wisdom to discern when to use them. We propose AdaTIR, a framework that shifts the paradigm from static tool invocation to difficulty-aware reasoning internalization. By introducing a difficulty-aware efficiency reward, AdaTIR dynamically adjusts tool budgets based on task complexity—internalizing reasoning for simple tasks while selectively invoking tools for complex tasks. Furthermore, we identify a sign reversal problem where tool penalties outweigh correctness rewards, mistakenly penalizing correct rollouts with negative advantages. To resolve this, we propose Clipped Advantage Shaping (CAS), which ensures that correctness remains the primary objective while using efficiency as a secondary constraint. Empirical results demonstrate that AdaTIR reduces tool calls by up to 97.6% on simple tasks and 28.2% on complex challenges while maintaining or enhancing accuracy. Notably, AdaTIR successfully internalizes reasoning, outperforming baselines by 4.8% on AIME 2024 even when tool access is strictly disabled. 1 Introduction Figure 1: Accuracy versus Tool Calls for Qwen2.5 under different training strategies. GRPO and AdaTIR denote the baseline and our proposed method, respectively. AdaTIR consistently achieves a superior efficiency-accuracy trade-off across both 3B and 7B backbones by internalizing reasoning capabilities. Large Language Models (LLMs)—such as OpenAI-o1 ( 20 ) and DeepSeek-R1 ( 5 ) —have achieved remarkable milestones in complex reasoning, manifesting superior capabilities in tasks ranging from mathematical problem-solving ( 21 ) to repository-level code generation ( 9 ) and open-domain question answering ( 12 ) . To further transcend the limitations of parametric knowledge, Tool-Integrated Reasoning (TIR) has emerged as a powerful paradigm ( 7 ; 15 ; 6 ; 28 ; 34 ) , augmenting Large Language Models (LLMs) with external executors to tackle problems intractable for pure-text generation ( 16 ) . However, this augmentation comes with a hidden cognitive cost, current agents exhibit a pathological tendency toward “cognitive offloading” ( 29 ; 33 ) . By frequently invoking tools for simple tasks, they not only incur unnecessary latency but also risk degrading their intrinsic reasoning consistency through environmental noise. The fundamental challenge lies in selectively internalizing tool-independent logic without compromising the robustness needed for genuinely complex tasks. To address these efficiency bottlenecks, recent research has explored ranging from inference-time budget constraints ( 18 ) to training-time reward shaping. However, these approaches often fail to address the root cause of tool dependency. Inference-time constraints function as shallow prompt-level limits, often leading to context drift or reasoning dead-ends when the model lacks the internalized logic to proceed without tools ( 17 ; 27 ) . Alternatively, training-based methods like OTC impose indiscriminate penalties on tool invocation. This creates a fundamental dilemma: a penalty strong enough to suppress redundancy on simple tasks often inadvertently impairs the model’s ability to solve complex tasks that genuinely require external assistance. Furthermore, approximating optimal budgets based on biased group-minima estimates often leads to sub-optimal convergence where tool redundancy persists. To address these limitations, we propose AdaTIR , a framework that establishes a difficulty-aware policy to adaptively regulate tool invocation frequency based on task complexity. Our core insight is that tool invocation is inherently difficulty-dependent. Different from static penalties that indiscriminately suppress actions, we suggest that constraints should be dynamic: stricter budgets on simple tasks compel the model to internalize reasoning capabilities and mitigate “cognitive offloading”, while relaxed constraints on complex tasks ensure the model retains the robustness to leverage external tools when necessary. Consequently, this paradigm enables the model to align computational expenditure with task difficulty, effectively pushing the cost-performance Pareto frontier. The implementation of this strategy, however, reveals a critical algorithmic challenge: naive reward shaping inherently risks sign reversal, where efficiency penalties can inadvertently outweigh correctness rewards. Specifically, as training progresses, rollouts within a group exhibit consistent correctness. This phenomenon dilutes the accuracy-based advantage signal, allowing the auxiliary efficiency signal to disproportionately influence the optimization objective. Consequently, even rollouts that yield correct answers could be assigned a negative advantage if their tool invocation exceeds the group average, creating an objective misalignment that suppresses valid reasoning paths and induces training instability. To address this, we propose Clipped Advantage Shaping (CAS) , a mechanism that directly reformulates the advantage term rather than modifying the scalar reward. By integrating a clipped efficiency penalty as an auxiliary signal, CAS effectively encourages the prioritization of correctness signals over efficiency signals, causing the model to optimize efficiency without suffering from the mode collapse ( 32 ; 4 ) observed in conventional methods. Our contributions are summarized as follows: • We propose AdaTIR, a framework that establishes a difficulty-aware policy to adaptively regulate tool invocations tailored to task complexity, effectively suppressing redundant "cognitive offloading" on simple tasks. • We introduce Clipped Advantage Shaping (CAS), a mechanism that mathematically reformulates the advantage term to maintain sign consistency, resolving the training instability and mode collapse often observed in naive reward shaping. • We demonstrate that our approach significantly enhances efficiency across varying difficulty levels, reducing tool calls by up to 97.6% (Figure 1 ) on simple tasks while maintaining robust performance on complex challenges. Notably, our method exhibits superior reasoning capabilities even when tool access is strictly disabled, indicating successful reasoning internalization. Figure 2: Overview of the proposed framework. Efficiency signals are conditionally injected only for correct rollouts on easy tasks (where difficulty estimation \phi falls below threshold \phi_{low} ). These auxiliary signals are explicitly clipped by \delta|A_{acc}| ( 0<\delta<1 ), to ensure efficiency optimization remains secondary to correctness, thereby maintaining sign consistency. 2 Preliminaries Tool-Integrated Reasoning (TIR). In the TIR paradigm, the large language model acts as a reasoning engine that interleaves text generation with external tool interactions ( 31 ; 24 ) . Given a task q , the reasoning process of the policy \pi_{\theta} forms a trajectory \tau consisting of sequential reasoning steps and environment feedbacks: \tau=(w_{1},c_{1},o_{1},w_{2},c_{2},o_{2},\dots,w_{K},c_{K},o_{K},\hat{y}) (1) where w_{k} represents the k -th reasoning thought, c_{k} is the tool invocation command, and o_{k} is the corresponding observation returned by code execution. Group Relative Policy Optimization (GRPO). We adopt GRPO as our base optimization framework. In contrast to traditional Actor-Critic ( 11 ) methods that require a value function, GRPO estimates the baseline directly from group scores, reducing memory overhead. The training objective is defined as: \displaystyle\mathcal{J}_{\text{GRPO}} \displaystyle=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|\tau_{i}|}\sum_{t=1}^{|\tau_{i}|}\bigg(\min\big(r_{i,t}\hat{A}_{i,t}, (2) \displaystyle\text{clip}(r_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\big)-\beta D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})\bigg) where the importance sampling ratio r_{i,t} is explicitly given by: r_{i,t}=\frac{\pi_{\theta}(\tau_{i,t}\mid q,\tau_{i,<t})}{\pi_{\text{old}}(\tau_{i,t}\mid q,\tau_{i,<t})} (3) Here, \tau_{i,t} denotes the t -th token in trajectory \tau_{i} , and \hat{A}_{i,t} represents the normalized advantage computed within group i . Figure 3: Comparison of reasoning processes between a standard tool-integrated baseline and our approach. (Left) The baseline model relies on external Python tools but fails to recover after the symbolic solver returns an error. It then attempts an irrelevant search and ends with an incorrect answer. (Right) Our approach correctly solves the problem by performing step-by-step algebraic deduction internally, reaching the correct result without depending on external code execution. 3 Method In this section, we present AdaTIR, a framework designed for