Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning
Siyu Gong, Linan Yue*, Weibo Gao, Fangzhou Yao, Shimin Di, Lei Feng, Min-Ling Zhang · 2026
Read Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning by Siyu Gong, Linan Yue*, Weibo Gao, Fangzhou Yao, Shimin Di, Lei Feng, Min-Ling Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23032.txt.
Introduction
Siyu Gong and five co-authors repair the practice data used to teach tool-using language models. Tool-integrated reasoning lets a model call external tools while solving hard tasks. Training data for it are usually synthesized trajectories, filtered by scores and sparse outcome rewards, which give limited and biased guidance.
Their trajectory-guided method repairs weak trajectories and rewards good tool use more richly. Better practice paths should teach models when and how to use tools, not just final answers. That skill matters for agents that search, compute, and act.
Catalog record
| CREATOR | Siyu Gong, Linan Yue*, Weibo Gao, Fangzhou Yao, Shimin Di, Lei Feng, Min-Ling Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning Siyu Gong, Linan Yue*, Weibo Gao, Fangzhou Yao, Shimin Di, Lei Feng, Min-Ling Zhang Abstract Tool-Integrated Reasoning (TIR) enables large language models (LLMs) to solve complex tasks by interacting with external tools, yet existing approaches depend on high-quality synthesized trajectories selected by scoring functions and sparse outcome-based rewards, providing limited and biased supervision for learning TIR. To address these challenges, we propose AutoTraj, a two-stage framework that automatically learns TIR by repairing and rewarding tool-use trajectories. Specifically, in the supervised fine-tuning (SFT) stage, AutoTraj generates multiple candidate tool-use trajectories for each query and evaluates them along multiple dimensions. High-quality trajectories are directly retained, while low-quality ones are repaired using an LLM (i.e., LLM-as-Repairer). The resulting repaired and high-quality trajectories form a synthetic SFT dataset, while each repaired trajectory paired with its original low-quality counterpart constitutes a dataset for trajectory preference modeling. In the reinforcement learning (RL) stage, based on the preference dataset, we train a trajectory-level reward model to assess the quality of reasoning paths and combine it with outcome and format rewards, thereby explicitly guiding the optimization toward reliable TIR behaviors. Experiments on real-world benchmarks demonstrate the effectiveness of AutoTraj in TIR. Code is available at https://github.com/GSYY0607/AutoTraj_code . 1 Introduction In recent years, large language models (LLMs) have demonstrated remarkable capabilities in natural language processing and complex reasoning tasks, driving rapid advances across a wide range of artificial intelligence applications ( 13 ; 26 ; 42 ; 21 ; 17 ) . However, in real-world reasoning scenarios, models often need to interact with external environments to coordinate multiple abilities to solve complex problems, such as information retrieval ( 34 ) and code execution ( 16 ) . To this end, Tool-Integrated Reasoning (TIR) ( 27 ; 11 ; 39 ; 2 ) has been proposed to enable models to invoke external tools during reasoning, thereby extending their problem-solving capabilities and improving practical performance. Table 1 : Differences between existing two-stage TIR paradigms and AutoTraj. Compared with previous methods, AutoTraj repairs low-quality TIR trajectories in Stage 1 for SFT and introduces trajectory-aware reinforcement learning in Stage 2 by leveraging automatically synthesized positive and negative trajectory pairs, enabling more scalable TIR. Property Existing Two-stage TIR AutoTraj (ours) Stage 1: Supervised Fine-tuning (SFT) Synthesized trajectories ✓ ✓ High-quality TIR trajectories Retain Retain Low-quality TIR trajectories Filter Repair Stage 2: Reinforcement Learning (RL) Reward target Answer Answer+Trajectory Format Reward ✓ ✓ Outcome Reward ✓ ✓ Trajectory Reward ✗ ✓ Trajectory RM Training \varnothing Synthesized trajectory pairs With the recent progress of large reasoning models (e.g., DeepSeek R1 ( 12 ) ) in long-chain reasoning and complex decision-making, researchers have begun to transfer these reasoning paradigms to the TIR setting. Specifically, existing approaches typically adopt a two-stage training paradigm to enhance TIR abilities ( 8 ; 9 ; 10 ) . As shown in Table 1 , In Stage 1 , models are initialized through supervised fine-tuning (SFT) on tool-use trajectories to achieve a cold start. In Stage 2 , the model is further optimized using reinforcement learning (RL) with outcome-based rewards to improve its ability to leverage tools for solving complex multi-step reasoning tasks. While this paradigm improves TIR capabilities to some extent, it still faces substantial challenges in practice. Specifically, in the first, during the SFT stage , acquiring high-quality expert tool trajectories is costly and difficult to scale ( 41 ; 30 ; 1 ) . Therefore, as shown in Table 1 , most existing approaches ( 8 ; 10 ) adopt automatically synthesized trajectories as a practical alternative to human-labeled ones, typically following a pipeline that generates a large pool of candidates and then selects higher-quality samples via heuristic scoring. However, this “synthesize-then-filter” paradigm tends to reduce data diversity. Specifically, the synthesis model is usually proficient in only certain reasoning patterns, producing high-quality trajectories in familiar cases but low-quality ones in challenging scenarios. Simply filtering these imperfect samples may lead to a biased training distribution, limiting robustness and generalization. Moreover, low-quality trajectories are not merely noise but informative negative examples, filtering them out wastes valuable supervision signals for distinguishing correct from incorrect reasoning paths. Second, during the RL stage , training strategies that rely solely on outcome-based rewards suffer from reward sparsity ( 10 ; 24 ; 7 ; 25 ) . Such rewards focus exclusively on the correctness of the final answer while ignoring the quality of the underlying TIR path. As a result, trajectories with flawed reasoning but accidentally correct outcomes may be rewarded, whereas trajectories with sound reasoning but minor final errors may be penalized, hindering the learning of stable and reliable tool-integrated reasoning behaviors. In this paper, we propose AutoTraj , a novel framework that learns tool-integrated reasoning from Auto matically repairing and rewarding tool-use Traj ectories. AutoTraj also follows a two-stage training paradigm. In the SFT stage , our goal is to synthesize high-quality tool-use trajectories. Specifically, given an input query, we first use a base model equipped with tool-calling capabilities to generate multiple candidate reasoning trajectories that involve tool usage. These trajectories are then jointly evaluated along multiple dimensions, including answer correctness, reasoning confidence, trajectory length, and text repetition, and categorized into high-quality and low-quality groups. Instead of simply discarding low-quality trajectories, we introduce a trajectory repair mechanism that leverages an LLM to repair and complete their TIR trajectories (LLM-as-Repairer). The repaired trajectories, together with the original high-quality trajectories, form a high-quality dataset \mathcal{D}_{\text{sft}} for SFT, which preserves diverse reasoning patterns from originally low-quality samples. Meanwhile, we further construct positive–negative trajectory pairs by pairing each repaired trajectory with its corresponding original low-quality trajectory, resulting in an auxiliary dataset \mathcal{D}_{\text{self}} that provides fine-grained supervision for subsequent reward modeling. At this stage, SFT on \mathcal{D}_{\text{sft}} improves the model’s basic TIR capabilities. In the RL stage , to mitigate reward sparsity, we first train a Trajectory Reward Model (Trajectory RM) on \mathcal{D}_{\text{self}} to distinguish high-quality and low-quality tool-use trajectories. We then optimize the model using Group Relative Policy Optimization (GRPO) ( 31 ) . In addition to standard format and outcome rewards, we incorporate reward scores from the Trajectory RM to evaluate the generated reasoning trajectories. This design enables the optimization process to explicitly account for reasoning path quality, alleviating reward sparsity and guiding the model toward more robust and reliable TIR strategies. Table 1 summarizes the differences between the existing two-stage TIR methods and our proposed AutoTraj. Our main contributions are summarized as follows: • We propose AutoTraj, a two-stage training framework for tool-integrated reasoning that reuses low-quality trajectories via automatic repair and incorporates trajectory-level rewards into RL, thereby providing more informative supervision and mitigating reward sparsity. • We construct two complementary datasets: a high-quality tool-use trajectory dataset for SFT, and a trajectory pair dataset consisting of positive–negative trajectory pairs for training a trajectory reward model. The learned reward model can also provide fine-grained trajectory-level rewards for training other TIR models. • Extensive experiments on diverse benchmark datasets, covering mathematical reasoning and knowledge-intensive reasoning, show that the proposed framework consistently achieves substantial improvements in both tool-use capability and reasoning quality. 2 Related Work 2.1 Tool-Integrated Reasoning Tool-Integrated Reasoning (TIR) ( 30 ; 27 ; 23