Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
Yabo Zhang Yihan Zeng Qingyun Li Zhen Hu Kavin Han Wangmeng Zuo · 2025
Read Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use by Yabo Zhang Yihan Zeng Qingyun Li Zhen Hu Kavin Han Wangmeng Zuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12867.txt.
Introduction
Yabo Zhang and 5 co-authors present Tool-R1 in a 2025 paper on teaching language models to use tools efficiently. Language models reason well about text but remain limited on real-world tasks needing up-to-date knowledge, precise operations, or specialized tools. Reinforcement learning offers a way to train for such multi-step tool use.
Tool-R1 enables general, compositional, multi-step tool use by generating executable Python code. It integrates user-defined tools and standard libraries, sharing variables across steps to build coherent workflows. Its reward combines a language-model judgment of the answer with whether the code executed successfully. A dynamic sample queue caches and reuses high-quality training trajectories, cutting costly online sampling. Experiments on the GAIA benchmark showed substantial gains in accuracy and robustness.
Catalog record
| CREATOR | Yabo Zhang Yihan Zeng Qingyun Li Zhen Hu Kavin Han Wangmeng Zuo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12867
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use Yabo Zhang Yihan Zeng Qingyun Li Zhen Hu Kavin Han Wangmeng Zuo Abstract Large language models (LLMs) have demonstrated strong capabilities in language understanding and reasoning, yet they remain limited when tackling real-world tasks that require up-to-date knowledge, precise operations, or specialized tool use. To address this, we propose Tool-R1, a reinforcement learning framework that enables LLMs to perform general, compositional, and multi-step tool use by generating executable Python code. Tool-R1 supports integration of user-defined tools and standard libraries, with variable sharing across steps to construct coherent workflows. An outcome-based reward function, combining LLM-based answer judgment and code execution success, guides policy optimization. To improve training efficiency, we maintain a dynamic sample queue to cache and reuse high-quality trajectories, reducing the overhead of costly online sampling. Experiments on the GAIA benchmark show that Tool-R1 substantially improves both accuracy and robustness, achieving about 10% gain over strong baselines, with larger improvements on complex multi-step tasks. These results highlight the potential of Tool-R1 for enabling reliable and efficient tool-augmented reasoning in real-world applications. Our code will be available at https://github.com/YBYBZhang/Tool-R1 . Figure 1 : Example of multi-step tool call. Tool-R1 supports compositional and customizable multi-step tool invocation through executable Python code, with the ability to perform reflection based on environmental feedback. 1 Introduction Large language models (LLMs) [ 14 , 4 , 22 , 8 , 28 ] have made significant progress in natural language understanding and generation, especially in challenging tasks such as mathematical reasoning and code generation. These strengths stem from the internal knowledge acquired during large-scale pretraining. However, LLMs still perform poorly on real-world problems that require up-to-date facts or specialized expertise. For example, they may struggle to query the latest weather conditions or accurately analyze massive structured data. To address these limitations, recent works [ 32 , 23 , 11 , 31 , 25 , 18 ] have focused on equipping LLMs with external tools such as search engines and code interpreters, so they can access accurate information and perform precise operations beyond their built-in knowledge. Existing methods enable LLMs to use external tools during the reasoning process through prompt engineering [ 32 , 11 , 25 , 17 ] or reinforcement learning (RL) [ 9 , 10 , 35 ] , but both approaches face fundamental limitations in handling dynamic real-world scenarios. Prompt engineering methods rely entirely on the internal knowledge to determine tool usage, making them inherently unable to adapt to the diverse and unpredictable feedback from real-world environments. This leads to poor robustness when encountering unexpected responses or environmental changes. While RL-based methods demonstrate superior capability in learning from various types of environmental feedback, they face two critical scalability challenges. First, current methods typically rely on JSON-formatted tool calls that are restricted to predefined user-provided tools or APIs, limiting their ability to leverage diverse tool combinations and create custom tool configurations necessary for complex real-world tasks. Second, the prohibitive cost of tool execution during training further confines existing RL approaches to specific domains ( e.g . , retrieval task [ 10 ] ) and prevents broader applicability. In this work, we introduce Tool-R1, a novel method that supports general and compositional tool use across complex real-world tasks, with LLMs trained through sample-efficient reinforcement learning. Unlike prior RL-based methods using JSON-formatted tool calls, Tool-R1 first adopts a more flexible and general tool-use mechanism by generating executable Python code. This design not only supports user-defined tools ( e.g . , web browser and multimedia parser) and standard Python libraries, but also enables intermediate variable sharing across steps, allowing LLMs to construct coherent and multi-round tool-use workflows. During fine-tuning, we employ an outcome-based reward function to encourage LLMs to freely explore when and how to use appropriate tools: an LLM judges whether the predicted answers align with open-ended reference answers, while the success rate of code execution serves as an auxiliary reward to promote reliability and correctness. However, training such tool-use policies through reinforcement learning can be time- and resource-intensive, as it often involves frequent online sampling and tool execution during fine-tuning. To alleviate this burden, Tool-R1 maintains a dynamic sample queue that caches recent high-quality trajectories, which are then reused to improve sample efficiency and stabilize policy learning. To evaluate the effectiveness of Tool-R1 in solving real-world tasks, we fine-tuned the model on approximately 1,300 moderately difficult question-answer pairs selected from the MAT-Agent and QA datasets [ 2 , 30 , 5 ] . Experimental results show that Tool-R1 significantly improves the performance of large language models, achieving \sim 10% accuracy gain over the Qwen-2.5-Instruct baselines on the GAIA benchmark. The improvement is particularly notable on complex questions that require multi-step reasoning and reliable tool use. Additionally, we observe that Tool-R1 enhances the robustness of LLM by improving its ability to reflection in the tool invocation process. To summarize, our core contributions are as follows: • We propose Tool-R1 , a novel reinforcement learning approach that empowers LLMs to solve complex real-world tasks through multi-step tool use via executable code generation. • We present a flexible Code Execution Tool Chain that enables unified multi-step tool use via executable Python code with variable sharing. To guide effective multi-step reasoning with multiple tools, we design an outcome-driven reward that encourages tool integration. • Tool-R1 introduces a dynamic sample queue for efficient GRPO training, caching the latest high-quality trajectories to reduce cost of online sampling. To ensure stable and effective updates, we further adopt difficulty-aware sampling within the dynamic queue. • The experiments show that Tool-R1 greatly improves LLM performance on real-world tasks, achieving notable gains on complex tasks through code-based tool use. 2 Related Work Reinforcement Learning in Language Models. Reinforcement learning (RL) has become a powerful tool for aligning and enhancing large language models (LLMs). Early methods like RLHF and InstructGPT [ 14 ] optimize LLMs based on reward models learned from human preferences using PPO [ 21 ] , but require complex multi-stage training and high computational cost. To improve efficiency, alternatives such as DPO [ 19 ] , SimPO [ 13 ] , and GRPO [ 22 ] have been proposed to simplify the training pipeline and enhance sample efficiency and stability. Lightweight REINFORCE-style methods such as RLOO [ 1 ] and REINFORCE++ [ 6 ] further reduce implementation complexity. Beyond alignment, recent works apply RL to improve the reasoning and decision-making abilities of LLMs. DeepSeek-R1 [ 3 ] and SimpleRL-Zoo [ 33 ] directly fine-tune base models with step-wise or outcome-driven reward signals to support long-form reasoning or multi-step planning. DeepScaler [ 12 ] and Light-R1 [ 27 ] explore curriculum-style rewards and scaling strategies to enhance reasoning depth and sample efficiency. Additionally, advanced RL algorithms like GRPO and RLOO have been extend to induce better reasoning behaviors of multimodal language models [ 7 , 29 , 16 , 34 ] , as well as improving their performance on visual tasks like object counting, visual math, and multimodal reasoning. However, existing RL methods encounter substantial efficiency bottlenecks in tool-use scenarios due to the prohibitive cost of tool execution during training. Our work addresses this challenge by introducing a dynamic sample queue mechanism that significantly reduces training costs