ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learning

Zihao Feng, Xiaoxue Wang, Bowen Wu, Hailong Cao, Tiejun Zhao, Qun Yu, Baoxun Wang · 2025

Read ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learning by Zihao Feng, Xiaoxue Wang, Bowen Wu, Hailong Cao, Tiejun Zhao, Qun Yu, Baoxun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14718.txt.

Introduction

Zihao Feng and six co-authors introduce ToolSample in 2025, dual dynamic sampling with curriculum learning for reinforcement learning based tool training of language models. Easy samples flood tool training and give diminishing value as skill grows. Generic dynamic sampling ignores tool learning multi task structure and fine grained rewards.

Their DSCL framework samples by reward, favouring cases that still teach something, and orders learning as a curriculum across interdependent sub tasks. Multi valued reward functions guide which samples to keep or drop at each stage. Efficiency gains come from spending updates where marginal learning is highest. Tool using agents are only as good as their training mix, so smarter sampling is a direct lever on agent reliability.

Catalog record

CREATORZihao Feng, Xiaoxue Wang, Bowen Wu, Hailong Cao, Tiejun Zhao, Qun Yu, Baoxun Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14718

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learning Zihao Feng, Xiaoxue Wang, Bowen Wu, Hailong Cao, Tiejun Zhao, Qun Yu, Baoxun Wang Abstract While reinforcement learning (RL) is increasingly used for LLM-based tool learning, its efficiency is often hampered by an overabundance of simple samples that provide diminishing learning value as training progresses. Existing dynamic sampling techniques are ill-suited for the multi-task structure and fine-grained reward mechanisms inherent to tool learning. This paper introduces Dynamic Sampling with Curriculum Learning (DSCL), a framework specifically designed to address this challenge by targeting the unique characteristics of tool learning: its multiple interdependent sub-tasks and multi-valued reward functions. DSCL features two core components: Reward-Based Dynamic Sampling, which uses multi-dimensional reward statistics (mean and variance) to prioritize valuable data, and Task-Based Dynamic Curriculum Learning, which adaptively focuses training on less-mastered sub-tasks. Through extensive experiments, we demonstrate that DSCL significantly improves training efficiency and model performance over strong baselines, achieving a 3.29% improvement on the BFCLv3 benchmark. Our method provides a tailored solution that effectively leverages the complex reward signals and sub-task dynamics within tool learning to achieve superior results. Introduction Large Language Models (LLMs) have shown promising capabilities in complex tasks, including tool learning ( Qin et al. 2024 ; Kumar et al. 2025 ; Qu et al. 2025 ) . Tool learning aims to unleash the power of LLMs by enabling them to interact with external APIs to complete complex human instructions ( Tang et al. 2023 ; Zeng et al. 2025 ) . Meanwhile, reinforcement learning (RL) has emerged as a key strategy for enhancing the instruction-following and reasoning capabilities of LLMs, as demonstrated by the performance of models such as DeepSeek-R1 ( Guo et al. 2025 ) . This success has validated the viability of using RL to train agents for complex, multistep tasks, thereby accelerating their adoption for the specific challenge of tool learning ( Feng et al. 2025a ; Feng et al. 2025b ; Chen et al. 2025 ; Li, Zou, and Liu 2025 ) . Figure 1: Mean-Variance Scatter Plots of training rewards across different stages of ToolRL training. (Warmup: The stage where the model’s format prediction is almost correct; Middle and Later: The middle and final stages of training, respectively) Key observations include: 1) In the Middle and Later stages, the reward variance for many samples collapses towards zero; 2) Unlike the distribution (red) assuming a binary reward, where variance is a direct function of the mean, the actual mean-variance distribution (blue) of tool-learning exhibits a wide range of variances for any given mean. 3) Different sub-tasks (e.g., a(2) and c(2)) exhibit asynchronous convergence within the same stage. To better adapt RL for tool learning task, Qian et al. 2025 have proposed ToolRL method that incorporates independent reward design for tool selection, tool parameter key extraction, and parameter value completion to provide fine-grained feedback to the model. While effective, this approach is limited to improvements in the aspect of the reward, lacking any sample-level filtering to optimize the RL training process. Figure 1 illustrates the distribution of sample reward means and variances at different stages of ToolRL training. It shows that during the middle to late stages of training, the reward variance for many samples approaches zero, particularly for parameter key and value prediction (subfigure b(2-3) and c(2-3)). Previous research has demonstrated that such homogeneous samples provide limited benefit to training effectiveness, and when their proportion becomes excessive, they may even impede further model improvement ( Team et al. 2025 ; Bae et al. 2025 ) . To address this issue, sample selection methods that prioritize valuable data have proven effective in various RL tasks such as mathematics ( Yu et al. 2025 ) and code generation ( Liu et al. 2025 ) , but their application in tool learning remains unexplored in current public research. The tool learning task differs fundamentally from those tasks in two critical aspects: 1)it comprises multiple interdependent sub-tasks 2) it benefits more from fine-grained than coarse-grained reward mechanisms ( Qian et al. 2025 ) , which make existing dynamic sampling methods less suitable for tool learning task. This transition from a binary to a multi-valued reward fundamentally alters the relationship between the mean and variance. As Figure 1 shows, a binary reward mathematically locks the variance to the mean (red curves), making a single metric (either mean or variance) sufficient for evaluating data samples. In contrast, ToolRL’s multi-valued reward setting decouples them (blue scatter plots), allowing a given mean to correspond to a wide range of variances. This enables the mean and variance to provide independent signals from distinct dimensions to guide data sampling during training. Moreover, Figure 1 also illustrates that, sub-tasks (e.g., subfigures a(2) and c(2)) exhibit asynchronous convergence as shown in Figure 1 . Therefore, an effective dynamic sampling method designed for tool learning must: (1) fully consider multi-dimensional data information; (2) dynamically adjust sampling strategy based on the current training stage and the degree of convergence of different sub-tasks; and (3) balance the sample distribution across sub-tasks to avoid over-focusing on well-learned components at the expense of those that still require improvement. To fully exploit the potential of fine-grained reward mechanisms within tool learning, we propose customized dynamic sampling methods with curriculum learning (DSCL). This methodology is composed of two strategies: Reward-Based Dynamic Sampling (RDS) and Task-Based Dynamic Curriculum Learning (TDCL). Within RDS, we propose comprehensive metrics to capture the task’s reward diversity. These dynamically track three dimensions: the reward mean and variance of a group of rollouts, and the mean reward variance across the entire training. This multi-dimensional approach facilitates an assessment of not only the instantaneous difficulty and stability of the samples but also their evolutionary trajectories throughout the learning history, enabling more efficient and exploratory training. Additionally, the TDCL method emphasizes targeted sub-tasks according to the training status of samples, so as to help model training via providing diverse variances between rollouts. Overall, this paper offers the following contributions: • We introduce Dynamic Sampling with Curriculum Learning (DSCL), the first dynamic sampling method specifically tailored to the unique characteristics of tool learning. • DSCL consists of two novel sub-methods: RDS, which dynamically samples training data according to multiple dimensions of the mean and variance of rewards, and TDCL, which constructs a three-stage curriculum by exploiting sub-task dependencies. • We conduct comprehensive experiments to evaluate our DSCL method against several strong baselines. The results, supported by our in-depth analysis, confirm its significant effectiveness and superiority. Related Work Our research builds upon recent advancements in two key areas: the development of tool learning in LLMs and the optimization of the training process in RL. Tool Learning The ability of LLMs to interact with external tools has evolved significantly. Foundational work like ToolLLM ( Qin et al. 2023 ) pioneered the field by creating large-scale benchmarks for supervised fine-tuning. To overcome the limitations of static imitation, subsequent research has employed Reinforcement Learning to optimize sequential tool use, as seen in StepTool ( Yu et al. 2024 ) , Search-R1 ( Jin et al. 2025 ) , and Tool-N1 ( Zhang et al. 2025 ) , etc. The success of RL, however, is highly dependent

Read the complete text