Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning

Youtu-Agent Team ∗ · 2025

Read Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning by Youtu-Agent Team ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22601.txt.

Introduction

Yulei Qin and 15 co-authors train tool-using language agents by letting them copy their own wins. Reinforcement learning sharpens such agents, but balancing exploration and exploitation over long tasks is tricky. Pushing entropy up mechanically can destabilize training as multi-turn data shifts. Their recipe, SPEAR, stores good past episodes in a replay buffer for self-imitation, while a curriculum steers entropy across stages.

Early stages reward frequent tool use to speed exploration. Later stages favor proven tactics as the agent masters the setting. On ALFWorld and WebShop, SPEAR lifted success rates for several base methods by wide margins. Math contest gains were smaller but real, at minor extra cost.

Catalog record

CREATORYoutu-Agent Team ∗
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning Youtu-Agent Team ∗ Abstract Reinforcement learning (RL) is the dominant paradigm for sharpening strategic tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks, yet it faces a fundamental challenge of exploration-exploitation trade-off. Existing studies stimulate exploration through the lens of policy entropy, but such mechanical entropy maximization is prone to RL training instability due to the multi-turn distribution shifting. In this paper, we target the progressive exploration-exploitation balance under the guidance of the agent’s own experiences without succumbing to either entropy collapsing or runaway divergence. We propose SPEAR , a curriculum-based self-imitation learning (SIL) recipe for training agentic LLMs. It extends the vanilla SIL framework, where a replay buffer stores self-generated promising trajectories for off-policy update, by gradually steering the policy evolution within a well-balanced range of entropy across stages. Specifically, our approach incorporates a curriculum to manage the exploration process, utilizing intrinsic rewards to foster skill-level exploration and facilitating action-level exploration through SIL. At first, the auxiliary tool call reward plays a critical role in the accumulation of tool-use skills, enabling broad exposure to the unfamiliar distributions of the environment feedback with an upward entropy trend. As training progresses, self-imitation gets strengthened to exploit existing successful patterns from replayed experiences for comparative action-level exploration, accelerating solution iteration without unbounded entropy growth. To further stabilize training, we recalibrate the advantages of experiences in the replay buffer to address the potential policy drift. Regularizations such as the clipping of tokens with high covariance between probability and advantage are introduced to the trajectory-level entropy control to curb over-confidence. We also combine bag-of-tricks, which are effective industrial optimizations of agentic RL, into a strong baseline, Dr.BoT, to demonstrate the practical effectiveness of our proposed SPEAR . In ALFWorld and WebShop, SPEAR increases the success rates of GRPO/GiGPO/Dr.BoT by up to 16.1%/5.1%/8.6% and 20.7%/11.8%/13.9%, respectively. In AIME24 and AIME25, SPEAR boosts Dr.BoT by up to 3.8% and 6.1%, respectively. Such gains incur only 10%–25% extra theoretical complexity and negligible runtime overhead in practice, demonstrating the plug-and-play scalability of SPEAR . Codes and checkpoints will be available soon. \correspondence {yuleiqin, arthurtan}@tencent.com * * footnotetext: Full author list in contributions. 1 Introduction Reinforcement Learning (RL) [ 1 , 2 , 3 ] has driven the development of reasoning capabilities of Large Language Models (LLMs). Built upon the reason-and-act (ReAct) paradigm [ 4 ] , LLMs have powered various agentic applications such as simulated robot navigation [ 5 , 6 ] , mobile assistant [ 7 , 8 ] , web navigator [ 9 , 10 ] , deep searcher [ 11 , 12 , 13 ] , and GUI master [ 14 , 15 ] . A fundamental challenge in applying RL to LLM agents, especially for long-horizon tasks with sparse rewards, is managing the balance between exploration and exploitation. The LLM agent needs to exploit both its pretrained knowledge and feedback from past interactions to identify and refine strategies that maximize ultimate reward. At the same time, it must explore novel behaviors, using different tools to discover more effective solutions, which involves extending text-based reasoning to tool-augmented reasoning. The interweaving between exploration and exploitation determines the emerging agent’s competence upon convergence. Figure 1 : The core concept of our proposed SPEAR for training long-horizon LLM agents via group-based RL. Compared with the vanilla GRPO-like algorithms, we introduce the curriculum-based self-imitation learning with intrinsic reward shaping. Given the same data input, a group of trajectories are generated with multi-turn tool interactions and then engaged for episode-level reward computation and advantage estimation. Then, we propose filtering valuable good trajectories to update the replay buffer, where the stored past experiences guide the agent to explore effectively on sparsely rewarded tasks via self-imitation. The total training batch contains both on-policy and off-policy data from the replay buffer. With self-guided progressive exploration, SPEAR boosts performance of various LLMs and baselines on ALFWorld, WebShop, and AIME24/25 in a plug-and-play manner. Existing studies often quantify the exploration potential through entropy [ 16 , 17 , 18 ] , where the decline of policy entropy indicates the over-confidence of the current policy with insufficient exploration. In this case, a series of regularization techniques [ 19 , 20 , 21 ] have been proposed to maximize entropy, particularly in the context of agent training [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ] . However, when it comes to LLM-driven agents, entropy-based control is fragile: the accumulation of low-probability tokens from the environment feedback induces severe distribution shifting, often leading to mode collapse [ 18 , 30 ] . Unlike conventional agents, these models may experience sustained entropy growth due to policy over-uncertainty during multi-turn interactions with external tools. As a result, the policy struggles to exploit tool-use skills effectively, and training instability becomes frequent [ 31 , 32 , 33 ] . Recent approaches attempt to mitigate this issue by relying either on cold-start supervised fine-tuning (SFT) [ 13 , 14 , 34 , 35 ] or on hybrid schemes that combine RL with SFT [ 36 ] . Although these methods improve stability, they compromise RL’s ability to discover strategies beyond those present in the SFT corpus. This limitation highlights the need for adaptive training frameworks that can dynamically schedule LLM-driven agents to decide when to explore and when to exploit . In this paper, we are trying to answer the following core research question: During RL of agentic LLMs, can we schedule a smooth transition between exploration and exploitation guided by the policy’s own experiences without going to extremes of either entropy collapsing or runaway divergence? We hypothesize that policy entropy should serve as a progressive guide for balancing exploration and exploitation throughout training. Specifically, the agent should maintain its entropy within a dynamic but controlled range that evolves over time: • 1) At the early stages of training, increasing entropy is beneficial for supporting broad skill-level exploration and rapidly developing tool-use capabilities. The agent is expected to encounter unfamiliar observations, engage in trial-and-error interactions, and discover diverse reasoning trajectories. • 2) As training advances, however, a shift toward converging entropy is required. This enables the agent to consolidate problem-solving heuristics and emphasize action-level exploration. In this phase, the agent exploits reward signals to choose comparatively more effective actions and adapts to changing trajectory distributions, thereby stabilizing its evolutionary path. Figure 2 : Overview of SPEAR in terms of data flow. During each episode, the agent interacts with the environment to generate a set of trajectories. These trajectories are processed along two complementary paths. First, they are used for intrinsic reward shaping, advantage estimation, and on-policy updates, following a mechanism similar to the vanilla GRPO. Second, they are selectively filtered and stored in a replay buffer, enabling off-policy updates through the proposed self-imitation scheme with advantage recalibration and regularization. This dual integration allows the agent to maximize the utility of rewarding past experiences, thereby expanding the exploration space effectively, while simultaneously mitigating persistent over-uncertainty in decision-making under shifting distributions of external feedback. As a result, SPEAR achieves a stable balance between exploration and exploitation through self-guided policy

Read the complete text