Dynamic Dual-Granularity Skill Bank for Agentic RL
Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dongbin Zhao, Institute of Automation, Chinese Academy of Sciences, Pengcheng Laboratory Sun Yat-Sen University · 2026
Read Dynamic Dual-Granularity Skill Bank for Agentic RL by Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dongbin Zhao, Institute of Automation, Chinese Academy of Sciences, Pengcheng Laboratory Sun Yat-Sen University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28716.txt.
Introduction
Tu and co-authors propose D2Skill, a dynamic skill bank for agentic reinforcement learning. Reusable experience can help agents, but past methods mostly kept trajectory-level guidance. D2Skill stores task skills for high-level guidance and step skills for fine-grained support and error correction. Paired rollouts, with and without skills, give hindsight utility signals. Those signals update both the skills and the policy. Tests on ALFWorld, WebShop, and search-based QA beat skill-free baselines across model scales.
Catalog record
| CREATOR | Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dongbin Zhao, Institute of Automation, Chinese Academy of Sciences, Pengcheng Laboratory Sun Yat-Sen University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32934 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28716
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Dual-Granularity Skill Bank for Agentic RL Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dongbin Zhao, Institute of Automation, Chinese Academy of Sciences, Pengcheng Laboratory Sun Yat-Sen University Abstract Agentic reinforcement learning (RL) can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill , a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld and WebShop with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 show that D2Skill consistently improves success rates over skill-free baselines by 10–20 points. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead. Project Page: https://github.com/TU2021/D2Skill-AgenticRL . (a) D2Skill Framework (b) Main Results (c) Training Curves of Success Rate (d) Skill Bank Dynamics of Utility Figure 1 : Overview of D2Skill. (a) The dynamic dual-granularity skill bank with retrieval, reflection-driven generation, and management. (b) Overall results on ALFWorld and WebShop . (c) ALFWorld training curves for the D2Skill skill group, paired baseline group, and GRPO . (d) Skill bank dynamics with and without management, shown by average skill utility and retrieval statistics. 1 Introduction Agentic reinforcement learning (RL) has recently emerged as a promising paradigm for training language-based agents to solve long-horizon decision-making tasks, including interactive environments ( 8 ) , web search ( 33 ) , and research scenarios ( 18 ) . In these settings, the policy interacts with the environment through a textual interface and selects actions based on the task description together with a limited history of past observations and actions. However, such history-based context is generally not a sufficient statistic of the underlying state, resulting in severe partial observability and making credit assignment increasingly difficult as the decision horizon grows ( 30 ) . Under sparse rewards and large action spaces, learning each task in isolation is highly inefficient ( 6 ) . Effective policies therefore require mechanisms for accumulating reusable knowledge that can be transferred across tasks. Recent studies alleviate these challenges by introducing additional supervision signals for agentic RL. Some methods employ outcome-based credit assignment to provide process rewards ( 6 ) , while others derive hindsight supervision from completed trajectories ( 27 ) . More recent work focuses on enabling agents to accumulate experience across tasks and iteratively refine it during training ( 28 ; 3 ; 2 ) . Within this line, reusable skills have emerged as an effective form of past experience and shown strong empirical gains in agentic RL ( 19 ) . For instance, SkillRL ( 21 ) builds a skill bank from past trajectories and retrieves relevant skills to guide policy interaction, improving exploration efficiency in long-horizon tasks. However, existing skill-based and reflection-driven frameworks remain limited in two respects. Most methods derive skills from complete trajectories and emphasize task-level reflection, which captures high-level guidance but is less effective for correcting fine-grained errors at individual interaction steps ( 21 ; 31 ) . In addition, as training progresses, the skill bank expands continuously, making retrieval and management increasingly challenging. Without principled mechanisms for skill evaluation and pruning, redundant or ineffective skills can degrade retrieved guidance and hinder policy optimization ( 36 ; 37 ) . To address these limitations, we propose D ynamic D ual-Granularity Skill Bank ( D2Skill ) for agentic RL, which maintains reusable skills at both the task and step granularities throughout training. As illustrated in Figure 1(a) , D2Skill distinguishes between task skills for high-level task guidance and step skills for fine-grained decision support and local error correction during interaction. During training, it contrasts skill-injected and non-injected trajectories under the same policy to estimate hindsight skill utility, which in turn guides skill maintenance, retrieval, and policy optimization; the overall training behavior is further reflected in the success-rate curves in Figure 1(c) . Meanwhile, D2Skill continuously expands and refines the skill bank through reflection while pruning redundant or ineffective skills, keeping the memory compact, informative, and beneficial throughout training, as further illustrated by the skill-bank dynamics in Figure 1(d) . Experiments on representative agentic tasks show that D2Skill consistently outperforms both non-skill baselines and existing skill-augmented methods (Figure 1(b) ), while effectively maintaining a dynamically updated skill bank with high utility throughout training. The main contributions of this work are as follows: 1. We present D2Skill , a dynamic dual-granularity skill bank framework for agentic RL, which organizes reusable experience into task skills for high-level guidance and step skills for fine-grained interaction support. 2. We develop a joint training paradigm where the skill bank and policy co-evolve, with skills expanded through reflection and maintained via utility-guided retrieval and pruning , ensuring an efficient memory throughout training. 3. We evaluate D2Skill on ALFWorld and WebShop using Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 . D2Skill achieves 10–20 point gains in success rate over skill-free baselines (GRPO), and ablations validate its core components. 2 Preliminaries 2.1 Agentic RL as a History-Augmented Decision Process We consider agentic RL in long-horizon environments modeled as a Markov decision process (MDP) \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma) , where s_{t}\in\mathcal{S} , a_{t}\in\mathcal{A} , and (s_{t+1},r_{t},d_{t})\sim P(\cdot\mid s_{t},a_{t}) . Unlike classical RL, the policy does not directly observe the environment state. Instead, the agent interacts through a textual interface that provides a partial description of the task and the interaction history. For a task instance g , let \tau_{g} denote the task specification, o_{t} the textual observation at step t , and \mathcal{H}_{t}^{L}=\{(o_{t-L},a_{t-L}),\dots,(o_{t-1},a_{t-1})\} the most recent L observation–action pairs retained in the prompt. Let \mathcal{A}_{t}^{\mathrm{adm}}\subseteq\mathcal{A} be the admissible action set. The policy acts on the effective context x_{t}=(\tau_{g},\mathcal{H}_{t}^{L},o_{t},\mathcal{A}_{t}^{\mathrm{adm}}) , and selects actions according to \pi_{\theta}(a_{t}\mid x_{t}) . Although the underlying dynamics are Markovian in (s_{t},a_{t}) , the context x_{t} is a fixed-window summary of past interactions and is generally not a sufficient statistic of the latent state, so the resulting MDP can be viewed as a history-augmented partially observable MDP. 2.2 Skill Bank as an External Knowledge Store In addition to the sliding window \mathcal{H}_{t}^{L} , we maintain a persistent skill bank \mathcal{M} , where each skill m\in\mathcal{M} stores language guidance for decision making. At step t , a retrieval operator selects a small set of relevant skills m_{t}\subseteq\mathcal{M} conditioned on the current context x_{t}=(\tau_{g},\mathcal{H}_{t}^{L},o_{t},\mathcal{A}_{t}^{\mathrm{adm}}) , and the policy acts on the augmented context \tilde{x}_{t}=(x_{t},m_{t}) . Taking GRPO ( 14 ) as the RL algorithm for example. For each task g , a group of N trajectories is sampled and advantages are computed by normalizing returns within the group. Under the skill-augmented context, the GRPO objective is \mathcal{L}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{i}\left[\min\!\left(r_{i}(\theta)\hat{A}_{i},\operatorname{clip}(r_{i}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{i}\right)-\beta D_{\mathrm{KL}}\big(\pi_{\theta}(\cdot\mid\tilde{x}_{i})\|\pi_{\mathrm{ref}}(\cdot\mid\tilde{x}_{i})\big)\right]. Here \tilde{x}_{t}^{i} denotes the context augmented with retrieved skills, \hat{A}_{i,t} denote the normalized advantage and r_{i,t}(\theta)=\frac{\pi_{\theta}(a_{i,t}\mid\tilde{x}_{i,t})}{\pi_{\theta_{\mathrm{old}}}(a_{i,t}\mid\tilde{x}_{i,t})} be the