ReSkill : Reconciling Skill Creation with Policy Optimization in Agentic RL
Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr[0.8em] Code Website · 2026
Read ReSkill : Reconciling Skill Creation with Policy Optimization in Agentic RL by Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr[0.8em] Code Website (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01619.txt.
Introduction
He and co-authors reconcile skill creation with policy learning in agent RL. RL lets LLM agents improve from environment rewards over time. The resulting policies rarely accumulate reusable strategies, though. Modular skills could supply those strategies across tasks. Past skill-augmented RL splits skill creation from policy optimization. Adopted skills can then conflict with the evolving policy. ReSkill is an RL-in-the-loop skill creation framework. It exploits the group-wise structure of GRPO for its design.
An assertion-driven creator diagnoses failures and proposes skill revisions. Within-group sampling compares skill versions in a controlled way. Thompson Sampling with adaptive discounting picks versions over time. Skills get created, tested, refined, and pruned automatically. Gains are largest on unseen tasks in their tests.
Catalog record
| CREATOR | Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr[0.8em] Code Website |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReSkill : Reconciling Skill Creation with Policy Optimization in Agentic RL Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, Matthew Reimherr[0.8em] Code Website Abstract Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic’s Skill Creator, we introduce ReSkill , an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy’s ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution. † † footnotetext: ∗ Equal contribution. ‡ Project lead. † Work done during an internship at Amazon. 1 The Pennsylvania State University 2 Amazon IntelliHub 3 AWS AI Labs 1 Introduction Large language models (LLMs) have enabled autonomous agents capable of multi-turn interaction with complex environments ( 53 ; 21 ) . Agentic reinforcement learning (RL), where agents are trained from verifiable environment rewards, has emerged as a promising paradigm ( 16 ; 44 ; 7 ) , yet the resulting policies do not systematically accumulate transferable problem-solving strategies across tasks. Recent work has shown that modular skills , reusable snippets encoding task-specific strategies, can fill this gap ( 40 ; 59 ; 24 ) . A pioneering effort is Anthropic’s Skill Creator ( 1 ) , which automates skill creation through an iterative human-in-the-loop testing process (Figure 1 (a)). However, the Skill Creator operates on a static policy at inference time and relies on human feedback. A natural question arises: can we build an RL-in-the-loop skill creation process that dynamically creates and evaluates skills alongside an evolving policy? Recent approaches accumulate experiential knowledge during RL training through memory evolution ( 57 ) , principle distillation ( 45 ) , and skill distillation ( 46 ) . However, they adopt external knowledge without principled testing, treating skill creation and policy learning as separate processes (Figure 1 (b)). Yet this approach risks skill-policy conflict, as externally produced skills may not align with the evolving policy. Ideally, whether a skill benefits learning should be assessed by testing it during policy training itself. (a) (b) Figure 1: (a) Inspired by Anthropic’s human-in-the-loop Skill Creator, ReSkill recasts skill creation as an RL-in-the-loop process. (b) Compared with decoupled skill-update methods, ReSkill evaluates different skill versions inside the policy optimization loop and uses rewards for guided updates, achieving reconciled skill-policy updates. In this paper, we propose ReSkill , a framework that reconciles skill creation with policy learning. ReSkill embeds the Anthropic Skill Creator cycle into the RL training loop and makes use of the group-wise structure of GRPO ( 31 ) : each training step samples a group of rollouts from the same task under the same policy, allowing us to assign competing skill versions within the same group and directly compare their effects. To orchestrate this allocation, Thompson Sampling ( 37 ) with adaptive discounting is introduced to balance exploration and exploitation in skill version control as the policy evolves. Each rollout thus serves a triple role : (i) policy optimization via GRPO gradients, (ii) failure diagnosis and skill revision through an assertion-driven creator adapted from Anthropic’s Skill Creator, and (iii) bandit evaluation that accepts or rejects skill versions based on how they support the policy’s ongoing learning. Our method introduces marginal training overhead and requires no additional rollout budget. We compare ReSkill against existing memory and skill-based agentic RL baselines on embodied reasoning (ALFWorld) and agentic search (seven benchmarks) across multiple model scales, with additional experiments on coding (InterCode-SQL), web shopping (WANDS), and science discovery (ScienceWorld). ReSkill achieves substantially improved performance, with the largest gains on unseen and hard tasks. The skill lifecycle analysis shows skills being automatically created, tested, refined, and pruned as the policy improves. Furthermore, the co-evolved policy can adapt to new domains (ALFWorld to ScienceWorld) at test time through skill creation alone, indicating that reconciled training develops a general capacity to follow and benefit from new skills. 2 Preliminaries Agentic Reinforcement Learning with Skills. We consider an LLM-based agent with policy \pi_{\theta} that interacts with an environment over multiple turns via a tool engine \mathcal{T} . Given a task q\sim\mathcal{D} and a skill bank \mathcal{S} that provides strategic guidance with conditional loading, the agent generates a trajectory \tau=(q,\,s_{0},a_{0},o_{0},\,\ldots,\,s_{K},a_{K}) , where a_{t}\sim\pi_{\theta}(\cdot\mid\tau_{<t},s_{t};\,\mathcal{T}) is the agent’s action, o_{t}=\mathcal{T}(a_{t}) is the environment observation, and s_{t}\subseteq\mathcal{S} is the subset of skills whose triggers fire. The learning objective maximizes the expected reward r(\tau) : \max_{\pi_{\theta}}\mathbb{E}_{q\sim\mathcal{D},\,\tau\sim\pi_{\theta}(\cdot|q,\mathcal{S};\mathcal{T})}[r(\tau)] . Group Relative Policy Optimization ( Grpo ). Grpo ( 31 ) optimizes the policy using group-relative advantages. For each task q\sim\mathcal{D} , Grpo samples G trajectories \{\tau_{i}\}_{i=1}^{G} from the current policy \pi_{\theta}(\cdot\mid q,\mathcal{S};\mathcal{T}) and optimizes \mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\;\tau_{i}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q,\,\mathcal{S};\,\mathcal{T})}\left[\mathcal{L}_{\text{GRPO}}(\theta;\,\{\tau_{i}\}_{i=1}^{G})\right] with a static skill bank \mathcal{S} , where: \mathcal{L}_{\text{GRPO}}(\theta;\,\{\tau_{i}\}_{i=1}^{G})=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|\tau_{i}|}\sum_{t=1}^{|\tau_{i}|}\min\left(r_{i,t}\hat{A}_{i},\;\text{clip}(r_{i,t},1{-}\epsilon,1{+}\epsilon)\hat{A}_{i}\right)-\beta\mathbb{D}_{\text{KL}}, (1) r_{i,t} is the token-level importance sampling ratio, \hat{A}_{i} is the group-normalized advantage, \epsilon is the clip ratio, and \beta controls the KL divergence penalty \mathbb{D}_{\text{KL}} (details in Appendix C.2 ). All G rollouts in a group share the same task q and policy \pi_{\theta} , differing only in their stochastic trajectories, providing a natural controlled setting for comparing skill versions (§ 3.1 ). 3 Reconciling Skill Creation with Policy Learning Figure 2 illustrates the overall framework. We first describe how within-group skill testing integrates into Grpo training, enabling each rollout to simultaneously serve policy optimization, failure diagnosis, and skill evaluation (§ 3.1 ). We then present the skill creator pipeline that produces candidate skill versions from ongoing training experience (§ 3.2 ). Finally, we introduce Thompson Sampling to allocate rollouts between competing skill versions while maintaining training efficiency (§ 3.3 ). The complete procedure is summarized in Algorithm 2 . Figure 2: Overview of ReSkill . (1) RL training with within-group skill testing (§ 3.1 ). (2) RL-in-the-loop skill creation (§ 3.2 ). (3) RL-guided skill evolution with Thompson Sampling (§ 3.3 ). 3.1 RL Training with Within-Group Skill Testing Within-Group Sampling. To evaluate whether a skill benefits the evolving policy, we need to observe how the policy behaves under it. However, dedicated testing rollouts are expensive in multi-turn settings, and training under a single fixed skill bank loses adaptivity as skills change. We leverage GRPO’s group structure to address both issues: for each rollout i in a group, a version assignment v_{i} is drawn from the Thompson Sampling posterior (§ 3.3 ), and the rollout is generated under the corresponding skill bank \mathcal{S}_{v_{i}} . The ReSkill objective extends \mathcal{J}_{\text{GRPO}}