ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning

2026

Read ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16060.txt.

Introduction

Li and co-authors grow a reusable skill library inside math reasoning training. Reward-only training treats each problem as a fresh island. Their ARISE frame splits one policy into Manager and Worker roles. The Manager curates a tiered library of solution skills. Winning traces get summarized into new library entries. A selection step pulls fit skills before each new attempt. Layered rewards tune reasoning and library quality together. Seven benchmarks show steady gains over GRPO kin. Out-of-distribution tasks gain the most from the library.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61260
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning Abstract The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework in which a shared policy operates both as a high-level Skills Manager and a low-level Worker. The Manager maintains a tiered skill library through two complementary mechanisms: before execution, a policy-driven selection mechanism retrieves relevant skills to condition future rollouts; after execution, a dedicated skill generation rollout performs structured summarization of successful solution traces. A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. 1 Introduction Figure 1: Overview of ARISE on Qwen3-4B, showing performance across seven benchmarks, training reward dynamics on DeepScaleR, skill utilization over training, and accuracy gain versus token overhead relative to GRPO. Reinforcement learning with verifiable rewards has emerged as a compelling paradigm for training mathematical reasoning in large language models, enabling policies to improve through trial-and-error without relying on expensive human annotation ( 4 ) . Despite strong performance on standard benchmarks, existing methods solve each problem instance via a separate process: Once a rollout concludes, the successful reasoning strategies generated in the process are discarded rather than retained and accumulated for future use ( 13 ; 21 ) . A natural remedy is to equip the agent with a persistent skill library that accumulates reusable reasoning strategies over time ( 16 ; 17 ) . Recent work has demonstrated the value of organizing past experience into structured skills and retrieving them at inference or training time ( 18 ; 19 ; 16 ) , allowing agents to accumulate and transfer knowledge across problems and avoid redundant exploration. However, existing approaches share a fundamental limitation: skill management, including both skill selection before execution and skill summarization after execution, is delegated to an external retriever, preventing the policy gradient from directly shaping skill selection. Furthermore, the skill library is updated independently of the RL objective, breaking the feedback loop between policy improvement and library enrichment ( 5 ) . We present ARISE, an integrated, hierarchical reinforcement learning framework for Agent Reasoning via Intrinsic Skill Evolution, that addresses both limitations through a unified design. The key insight is that the skill library should not be a static external resource but an intrinsic component of the agent’s state, co-evolving with the policy throughout training. ARISErealizes a Manager-Worker hierarchy within a single shared policy \pi_{\theta} : the manager selects skills using the policy’s own log-probabilities and generates new skills by summarizing successful solution traces, while the worker generates solution traces conditioned on the selected skill. Since the same parameters govern both skill selection and solution generation, the advantage signal from the hierarchical reward propagates end-to-end, reinforcing selection preferences for skills that demonstrably improve reasoning outcomes. To incentivize skill utilization, we introduce a hierarchical reward R\in\{r_{0},r_{1},r_{2}\} with r_{2}>r_{1}>r_{0} , distinguishing correct solutions with skill augmentation ( r_{2} ), without ( r_{1} ), and incorrect solutions ( r_{0} ). Under group-relative advantage, this differential signal steers the policy toward consistently leveraging useful skills. The skill library adopts a two-tier cache-reservoir architecture with five management operations, maintaining a compact active pool while preserving skills that may regain relevance as training progresses. Training proceeds in two phases: a warm-up that builds the base policy and populates the library via O_{G+1} , followed by a skill-augmented phase activating the full hierarchical pipeline. We evaluate ARISE on two instruction-tuned base models, Qwen3-4B-Instruct-2507 ( 14 ) and Phi-4-mini-instruct ( 1 ) , trained on the DeepScaleR dataset. Results on both in-distribution competition benchmarks and the out-of-distribution benchmark with four mathematical domains ( 2 ) demonstrate consistent improvements over GRPO-family baselines and existing memory- and skill-augmented methods. Our main contributions are summarized as follows: • We propose the Evolving-Skill MDP, a formal framework that models the skill library as an endogenous component of the agent’s state, enabling joint optimization of policy and library under a unified RL objective. • We introduce a policy-driven skill selection mechanism based on conditional log-probability scoring, allowing the policy gradient to directly shape selection preferences end-to-end without relying on an external retriever. • We design a hierarchical reward and a two-tier skill library architecture that together create a co-evolutionary dynamic between policy improvement and library enrichment. • Empirical results on competition and Olympiad-level benchmarks demonstrate that ARISE consistently outperforms both vanilla GRPO variants and skill-augmented baselines across two base models. 2 Backgrounds Reinforcement learning has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models. Early RLHF methods ( 10 ) relied on learned reward models trained on human preferences, but are susceptible to reward model overoptimization ( 3 ) and require expensive annotation. 4 introduced Group Relative Policy Optimization (GRPO), a critic-free variant of PPO ( 11 ) that estimates advantages via normalized rewards within a group of G rollouts per query, giving rise to the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm now dominant in mathematical reasoning. Several refinements have since addressed GRPO’s limitations: Dr.GRPO ( 7 ) corrects normalization biases in advantage estimation; DAPO ( 20 ) introduces asymmetric clipping and dynamic sampling to mitigate entropy collapse; and GSPO ( 23 ) replaces per-token importance ratios with sequence-level correction to better align with the reward signal. A complementary line of research augments LLM agents with external memory or reusable skill structures to enable experience transfer across episodes. Inference-time methods such as Reflexion ( 12 ) , ExpeL ( 22 ) , and SimpleMem ( 6 ) retrieve past trajectories or distilled knowledge into the agent’s context, but the resulting memory is populated independently of policy learning and remains fixed once constructed. More recent work integrates skill structures directly into the training loop: EvolveR ( 18 ) maintains a co-evolving skill library and SkillRL ( 19 ) builds a hierarchical skill bank via trajectory distillation. Building on this direction, SAGE ( 16 ) further incorporates skill augmentation into GRPO through sequential rollouts, coupling skill generation with policy optimization in a unified framework. 3 ARISE: Hierarchical Agent RL with Evolving Skills We present ARISE, a hierarchical reinforcement learning framework that addresses two limitations of existing skill-augmented approaches: skill selection is delegated to an external retriever decoupled from the policy gradient, and the skill library is updated independently of the RL objective. As illustrated in Figure 2 , ARISE resolves both issues through a unified Manager-Worker architecture in which a single policy \pi_{\theta} governs all skill interactions. Through a Download channel, the manager retrieves a relevant skill to condition the worker’s rollouts and through an Upload channel, the manager distills successful solution traces into new skill documents. Five management operations ( Add , Update ,

Read the complete text