Self-Distilled Agentic Reinforcement Learning
Zhengxi Lu 1,2 , Zhiyuan Yao 1,2 , Zhuowen Han 2 , Zi-Han Wang 2,3 , Jinyang Wu 3 Qi Gu 2 , Xunliang Cai 2 , Weiming Lu 1 , Jun Xiao 1 , Yueting Zhuang 1 , Yongliang Shen 1 1 Zhejiang University 2 Meituan 3 Tsinghua University {zhengxilu, syl}@zju.edu.cn [email protected] · 2026
Read Self-Distilled Agentic Reinforcement Learning by Zhengxi Lu 1,2 , Zhiyuan Yao 1,2 , Zhuowen Han 2 , Zi-Han Wang 2,3 , Jinyang Wu 3 Qi Gu 2 , Xunliang Cai 2 , Weiming Lu 1 , Jun Xiao 1 , Yueting Zhuang 1 , Yongliang Shen 1 1 Zhejiang University 2 Meituan 3 Tsinghua University {zhengxilu, syl}@zju.edu.cn [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15155.txt.
Introduction
Zhengxi Lu and ten others present SDAR, self-distilled agentic reinforcement learning for long-horizon language-model agents. Trajectory-level rewards give only coarse supervision, and naive on-policy self-distillation shakes multi-turn agents. SDAR keeps RL as the backbone and adds distillation as a gated auxiliary goal, strengthening distillation on teacher-endorsed positive-gap tokens while easing negative teacher rejections. Across Qwen2.5 and Qwen3 models on ALFWorld, WebShop, and Search-QA, SDAR beats GRPO by 7 to 10 percent.
Catalog record
| CREATOR | Zhengxi Lu 1,2 , Zhiyuan Yao 1,2 , Zhuowen Han 2 , Zi-Han Wang 2,3 , Jinyang Wu 3 Qi Gu 2 , Xunliang Cai 2 , Weiming Lu 1 , Jun Xiao 1 , Yueting Zhuang 1 , Yongliang Shen 1 1 Zhejiang University 2 Meituan 3 Tsinghua University {zhengxilu, syl}@zju.edu.cn [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53502 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15155
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Distilled Agentic Reinforcement Learning Zhengxi Lu 1,2 , Zhiyuan Yao 1,2 , Zhuowen Han 2 , Zi-Han Wang 2,3 , Jinyang Wu 3 Qi Gu 2 , Xunliang Cai 2 , Weiming Lu 1 , Jun Xiao 1 , Yueting Zhuang 1 , Yongliang Shen 1 1 Zhejiang University 2 Meituan 3 Tsinghua University {zhengxilu, syl}@zju.edu.cn [email protected] Abstract Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR ( S elf- D istilled A gentic R einforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL–OPSD baselines across model scales. Code available: https://github.com/ZJU-REAL/SDAR . Figure 1: (a) Comparison between GRPO+OPSD and SDAR ; (b) Overall Performance. 1 Introduction Agentic post-training has become a central challenge for Large Language Models (LLMs) ( 10 ; 31 ; 41 ; 4 ; 32 ) . Unlike static single-turn reasoning, multi-turn agents interact with environments over extended horizons, where each action changes future observations and each generated response becomes part of the context for subsequent decisions ( 26 ; 28 ; 13 ) . Two paradigms naturally emerge as complementary forces: Reinforcement Learning (RL) ( 25 ; 6 ; 7 ) provides task-level optimization grounded in environment or verifier feedback, whereas On-Policy Distillation (OPD) ( 48 ; 43 ; 30 ; 8 ) and On-Policy Self-Distillation (OPSD) ( 51 ; 11 ; 50 ) provide dense token-level guidance from a teacher branch. Yet, OPSD does not transfer cleanly to multi-turn agent training. We attribute this to two observations: [1] Multi-turn OPSD Instability and [2] Asymmetric Trust in Privileged Guidance . [Observation-1] Multi-turn OPSD Instability Once the student agent inevitably drifts Figure 2: Left : Multi-turn OPSD Instability, with performance and KL reported. Right : RLSD-Style Instability, with KL loss. from the teacher-supported trajectory, the once-helpful token-level supervision becomes increasingly unreliable. This compounding error leads to surging per-turn KL divergence and catastrophic degradation in task performance, as shown in Figure 2 (Left). TCOD ( 36 ) attempts to address this through curriculum learning, but relies on rigid temporal schedules or trajectory-depth thresholds. [Observation-2] Asymmetric Trust in Privileged Guidance. In OPSD, the teacher branch is not an independently stronger model, but the same policy augmented with privileged training-only context, such as retrieved skills. This makes its token-level guidance inherently asymmetric. For a student-sampled token y_{t} , if the privileged teacher assigns a higher probability than the student, the retrieved skill provides an endorsement signal: it supports an on-policy behavior that the student can already generate but has not fully internalized. Such positive guidance is particularly suitable for distillation. In contrast, if the privileged teacher assigns a lower probability to the sampled token, the signal should be interpreted more cautiously. A negative gap may indicate that the token should indeed be suppressed, but in skill-conditioned OPSD it may also arise from the instability of privileged context: (1) Skill Quality. Retrieved skills may be irrelevant, incomplete, or redundant. (2) Skill Utilization. The teacher may fail to ground even relevant skills into reliable token-level preferences ( 2 ) . (3) Multi-turn Drift. As trajectories unfold, the teacher-student gap tends to widen across turns (Figure 3 , Middle), amplifying early mismatches over successive decisions ( 24 ) . Our preliminary study on Qwen2.5-3B-Instruct shows that negative-gap tokens exceed 50% of all tokens (Figure 3 ), making this issue pervasive. This motivates an asymmetric treatment of privileged guidance: trust positive teacher endorsements more strongly, while applying negative teacher rejections more conservatively. Figure 3: Teacher-Student Gap Analysis. Left : Token count distribution partitioned by Teacher-Student gap value. Middle : Average teacher-student gap indexed by multi-turn step. Right : Average teacher-student gap indexed by relative position within a single turn. A stark realization emerges: for multi-turn agents, RL could reign as the primary optimization backbone, while OPSD is relegated to a carefully controlled auxiliary role. But how should this auxiliary role be controlled? RLSD ( 42 ) directly uses self-divergence to re-weight token-level RL advantages, but can substantially amplify updates especially early in training when teacher-student mismatch is large (see Figure 2 , Right). We take a different path: the OPSD loss is treated as a direct, auxiliary optimization objective, leaving the verifier-driven RL policy loss untouched and thereby strictly preserving the semantics and unbiasedness of the RL advantage. To overcome instability of multi-turn OPSD and privileged guidance, distillation is not performed uniformly on every token. Instead, tokens are selectively distilled via an adaptive, smooth gating mechanism rather than a hand-crafted, rigid schedule (such as Skill-SD ( 35 ) and HDPO ( 5 ) ). Inspired by TIP ( 40 ) , we use token-level signals (such as student entropy or teacher-student divergence) to control the gate’s activation. The core philosophy is simple: let each token decide the intensity of its own supervision. This yields a dynamic, self-paced curriculum operating at the finest possible granularity: the individual token level. We validated our method across the Qwen2.5 and Qwen3 model families on three diverse benchmarks for llm-based agents: ALFWorld ( 29 ) , WebShop ( 45 ) , and Search-QA ( 14 ) . SDAR achieves substantial improvements over GRPO ( +9.4\% on ALFWorld, +7.0\% on Search-QA, and +10.2\% on WebShop-Acc for 7B), entirely avoids the catastrophic instability of naïve GRPO+OPSD, and consistently outperforms RL–OPSD hybrid methods such as Skill-SD and RLSD across all three model scales (Qwen3-1.7B included). Furthermore, robustness analysis shows that SDAR degrades gracefully with retrieval quality: even random retrieval outperforms the GRPO baseline, as our gating design filters out noise from low-quality skills and distills beneficial signals only. 2 Method 2.1 Problem Setup We consider a multi-turn agent that interacts with an environment over a finite horizon. Given an initial prompt or task description x , at turn k the agent receives an observation o_{k} , generates a response a_{k} , and the environment returns the next observation o_{k+1} . Each response a_{k} may contain both intermediate reasoning tokens and executable action tokens. For notational simplicity, we flatten all valid response tokens in one trajectory into a single token sequence y=(y_{1},\dots,y_{T})\sim\pi_{\theta}(\cdot\mid x), where \pi_{\theta} denotes the student policy and T is the total number of valid response tokens. Figure 4: Illustrations of SDAR framework, which trains multi-turn agents using token-level OPSD loss and verifier-driven RL loss. At token position t , we denote the self-student context by s_{t}=(x,y_{<t}), and the self-teacher context by s_{t}^{+}=(x,c^{+},y_{<t}), where c^{+} denotes privileged training-only context available only to the teacher branch, such