UCOB : Learning to U tilize and Evolve Agentic Skills via C redit-Aware O n-Policy B idirectional Self-Distillation

Songjun Tu, Chengdong Xu, Qichao Zhang✉, Yiwen Ma, Yaocheng Zhang, Linjing Li, Dong Li, Xiangyuan Lan, Dongbin Zhao ♠ \spadesuit Institute of Automation, Chinese Academy of Sciences ♣ \clubsuit Pengcheng Laboratory ♢ \diamondsuit Memorax AI {tusongjun2023,zhangqichao2014}@ia.ac.cn June 28, 2026. · 2026

Read UCOB : Learning to U tilize and Evolve Agentic Skills via C redit-Aware O n-Policy B idirectional Self-Distillation by Songjun Tu, Chengdong Xu, Qichao Zhang✉, Yiwen Ma, Yaocheng Zhang, Linjing Li, Dong Li, Xiangyuan Lan, Dongbin Zhao ♠ \spadesuit Institute of Automation, Chinese Academy of Sciences ♣ \clubsuit Pengcheng Laboratory ♢ \diamondsuit Memorax AI {tusongjun2023,zhangqichao2014}@ia.ac.cn June 28, 2026. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29502.txt.

Introduction

Tu and colleagues improve how AI agents use skill memories, stored textual guidance drawn from past experience. Such skills can help in one state yet mislead the same policy in another. Treating a skill-conditioned prompt as a fixed teacher is therefore fragile. Their framework is called UCOB, short for a credit-aware on-policy bidirectional self-distillation method.

UCOB compares two views of the same model on the same task and anchor state: one prompt with a skill, one without. Whichever view earns the higher return becomes the local teacher for that moment. This credit signal internalizes useful skill behaviour, corrects misleading skill use, and guides memory updates, retrieval choices, and reflection training. Tests cover ALFWorld, WebShop, and Search-QA tasks. UCOB beats skill-free reinforcement learning, skill-memory baselines, and other self-distillation methods across model scales, with gains over top baselines above twenty points on ALFWorld.

Catalog record

CREATORSongjun Tu, Chengdong Xu, Qichao Zhang✉, Yiwen Ma, Yaocheng Zhang, Linjing Li, Dong Li, Xiangyuan Lan, Dongbin Zhao ♠ \spadesuit Institute of Automation, Chinese Academy of Sciences ♣ \clubsuit Pengcheng Laboratory ♢ \diamondsuit Memorax AI {tusongjun2023,zhangqichao2014}@ia.ac.cn June 28, 2026.
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29502

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UCOB : Learning to U tilize and Evolve Agentic Skills via C redit-Aware O n-Policy B idirectional Self-Distillation Songjun Tu, Chengdong Xu, Qichao Zhang✉, Yiwen Ma, Yaocheng Zhang, Linjing Li, Dong Li, Xiangyuan Lan, Dongbin Zhao ♠ \spadesuit Institute of Automation, Chinese Academy of Sciences ♣ \clubsuit Pengcheng Laboratory ♢ \diamondsuit Memorax AI {tusongjun2023,zhangqichao2014}@ia.ac.cn June 28, 2026. Abstract Skill memories can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not oracular: they may help in one state while misleading the same policy in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned prompt can be treated as a fixed teacher for the no-skill prompt. We introduce UCOB , a framework for learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation. UCOB treats skill-conditioned and no-skill prompts as two on-policy context views of the same model, compares their return-to-go within the same task and anchor state, and uses the higher-return view as the local teacher. This local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and guides task/state skill memory updates, utility-aware retrieval, and reflection self-training. Experiments on agentic tasks, including ALFWorld , WebShop , and Search-QA , show that UCOB outperforms skill-free RL, skill-memory baselines, and self-distillation methods across model scales, with up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop . Ablations and analyses further validate its core mechanisms and efficiency. Code is available at https://github.com/TU2021/UCOB . 1 Introduction Figure 1: Overview and empirical summary of UCOB . Agentic reinforcement learning (RL) has become a central paradigm for improving language agents in long-horizon interactive tasks. Unlike single-turn reasoning, these environments require agents to act under partial observations, recover from local mistakes, and assign sparse outcome rewards to many intermediate decisions. A natural way to reduce this difficulty is to reuse past experience: agents can reflect on trajectories, store reusable textual skills, retrieve relevant skills in later states, and condition future actions on this external memory ( 22 ; 28 ) . Recent skill-augmented agent methods show that such memory can improve exploration and sample efficiency ( 26 ; 21 ) . When these retrieved skills are further used for training, however, a stronger assumption often enters implicitly: the skill-conditioned behavior is treated as a better source of supervision. Can skill-conditioned context be safely treated as a privileged teacher during agent training? This assumption is appealing but fragile. Retrieved skills may be irrelevant, stale, overly generic, or locally mismatched to the current state, and the agent may fail to ground even useful skills in the present observation. Consequently, a skill-conditioned prompt can help in one state but mislead the same policy in another. Prior methods improve skill construction, curation, or skill-conditioned self-distillation ( 15 ; 29 ; 12 ) , but the supervision direction often remains fixed from the skill-conditioned view to the no-skill view. Our diagnostics in Section 2 challenge this fixed-teacher view: the skill-conditioned branch is not consistently better than the no-skill branch, and making skill-conditioned rollouts on-policy does not remove this ambiguity. When skill and no-skill views disagree at the same state, which view should teach the other? To address this, we propose UCOB (learning to U tilize and evolve agentic skills via C redit-aware O n-policy B idirectional self-distillation), which treats the skill-conditioned prompt and the no-skill prompt as two on-policy context views of the same online model, rather than as a teacher and a student fixed in advance. During training, UCOB samples mixed skill/no-skill rollouts, groups records by task and anchor state, and compares return-to-go within each local group. The higher-return view becomes the local teacher for the opposite view: useful skill-conditioned behavior is distilled into the no-skill context, while misleading skill-induced behavior is corrected by the no-skill view. The same local return gap is further reused to update task-level and state-level skill memories, guide utility-aware retrieval, and train the reflection-based skill writer, closing the loop between policy learning and skill evolution. Figure 1 summarizes the resulting framework and its empirical behavior across training and evaluation. Experiments on agentic tasks, including ALFWorld , WebShop , and Search-QA , show that UCOB improves over skill-free RL, skill-memory baselines, and self-distillation methods across model scales. Ablations verify the contribution of each module and design choice, while further analyses examine teacher routing, memory evolution, and training cost. The main contributions of this work are as follows: 1. We present observations showing that skill-conditioned teachers are not reliably better than the no-skill view, even when skill-conditioned rollouts are made on-policy. 2. We introduce Credit-Aware Bidirectional Self-Distillation (CBSD) , which selects on-policy teacher directions from same-task, same-anchor-state returns. 3. We develop UCOB , a unified framework for utilizing and evolving agentic skills through CBSD, task/state skill memory, utility-aware retrieval, and reflection self-training. 4. We validate UCOB across model scales, outperforming SOTA baselines such as SDAR and Skill1 by up to 23.5 and 18.0 points on ALFWorld and WebShop ; analysis further confirms its module and design effectiveness. 2 Observations We examine whether skill-conditioned context can be safely treated as a privileged teacher for multi-turn agent training through three diagnostics: (i) whether a skill-conditioned teacher is reliably better than a no-skill view, (ii) whether making the skill-conditioned view on-policy resolves this issue, and (iii) what local evidence should determine the distillation direction when the two views disagree. (a) Fixed-direction skill/no-skill protocol (b) Anchor-state return-gap diagnostic Figure 2: Unified schematic of the observation-study protocols. (a) Fixed-direction skill/no-skill protocol for the no-skill-only and dual-rollout settings. (b) Anchor-state return-gap diagnostic. Detailed settings and rollout details are given in Appendix A . Skill-conditioned teachers are unreliable and not self-correcting. Figure 3: SDAR Evaluation w/ and w/o skills during training on Qwen3-1.7B. We first revisit SDAR ( 12 ) , an asymmetric self-distillation setup where rollouts use the no-skill prompt while a skill-conditioned prompt serves as the privileged teacher. This design assumes that, for states induced by the no-skill rollout, the skill-conditioned view provides a more reliable distillation target than the corresponding no-skill view. Figure 3 challenges this assumption on ALFWorld and WebShop : the with-skill evaluation does not consistently dominate the no-skill evaluation, and on WebShop , it remains weaker throughout training. Thus, adding a skill context does not automatically yield a reliable teacher . Since the privileged view is not directly corrected by return feedback under its own context distribution, a weak teacher is also not self-correcting ; distillation may continue to transfer supervision from unverified conditioned behavior. Rollouts with skills mitigate exposure mismatch but not teacher ambiguity. Figure 4: Dual-rollout SDAR evaluation and training on WebShop . A natural remedy is to place skills into the rollout itself, so the skill-conditioned view is optimized through environment interaction rather than only serving as a teacher outside the rollout path. We therefore test a dual-rollout fixed-direction variant: each training batch samples half of the trajectories with skills and half without skills, following the training design of Skill-SD ( 29 ) , while keeping the SDAR-style skill-to-no-skill distillation direction. Both context views therefore receive on-policy RL updates, reducing exposure mismatch. This dual-rollout setting is summarized in Figure 2 (a). However, distillation remains one-way: only the no-skill

Read the complete text