From Raw Experience to Skill Consumption:

A Systematic Study of Model-Generated Agent Skills · 2026

Read From Raw Experience to Skill Consumption: by A Systematic Study of Model-Generated Agent Skills (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23899.txt.

Introduction

Zisu Huang and colleagues run the first full lifecycle study of model generated agent skills. They track skills from experience generation, extraction and use across five task domains. Skills help on average but sometimes transfer negatively. Extractor strength does not predict consumer strength, nor do model scale or baseline skill. Digging into each stage, the team traces what makes skills useful and how they travel between consumers. They then distill a meta-skill guiding extraction toward utility linked features. That improves quality and cuts negative transfer.

Catalog record

CREATORA Systematic Study of Model-Generated Agent Skills
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69574
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills Abstract May 2026 Language agents increasingly improve by reusing skills —structured procedural artifacts distilled from past experience. In particular, domain-level and model-generated skills are especially promising. They offer fast adaptation within a domain by encoding domain-specific recurring procedures, and they scale beyond labor-intensive hand-crafting. However, while extraction methods continue to proliferate, understanding remains limited, with no comprehensive study spanning the full skill lifecycle— experience generation , skill extraction , and skill consumption —to ask whether such skills actually work, when they work, and what makes them succeed or fail. To close this gap, we build a utility-grounded evaluation framework that provides systematic experimental results across extractors and target agents, covering five diverse agentic task domains. We find that model-generated skills are beneficial on average but exhibit non-trivial negative transfer, and that neither extractors nor targets behave uniformly. A model can be a strong extractor yet a weak consumer, or vice versa, with skill utility independent of model scale or baseline task strength. To explain these patterns, we then dissect each lifecycle stage in depth, analyzing how experience composition shapes skill quality, what properties characterize useful skills, and how the same skill transfers across different consumers. Finally, we translate these findings into a concrete meta-skill that guides skill extraction toward the features tied to actual utility, which consistently improves skill quality across domains and substantially reduces negative transfer. Correspondence: [email protected] , [email protected] Code: https://aka.ms/SkillLens ∗ Equal contribution. † Work done during an internship at MSRA. ‡ Corresponding authors. Introduction Language agents increasingly improve by reusing knowledge distilled from past trajectories: skills —short, structured procedural artifacts—can be loaded at inference time without retraining and have become a defining mechanism for accumulating experience in modern agent stacks [ 1 , 2 ] . In particular, domain-level skills package a domain’s recurring procedures into a single reusable artifact or a coordinated set of them, enabling fast adaptation to new tasks within the domain rather than per-task optimization. As the practical value of hand-crafted skills has been progressively demonstrated in real-world deployments, skills have become a standard component in several commercial agent platforms [ 3 ] . However, hand-crafting skills is labor-intensive and cannot keep pace with the rapidly expanding scope of agent capabilities and deployment. Therefore, a growing literature turns to model-generated skills , producing them automatically at scale [ 4 , 5 , 6 , 7 , 8 ] , with featured works either directly distilling them from execution logs as in Trace2Skill [ 9 ] , or iteratively refining multi-file skill packages with a co-evolving verifier as in CoEvoSkills [ 10 ] —offering scalability and automated iteration for agent skills. At their core, all these methods follow the same skill lifecycle: generating execution trajectories through agent–environment interaction ( experience generation ), extracting reusable knowledge or patterns from them ( skill extraction ), and consuming the resulting skills at inference time ( skill consumption ). Despite this methodological momentum, evaluation and understanding lag behind. Recent benchmarks each illuminate one slice of the picture but leave the lifecycle as a whole opaque. Most existing efforts study only the skill consumption stage, measuring the marginal performance gain from skill equipment: SkillsBench [ 11 ] uses task-seeded, human-authored skills, while SWE-Skills-Bench [ 12 ] and Skills-in-the-Wild [ 13 ] draw skills from existing public skill repositories instead—all leaving the skill extraction stage outside the loop. A notable step toward studying the skill extraction stage is SkillCraft [ 14 ] , which extracts skills as executable compositions of atomic tools and studies their reuse across tasks. However, it has notable limitations: skills are restricted to executable function compositions, and the benchmark’s tasks are designed and scaled to admit such compositions, making it unclear whether the paradigm generalizes to broader domains whose tasks are not designed around function-style reuse. Taken together, these efforts leave a clear gap: no comprehensive study examines all three stages of the skill lifecycle and systematically asks whether domain-level, model-generated skills actually work, when they work, and what makes them work or fail. To close this gap, we conduct a comprehensive, utility-grounded study of model-generated, domain-level skills that analyze all three stages of the skill lifecycle. Specifically, we follow a three-step pipeline: a target agent first executes an experience-generation split to produce an experience pool; an extractor then distills this pool into a single domain-level skill through an extraction framework with minimal design, reflecting the extractor’s own ability rather than scaffolding tricks; the resulting skill is finally applied back to the same target and evaluated on the held-out test split to obtain the performance change relative to a no-skill baseline, which we use as a proxy for skill utility. We instantiate this pipeline across five domains, spanning embodied planning, productivity software, software engineering, web search, and tool calling, and systematically vary the extractor and target. Based on these experiments, we further introduce two metrics that disentangle the two roles: the Extraction Efficacy ( \mathrm{EE} )—how reliably a fixed extractor produces helpful skills across targets—and the Target Evolvability ( \mathrm{TE} )—how much a fixed target benefits from skills extracted by different extractors from its own experience. Beyond reporting these metrics, we further provide an in-depth analysis spanning all three lifecycle stages, aiming to explain the observed utility patterns and to point toward concrete directions for improving skill extraction. The pipeline and analysis are summarized in Figure 1 . Overall, our study is organized around three research questions: • RQ1 Do model-generated, domain-level skills reliably benefit downstream agents across targets, extractors, and domains? (Section 4 ) • RQ2 Across the three lifecycle stages of experience generation (Section 5.1 ), skill extraction (Section 5.2 ), and skill consumption (Section 5.3 ), what actually drives a skill’s downstream utility? • RQ3 Can the empirical findings in our study be transformed into a concrete, drop-in improvement to skill extraction itself? (Section 6 ) Answering these questions, we aim to move the entire skill lifecycle from heuristic, intuition-driven practice toward a principled, utility-grounded discipline. As skill libraries proliferate across heterogeneous models and domains, our study helps practitioners obtain skills that are genuinely stable and effective when deployed in real agent systems. Figure 1 : Overview of our study design. We evaluate the full trajectory-to-skill lifecycle across three stages: experience generation, skill extraction, and skill consumption. Related Work Automatic Generation of Reusable Knowledge from Agent Experience. Recent surveys identify agent skills—composable packages of instructions, code, and resources loaded on demand—as a defining mechanism for extending LLM capabilities without retraining [ 1 ] , motivating a growing body of work on automatically extracting such skills from execution trajectories. These methods scale with collected experience, transfer across tasks and environments, and largely organize around trajectory-to-skill extraction as the core primitive. Prompt-based distillation methods directly summarize trajectories into structured skill artifacts: Trace2Skill [ 9 ] employs parallel sub-agents followed by hierarchical consolidation, AutoRefine [ 4 ] induces dual-form experience patterns, PRAXIS [ 5 ] builds state-indexed procedural memory, and MemP [ 15 ] formalizes the build–retrieve–update cycle of agent procedural memory. Optimization and RL-based methods further refine extracted skills: ProcMem [ 6 ] applies non-parametric PPO, CoEvoSkills [ 10 ] uses

Read the complete text