SkillFlow : Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie, Yu Zeng, Yiming Zhao, Zhen Fang Qisheng Su Haibo Qiu Wei Yang Qingnan Ren Shun Zou, Wenxuan Huang Lin Chen Zehui Chen Feng Zhao · 2026
Read SkillFlow : Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents by Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie, Yu Zeng, Yiming Zhao, Zhen Fang Qisheng Su Haibo Qiu Wei Yang Qingnan Ren Shun Zou, Wenxuan Huang Lin Chen Zehui Chen Feng Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17308.txt.
Introduction
Zhang and colleagues introduce SkillFlow, a benchmark asking whether AI agents can discover, repair, and carry skills across tasks instead of merely using supplied ones. Its 166 tasks span 20 families under a shared workflow, with agents starting skill-less and patching their library from experience as they proceed. Lifelong learning lifted one leading model by about eight points, yet heavy skill use did not always help, exposing a gap between using skills and using them well.
Catalog record
| CREATOR | Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie, Yu Zeng, Yiming Zhao, Zhen Fang Qisheng Su Haibo Qiu Wei Yang Qingnan Ren Shun Zou, Wenxuan Huang Lin Chen Zehui Chen Feng Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99278 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SkillFlow : Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie, Yu Zeng, Yiming Zhao, Zhen Fang Qisheng Su Haibo Qiu Wei Yang Qingnan Ren Shun Zou, Wenxuan Huang Lin Chen Zehui Chen Feng Zhao Abstract As the capability frontier of autonomous agents continues to expand, they are increasingly able to complete specialized tasks through plug-and-play external skills. Yet current benchmarks mostly test whether models can use provided skills, leaving open whether they can discover skills from experience, repair them after failure, and maintain a coherent library over time. We introduce SkillFlow , a benchmark of 166 tasks across 20 families in which task construction within each family follows a Domain-Agnostic Execution Flow (DAEF) that defines an agent workflow framework, allowing these tasks to share a consistent workflow. Agents are evaluated under an Agentic Lifelong Learning protocol in which they begin without skills, solve tasks sequentially within each family, externalize lessons through trajectory- and rubric-driven skill patches, and carry the updated library forward. Experiments reveal a substantial capability gap. For Claude Opus 4.6, lifelong skill evolution improves task success from 62.65% to 71.08% (+8.43 points). However, high skill usage does not necessarily imply high utility: Kimi K2.5 gains only +0.60 points despite 66.87% skill usage, while Qwen-Coder-Next reaches only a 44.58% task completion rate and still regresses relative to the vanilla setting. SkillFlow contributes a structured testbed for this direction and an in-depth empirical analysis of skill discovery, patching, transfer, and their failure modes under lifelong evaluation. The code is available at https://github.com/ZhangZi-a/SkillFlow . Figure 1: Conceptual Overview of SkillFlow . The figure contrasts conventional static-skill evaluation with our lifelong setting, in which agents externalize experience into reusable skill artifacts, revise them through patches, and transfer them across tasks that share a common DAEF. Figure 2: Task-Construction Pipeline of SkillFlow . Step 1 collects candidate seed tasks and a curated external skill pool. Step 2 uses embedding-based pair matching to attach relevant skills to each seed and form task–skill reference pairs. Step 3 performs domain expansion under a fixed DAEF: the Architect Agent generates an initial cross-domain task set, the Critic Agent either rejects or accepts the family, and accepted families proceed to a second expansion round that adds more tasks and establishes a difficulty gradient. Step 4 applies human review over robustness, logical soundness, difficulty calibration, and instruction-leakage risk before final benchmark inclusion. 1 Introduction Frontier Large Language Models(LLMs) systems are increasingly deployed as autonomous command-line agents that solve realistic multi-step tasks in terminal environments. Claude Code 2 , Gemini CLI 11 , Codex CLI 25 , and Qwen-Coder 30 are prominent examples. Agent skills have recently emerged as external packages of procedural knowledge that augment model capabilities for specialized tasks. These skills can encode usage scenarios, core guidance, and reusable code or documentation assets. As a result, many vendors are beginning to release agent versions with native skill support. SkillsBench 19 provides an early demonstration that skills can substantially assist models in solving tasks, while also suggesting that current models remain limited in generating skills autonomously. However, it does not examine whether an LLM can discover and summarize reusable skills from its own task-solving process and apply them to future tasks of the same type. SkillWeaver 43 , SkillRL 37 , MemSkill 41 , and related systems 44 also show that experience-derived skills can improve downstream performance. This leads to a central question: can an autonomous agent extract reusable skills from its own experience, repair them after failures, and maintain an evolving skill library across a sequence of tasks? To answer this question, we introduce SkillFlow , a benchmark of 166 runnable tasks across 20 task families for measuring lifelong skill discovery and evolution. To construct the benchmark, we first analyze existing agent loops and find that mature agents 17 ; 40 ; 10 typically rely on stable AI workflows, which allow them to solve tasks of the same type in a consistent manner. Based on this observation, we define the Domain-Agnostic Execution Flow (DAEF), a workflow framework that preserves operational topology while abstracting away domain-specific entities and serves as a scaffold for benchmark construction. We use this structure to support controlled progression within each task family. Concretely, we select realistic tasks from GDPval 28 and SkillsBench 19 and extract a DAEF from each seed task. We then instantiate each DAEF through a dual-agent iterative pipeline using the original task as a reference instance, where one agent constructs task assets and the other reviews the generated tasks and provides timely feedback for revision. This process yields 8 to 9 tasks in the Harbor format 14 for each DAEF, which together form a task family. After human verification, the final benchmark contains 20 task families and 166 tasks. Evaluation then follows an Agentic Lifelong Learning protocol in which an agent begins without skills, solves tasks sequentially within each family, summarizes lessons from trajectories and rubric feedback, and updates the library through explicit skill patches after each task. Experiments reveal a substantial capability gap . On Claude Opus 4.6, lifelong skill evolution improves task success from 62.65% to 71.08% (+8.43 points). Yet high usage does not imply high utility: Kimi K2.5 gains only +0.60 points despite a 66.87% skill usage rate, while Qwen-Coder-Next reaches only a 44.58% task completion rate and still regresses relative to the vanilla setting. We find that most current models fail to achieve stable self-evolution through iterative skill updates. Stronger models tend to consolidate prior experience into stable workflows and continue refining them over time, thereby realizing sustained capability gains. Weaker models struggle to integrate multiple experiences into a coherent procedure, which leads to cognitive overload in subsequent tasks. Contributions. Our contributions are listed: • We introduce SkillFlow , a benchmark of 166 runnable tasks across 20 task families for evaluating lifelong skill discovery and evolution in autonomous agents. • We design task families with a shared agent workflow framework based on DAEF, enabling systematic evaluation of cross-task skill transfer learning. • We specify an Agentic Lifelong Learning protocol in which agents begin without skills, evolve an external skill library through trajectory- and rubric-driven skill patches , and progressively learn and are evaluated under a fixed task order within each family. • We identify and closely analyze common failure patterns in skill evolution, including fragmented skill growth, reinforcement of erroneous logic, and the gap between writing skills and reliably repairing them, revealing current models’ limitations in producing compact, transferable procedural skills. 2 SkillFlow 2.1 Benchmark Overview SkillFlow contains 20 workflow families and 166 tasks spanning five broad domains: Finance & Economics , Operations & Supply Chain , Healthcare & Life Sciences , Governance & Strategy , and Data & Document Intelligence . Each benchmark instance is defined by a family-local task sequence, a fixed within-family difficulty order, and an associated verifier style. Family sizes range from 8 to 9 tasks. Overall, the benchmark encompasses a range of realistic workplace workflows, including spreadsheet-centric planning, OCR and PDF extraction, office-document editing, compliance analysis, and quantitative reasoning. 2.2 Domain-Agnostic Execution Flow The central design principle of SkillFlow is a Domain-Agnostic Execution Flow (DAEF) : a workflow skeleton shared by a family