SkillFlow: Scalable and Efficient Agent Skill Retrieval System
Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos · 2025
Read SkillFlow: Scalable and Efficient Agent Skill Retrieval System by Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06188.txt.
Introduction
Fangzhou Li and co-authors build SkillFlow, a search system that fetches the right skills for an AI agent. Agents can load reusable skill files at run time, but loading too many, or the wrong ones, hurts performance. Their 2025 paper treats skill picking as a retrieval problem over about thirty-five thousand community skill files gathered from GitHub.
The pipeline narrows candidates in four stages: dense retrieval, two rounds of reranking, then a model-based final pick. On a skills benchmark, fetched skills raised task pass rates sharply and reached most of the oracle ceiling. A second benchmark with no matched skills still gained. Agent builders get an open design for skill libraries that keep growing.
Catalog record
| CREATOR | Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06188
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SkillFlow: Scalable and Efficient Agent Skill Retrieval System Fangzhou Li, Pagkratios Tagkopoulos, Ilias Tagkopoulos Abstract AI agents can extend their capabilities at inference time by loading reusable skills into context, yet equipping an agent with too many skills, particularly irrelevant ones, degrades performance. As community-driven skill repositories grow, agents need a way to selectively retrieve only the most relevant skills from a large library. We present SkillFlow , the first open, multi-stage retrieval system for agent skill discovery that frames skill acquisition as an information retrieval problem over a corpus of ∼ \scriptstyle\sim 35K community-contributed SKILL.md definitions indexed from GitHub. The pipeline progressively narrows a large candidate set through four stages (dense retrieval, two rounds of cross-encoder reranking, and LLM-based selection), balancing recall and precision at each stage. We evaluate SkillFlow on two coding benchmarks: SkillsBench, a benchmark of 87 tasks and 229 matched skills; and Terminal-Bench, a benchmark that provides only 89 tasks, and no matched skills. On SkillsBench, SkillFlow-retrieved skills raise Pass@1 from 9.2% to 16.4% (+78.3%, p adj = 3.64 × 10 − 2 p_{\text{adj}}=3.64\times 10^{-2} ), reaching 84.1% of the oracle ceiling, while on Terminal-Bench, agents readily use the retrieved skills (70.1% use rate) yet show no performance gain, revealing that retrieval alone is insufficient when the corpus lacks high-quality, executable skills for the target domain. SkillFlow demonstrates that framing skill acquisition as an information retrieval task is an effective strategy, and that the practical impact of skill-augmented agents hinges on corpus coverage and skill quality, particularly the density of runnable code and bundled artifacts. ( https://github.com/IBPA/skill-flow ) † † footnotetext: \ast Equal contribution. † † footnotetext: \dagger Correspondence to: [email protected] 1 Introduction AI agents (systems that perceive their environment and act autonomously to achieve specific goals ( Wooldridge & Jennings, 1995 ; Russell & Norvig, 2020 ; Franklin & Graesser, 1996 ) ) have been a long-standing pursuit, from ancient automata ( Price, 1964 ) to the intelligent agent programs of the 1990s and today’s large language model (LLM) based agents ( Durante et al., 2024 ) . LLMs have dramatically expanded what agents can do, enabling them to reason over complex environments, generate and execute code, and solve multi-step tasks that were previously out of reach. Despite these advances, agents today are typically equipped with a fixed set of capabilities, and the question of how they acquire and integrate new ones remains largely open. Agent Skills, introduced by Anthropic ( Anthropic, 2025 ) , handles this by defining a standardized format (SKILL.md) for encoding reusable procedural knowledge. The convention has been widely adopted by the open-source community, with a rapidly growing number of skill definitions shared across repositories on GitHub. However, agents must load skills into their context at inference time, and equipping them with too many skills, particularly irrelevant ones, degrades both accuracy and efficiency ( Li et al., 2026 ; Gao, 2026 ) . Simply pre-loading all available skills is therefore impractical. What is needed is a mechanism that can selectively retrieve only the most relevant skills from a large and growing library. In this work, we address this retrieval challenge and present SkillFlow , a multi-stage pipeline that, given a task description, efficiently identifies and retrieves relevant skills from a large library of community-contributed skill definitions. The pipeline consists of four stages: dense retrieval, shallow reranking, deep reranking, and final selection ( Figure 1 ); each progressively refining the candidate set to balance recall and precision. We evaluate SkillFlow on two coding benchmarks: on SkillsBench, retrieved skills raise Pass@1 from 9.2% to 16.4% ( +78.3\% , p_{\text{adj}}=3.64\times 10^{-2} ), reaching 84.1% of the oracle ceiling. On Terminal-Bench, however, agents readily adopt retrieved skills (70.1% use rate) yet show no performance gain, revealing that retrieval alone is insufficient when the corpus lacks high-quality skills for the target domain. This contrast suggests that the primary factor limiting skill-augmented agents is not retrieval but the quality and coverage of the underlying skill library. Our contributions are as follows: 1. We frame agent skill acquisition as information retrieval over a large, real-world corpus of community-contributed SKILL.md definitions and, to our knowledge, are the first to study skill retrieval at this scale. We build a standard multi-stage cascade (dense retrieval \rightarrow shallow reranking \rightarrow deep reranking \rightarrow selection) for this setting and claim no novel architecture, using a standard cascade by design, so the results reflect skill retrieval itself rather than bespoke choices. 2. We construct an openly available skill library by indexing \scriptstyle\sim 35K community-contributed SKILL.md files from GitHub, providing a reusable resource for skill-augmented agents. 3. We evaluate SkillFlow on two benchmarks and, through a contrasting null result on Terminal-Bench, demonstrate that skill library quality, and not retrieval, is the primary bottleneck for skill-augmented agents. A structural comparison of 229 oracle skills against \scriptstyle\sim 35K community-authored skills identifies the gap precisely: oracle skills contain significantly more executable code ( p_{\text{adj}}=3.38\times 10^{-12} ) and are roughly 2.5 times more likely to bundle runnable scripts ( p_{\text{adj}}=5.57\times 10^{-18} ), while length and documentation volume do not differ. These findings translate directly into measurable authoring guidelines about skill contribution and retrieval. Figure 1: Overview of the SkillFlow pipeline. A user task is passed to an agent, which invokes SkillFlow to retrieve relevant skills from a library of 35K community-contributed skill definitions. The pipeline progressively narrows the candidate set through four stages: a bi-encoder retriever (1K skill candidates), a shallow cross-encoder reranker (100), a deep cross-encoder reranker (10), and an LLM-based selector ( \leq 5 skills). Retrieved skills are returned to the agent to augment task execution. 2 Related Work Skill Acquisition and Tool Discovery. LLM agents increasingly leverage external capabilities at inference time. VOYAGER ( Wang et al., 2023 ) synthesizes and retrieves skills via embedding similarity, but its library is self-generated rather than drawn from a shared corpus. Tool-calling approaches (Toolformer ( Schick et al., 2023 ) , Gorilla ( Patil et al., 2023 ) , ToolLLM ( Qin et al., 2024 ) , and Chen et al. (2025) ) retrieve atomic API signatures, whereas SkillFlow retrieves higher-level procedural knowledge: structured SKILL.md bundles that may include multi-step instructions, executable scripts, and contextual references. Agent Skills ( Anthropic, 2025 ) formalized this bundle format, and a rapidly growing ecosystem now hosts tens of thousands of community-contributed definitions. Despite this growth, existing skill-augmented agents assume a small, manually curated skill set or rely on brute-force enumeration; scalable retrieval over a large, heterogeneous skill library has not been addressed. Multi-Stage Retrieval and Reranking. Retrieval-augmented generation ( Lewis et al., 2020 ) has evolved from sparse methods like BM25 ( Robertson & Zaragoza, 2009 ) to dense bi-encoders ( Karpukhin et al., 2020 ) , with cross-encoder rerankers trading throughput for accuracy ( Nogueira & Cho, 2019 ; Nogueira et al., 2019 ) . SkillFlow adopts this multi-stage architecture but operates over structured skill definitions (documents mixing natural-language descriptions with YAML metadata and embedded code) rather than homogeneous text passages, and uses a two-tier cross-encoder cascade to keep latency practical for interactive agent workflows. 3 SkillFlow Given a task description t and a skill library \mathcal{S} of N skills, the goal is to return a small subset \mathcal{S}^{*}\subset\mathcal{S} of the