Resource2Skill : Distilling Executable Agent Skills from Human-Created Multimodal Resources
Yijia Fan 3,∗ Zonglin Di 1,∗ Zimo Wen 2,∗ Yifan Yang 3,+ Mingxi Cheng 3 Qi Dai 3 Bei Liu 3 Kai Qiu 3 Yue Dong 3 Ji Li 3 Chong Luo 3 1 University of California, Santa Cruz 2 Shanghai Jiao Tong University 3 Microsoft ∗ Equal contribution. + Corresponding author: [email protected] . Code Dataset Project Page Video · 2026
Read Resource2Skill : Distilling Executable Agent Skills from Human-Created Multimodal Resources by Yijia Fan 3,∗ Zonglin Di 1,∗ Zimo Wen 2,∗ Yifan Yang 3,+ Mingxi Cheng 3 Qi Dai 3 Bei Liu 3 Kai Qiu 3 Yue Dong 3 Ji Li 3 Chong Luo 3 1 University of California, Santa Cruz 2 Shanghai Jiao Tong University 3 Microsoft ∗ Equal contribution. + Corresponding author: [email protected] . Code Dataset Project Page Video (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29538.txt.
Introduction
Fan and colleagues at Microsoft present Resource2Skill, a framework that turns human-made learning materials into executable skills for software agents. Skills package experience as reusable procedural knowledge. Most existing skill libraries are hand-written, text-only, or mined from agent traces. Tutorial videos and other multimodal resources, which mix text, code, and images, stay unused.
Resource2Skill distils videos, repositories, articles, and reference artifacts into a hierarchical Skill Wiki. Each entry packs structured text, code, and visual examples together with metadata and provenance, a record of where it came from. Videos contribute timing and visual effects, code contributes tool patterns, and articles contribute concepts and style. At work time, agents retrieve and compose the skills they need. When coverage falls short, the construction process picks up new skills online. In tests across seven authoring domains, scores rise by about twelve points over agents with no skills.
Catalog record
| CREATOR | Yijia Fan 3,∗ Zonglin Di 1,∗ Zimo Wen 2,∗ Yifan Yang 3,+ Mingxi Cheng 3 Qi Dai 3 Bei Liu 3 Kai Qiu 3 Yue Dong 3 Ji Li 3 Chong Luo 3 1 University of California, Santa Cruz 2 Shanghai Jiao Tong University 3 Microsoft ∗ Equal contribution. + Corresponding author: [email protected] . Code Dataset Project Page Video |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29538
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Resource2Skill : Distilling Executable Agent Skills from Human-Created Multimodal Resources Yijia Fan 3,∗ Zonglin Di 1,∗ Zimo Wen 2,∗ Yifan Yang 3,+ Mingxi Cheng 3 Qi Dai 3 Bei Liu 3 Kai Qiu 3 Yue Dong 3 Ji Li 3 Chong Luo 3 1 University of California, Santa Cruz 2 Shanghai Jiao Tong University 3 Microsoft ∗ Equal contribution. + Corresponding author: [email protected] . Code Dataset Project Page Video Abstract Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present Resource2Skill , a framework that distills multimodal resources—tutorial videos, repositories, articles, and reference artifacts—into executable skills for software agents. Resource2Skill organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, Resource2Skill improves average overall score by + 11.9 +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 26 of 28 28 main-aggregate model–domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition. Figure 1: Resource2Skill distills multimodal resources into a hierarchical Skill Wiki across seven creative software domains. 1 Introduction Large language model agents ( Luo et al. 2025 ) are increasingly expected to do more than answer questions: they must operate software, call tools, inspect intermediate results, and produce high-quality artifacts such as slide decks, spreadsheets, web pages, 3D scenes, CAD designs, and audio projects. In these settings, success often depends less on isolated factual knowledge than on reusable procedural know-how: how to decompose a goal, which tool or API pattern to use, what intermediate state to inspect, and how to recover when an operation fails. We refer to such reusable procedural knowledge as skills . In the agent era, skills provide a natural abstraction from experience to reusable expertise : they convert observations of how tasks are solved into compact instructions, code fragments, visual references, and execution recipes that can be repeatedly invoked by future agents. Recent skill-based agent systems ( Xu and Yan 2026 ) have already shown that such abstractions can be highly valuable. Code agents and software assistants benefit from reusable tool-use routines, while manually authored skill hubs have become a practical way to encode expert conventions for document editing, programming, data analysis, and other text-heavy workflows. However, existing skill libraries are still largely limited by how they are created: they are either hand-written by experts, accumulated from an agent’s own interaction traces, or mined from text/code resources. This leaves a major source of human expertise underused. For many commercial and creative software tasks, the most natural way humans learn is not by reading static documentation, but by watching tutorials, demonstrations, and screen-recorded workflows. A video tutorial can reveal the temporal order of operations, the visual effect of each editing step, and tacit design choices that are difficult to express in text alone. This observation motivates a broader question: Can we automatically distill skills from multimodal human-created resources, especially tutorial videos, and use them to build a scalable skill library for software agents? The answer is not straightforward. Although frontier models have seen massive amounts of text during pretraining, and text resources are routinely exploited through retrieval and search, high-dimensional multimodal resources remain much harder to use effectively at inference time. Directly placing raw videos into an agent’s memory is expensive, redundant, and often impractical. A single tutorial may contain minutes of irrelevant setup, repeated narration, and visual details that are important only at a few key moments. At the same time, compressing video into a plain text summary discards precisely the information that makes video useful: dynamic operations, before-after visual changes, animation quality, spatial layout, timing, and tool interaction order. A practical system must therefore extract the procedural signal from videos and other resources, normalize it into a reusable representation, and organize the resulting knowledge so that agents can efficiently retrieve and execute the right skill for a new user request. We introduce Resource2Skill , a framework for distilling executable skills from human-created resources and organizing them into a maintainable multimodal skill library for software agents. Given multimodal resources—including tutorial videos, source repositories, articles, documentation, and reference artifacts— Resource2Skill extracts domain-specific skills and stores them in a hierarchical Skill Wiki. Each skill is represented as a multimodal entry that may contain structured text, executable or adaptable code, visual examples, metadata, and provenance. This design makes the skill library more than a flat collection of retrieved passages: text explains applicability and mechanism, code provides tool-grounded execution patterns, and visual examples preserve layout, style, motion, and other perceptual information that text alone under-specifies. A central design choice of Resource2Skill is to treat skill construction and skill use as a unified pipeline. Offline, we distill large-scale resources into a domain wiki for important commercial software scenarios. At inference time, given a user requirement, the agent first navigates the hierarchical index to form a candidate skill pool, then reads the relevant multimodal entries and composes them during execution. This hierarchical organization improves over flat retrieval because it encodes domain structure, narrows the search space, and exposes skill candidates at the appropriate level of abstraction. Moreover, when the offline library does not cover a requested capability, the same resource-to-skill operator can be invoked online to search for new resources, extract additional skills, and incrementally extend the library. The resulting system is therefore not a fixed prompt collection, but a growing and maintainable procedural memory. We study Resource2Skill across seven practical software-authoring domains, including slide design, web page generation, spreadsheet authoring, Blender scene creation, CAD design, UE5 scene construction, and music production. These domains are chosen because they require different forms of procedural knowledge: some depend heavily on code and API conventions, some on visual design and layout, and others on temporal operations that are naturally conveyed by video. Our experiments show that skill access consistently improves agent performance across model backends and domains. Across our seven authoring benchmark suites (Web, Excel, Reaper, PPT, Blender, CAD, UE5), Resource2Skill improves the average overall score by +11.9 percentage points over the same agents without skills, and outperforms strong agentic-harness baselines in 26 of 28 main-aggregate cells. Ablations further show that the hierarchical wiki interface, source diversity, multimodal skill format, library scale, online skill acquisition, and hierarchy-then-LM selection strategy each contribute to the final performance. Our contributions are summarized as follows: • Resource-to-skill learning for software agents. We formulate the problem of distilling reusable executable skills from multimodal human-created resources, with tutorial videos as a key underused source of procedural and perceptual knowledge. • A hierarchical multimodal skill library. We propose a Wiki-based organization in which each skill combines structured text, executable or adaptable code, visual examples, metadata,