Evidence Over Plans: Online Trajectory Verification for Skill Distillation
Yang Zhou Zihan Dong Zhenting Wang Can Jin Shiyu Zhao Bangwei Guo Difei Gu Linjun Zhang Mu Zhou Dimitris N. Metaxas Rutgers University [email protected] [email protected] · 2026
Read Evidence Over Plans: Online Trajectory Verification for Skill Distillation by Yang Zhou Zihan Dong Zhenting Wang Can Jin Shiyu Zhao Bangwei Guo Difei Gu Linjun Zhang Mu Zhou Dimitris N. Metaxas Rutgers University [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09192.txt.
Introduction
Zhou and nine colleagues argue agent skills should be distilled from evidence, not advance plans. Skills, the guides that steer agents, can greatly raise task success, but their quality is hard to judge without environment checks. Existing methods lean on preference logs instead of direct environment contact, often giving weak or degraded gains.
The team identifies a timing bottleneck: robust skills should be evidence based, distilled from real contact rather than prior plans. They introduce the Posterior Distillation Index, a path level metric judging how firmly a distilled skill rests on evidence. To compute it, they present SPARK, which generates environment checked paths and applies the index as a live diagnostic signal. Across 86 runnable tasks, SPARK skills beat no skill baselines and top human written skills on student models, at lower cost than teachers. The findings show index guided distillation produces efficient, portable skills grounded in real contact.
Catalog record
| CREATOR | Yang Zhou Zihan Dong Zhenting Wang Can Jin Shiyu Zhao Bangwei Guo Difei Gu Linjun Zhang Mu Zhou Dimitris N. Metaxas Rutgers University [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evidence Over Plans: Online Trajectory Verification for Skill Distillation Yang Zhou Zihan Dong Zhenting Wang Can Jin Shiyu Zhao Bangwei Guo Difei Gu Linjun Zhang Mu Zhou Dimitris N. Metaxas Rutgers University [email protected] [email protected] Abstract Agent skills can remarkably improve task success rates by using human-written procedural documents, but their quality is difficult to assess without environment-grounded verification. Existing skill generation methods heavily rely on preference logs rather than direct environment interaction, often yielding negligible or even degraded gains. We identify that it is a fundamental timing bottleneck: robust skills should be posterior-based , distilled from empirical environment interaction rather than prior plans. In this study, we introduce the Posterior Distillation Index (PDI) , a trajectory-level metric that quantifies how well a distilled skill is grounded in the task-environment evidence. To operationalize PDI, we present SPARK ( S tructured P ipelines for A utonomous R unnable tas K s and s K ill generation) for preserving task execution evidence towards full trajectory-level analysis. SPARK generates environment-verified trajectories used to compute PDI, and it applies PDI as an online diagnostic and intervention signal to ensure posterior skill formation. Across 86 runnable tasks, SPARK-generated skills consistently surpass no-skill baselines and outperform human-written skills on student models (inference cost up to 1,000 × \times cheaper than teacher models). These findings show that PDI-guided distillation produces efficient and transferable skills grounded in the task-environment interaction. We release our code in SPARK . 1 Introduction Large Language Model (LLM) agents are increasingly deployed on wide-ranging interactive software tasks [ 25 , 17 , 30 , 29 , 7 ] . An emerging strategy is to equip agents with skills [ 8 ] , defined as in-context procedural documents that encode task-specific knowledge such as execution ordering, environment constraints, and verifier contracts. Benchmarks [ 8 , 3 ] demonstrate that prior-based, human-written skills can remarkably boost task success rates, making such reusable procedural knowledge a promising abstraction for building scalable agent systems. Yet a pessimistic picture appears that the quality of these skills is nearly impossible to assess without environment-grounded verification [ 8 ] . Researchers propose lifelong skill accumulation [ 24 ] , large-scale skill cataloging [ 9 ] , or RL-based skill evolution [ 22 ] . These studies lack trajectory verification and solely rely on preference logs. As a result, they can not produce transferable skills between teacher and student models at different capabilities. This raises a fundamental question: What makes skill distillation transferable across tasks and models efficiently and reliably? We identify that a key bottleneck is the timing of skill formation relative to the task-environment interaction. Existing skill generation [ 21 , 9 ] asks the model to prescribe how a task should be solved before interacting with the environment, yielding skills dominated by generic priors and heuristics. Instead, we propose that effective procedural knowledge should be posterior-based since its creation time: it must encode environment-specific constraints, execution dependencies, and failure modes discoverable only through environmental interactions. Once distilled, this posterior experience becomes reusable and verifiable guidance for future agent development. Figure 1: PDI-based SPARK Illustration. 1) Left (Skill Generation) : Starting from a task description, a teacher agent (e.g., Claude Opus 4.6) interacts with a Dockerized environment (up to N_{\max} attempts) and updates an exploration memo from execution feedback. Upon success, the full trajectory trace is distilled into SKILL.md . Upon failure, a PDI-based proxy triggers targeted interventions. 2) Right (Task Construction): To validate the quality of skill generation, this is a supporting pipeline that converts prompt into oracle-verified benchmark instances via blueprint generation and critique checks. Then, a student agent (e.g., GPT 5.4 mini) evaluates the generated skill on these independently constructed tasks, testing whether PDI-grounded knowledge transfers across tasks rather than overfitting to the teacher’s original trajectory. We introduce the Posterior Distillation Index (PDI) 1 1 1 ‘Posterior’ denotes knowledge distilled from environment interaction. No probabilistic or Bayesian update is assumed. as a principled metric that quantifies whether skills are grounded in environment-verified evidence rather than purely prior-based plans. By design, PDI aggregates three interpretable trajectory-level signals, including execution grounding, plan copying, and memo ossification, using an equal-weight linear combination. We deliberately avoid nonlinear fitting to ensure benchmark independence. To operationalize PDI, we present SPARK ( S tructured P ipelines for A utonomous R unnable tas K s and s K ill generation). In Figure 1 , SPARK serves dual roles: it generates the environment-grounded trajectories from which PDI is computed; and it uses PDI as an online intervention signal to improve skill generation before its completion. Our main contributions are as follows. (1) Trajectory-verified Metric : PDI introduces a trajectory-level measure whether generated skills are grounded in task-environment evidence rather than unverified prior plans. PDI also provides a mechanistic assessment of which trajectories yield truly transferable skills ( Section 4.2.5 ). (2) Skill Generation : SPARK makes skill generation transparent and analyzable by preserving execution logs, verifier signals, and memo histories towards full trajectory-level analysis ( Section 3.1.2 ). (3) Online Intervention : A memo-based proxy of PDI enables an online monitoring and intervention during skill exploration. Our primary with/without PDI ablation quantifies the performance gain enabled by this online intervention ( Section 4.3 ). (4) Trajectory Benchmark : We provide a trajectory-level benchmark with full execution traces, verifier signals, memo histories, and distilled skills. This collected environment-verified evidence allows for evaluating skill transfer ability across environment interactions. 2 Related Work Agent skill acquisition and evaluation. Agent skill acquisition has increasingly shifted from collecting experiences to distilling and deploying skills across tasks. Voyager [ 21 ] accumulates executable skills in an embodied environment. ExpeL [ 27 ] stores natural-language lessons from prior failures. Growing attempts distill the experience into transferable skill libraries, including AutoRefine [ 15 ] , Trace2Skill [ 12 ] , AutoSkill [ 24 ] , and CUA-Skill [ 4 ] . SkillRouter [ 28 ] further addresses the deployment by selecting highly-relevant skills from large registries. Beyond skill acquisition, skill evaluation is crucial to assess the quality of skills in real-world applications. SkillsBench [ 8 ] evaluates when curated skills help at inference time. SkillWild [ 10 ] studies more realistic retrieval-and-adaptation settings. Unlike these efforts, SPARK emphasizes interaction-grounded skill documents, cross-instance skill transfer, and trajectory-level analysis. LLM agents with external feedback. SPARK follows the interact–reflect–retry line of agentic system development, including ReAct [ 25 ] , Reflexion [ 17 ] , and ETO [ 19 ] . These studies mainly improve the decision making or trajectory revision within a single task-solving episode (one attempt at completing a given task). By contrast, we emphasize how agent execution traces can be distilled [ 6 , 5 , 7 ] from the environment interaction that can transfer across tasks, turning trial-and-error records into a reusable knowledge base. Agent tooling and skill utility. LLM agent tooling and its transfer capability have evolved from learning tool policies to structuring skills. For instance, Toolformer [ 16 ] and ToolLLM [ 14 ] focus on learning to invoke tools. Meanwhile, CREATOR [ 13 ] and LATM [ 2 ] show that stronger models can synthesize reusable artifacts for weaker ones. Skill-oriented frameworks go further by packaging capabilities into modular, self-contained