ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation

Boqin Yuan Renchu Song Yue Su Sen Yang Jing Qin · 2026

Read ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation by Boqin Yuan Renchu Song Yue Su Sen Yang Jing Qin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23853.txt.

Introduction

Yuan and colleagues add cost awareness to skill distillation for language-model agents, where reusable rules are learned from agent trajectories. Their ClawTrace system records cost-attributed traces as TraceCards, and CostCraft turns them into preserve, prune, and repair patches. Tests reveal a hidden pattern: prune patches guard quality, while preserve patches cause regressions. The authors argue skills should be judged by rule type, not as whole packages.

Catalog record

CREATORBoqin Yuan Renchu Song Yue Su Sen Yang Jing Qin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45051
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation Boqin Yuan Renchu Song Yue Su Sen Yang Jing Qin Abstract Skill-distillation pipelines learn reusable rules from LLM agent trajectories, but they lack a key signal: how much each step costs. Without per-step cost, a pipeline cannot distinguish adding a missing step to fix a bug from removing an expensive step that never affected the outcome. We introduce ClawTrace , an agent tracing platform that records every LLM call, tool use, and sub-agent spawn during an agent session and compiles each session into a TraceCard : a compact YAML summary with per-step USD cost, token counts, and redundancy flags. Built on ClawTrace, CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches. Preserve patches keep behaviors that led to success. Prune patches remove expensive steps that did not matter, each backed by a counterfactual argument against a named high-cost step. Repair patches fix failures grounded in oracle evidence. Ablations on 30 held-out SpreadsheetBench tasks show that both cost attribution and prune patches independently reduce quality regressions. When the same skill is applied to 30 unrelated SkillsBench tasks, an unexpected asymmetry emerges: prune rules transferred across benchmarks and cut median cost by 32%, while preserve rules, trained on benchmark-specific conventions, caused regressions on new task types. We release ClawTrace and TraceCards as open infrastructure for cost-aware agent research. Code ClawTrace Figure 1: End-to-end architecture. Capture : ClawTrace instruments an agent session via eight event hooks. Compile : a deterministic compiler produces a TraceCard per session. Distill : CostCraft emits preserve, prune, and repair patches that merge into an evolved SKILL.md . 1 Introduction LLM agents can be improved by giving them skills: structured instruction packages that guide their behavior without changing model weights [ 18 ] . Recent work shows that these skills can be automatically distilled from agent execution traces [ 13 , 26 , 25 ] . On SkillsBench [ 11 ] , curated skills raise the mean pass rate by 16.2 percentage points. But the gains are uneven: 16 of 84 tasks get worse, and self-generated skills offer no average benefit. The aggregate numbers hide offsetting wins and losses. A central reason for these regressions is that existing distillation pipelines treat all improvements the same way. They split trajectories into successes and failures, then extract rules from each group [ 13 ] . This split misses an important distinction. Fixing a bug by adding a missing step to a failed trajectory is not the same operation as cutting waste by removing an expensive but unnecessary step from a successful trajectory. The second requires knowing how much each step cost, so the pipeline can identify which steps were expensive and argue that removing them would not hurt quality. Existing observability tools (LangSmith [ 8 ] , Langfuse [ 9 ] , Phoenix [ 2 ] ) do track per-span token counts and cost, following OpenTelemetry conventions [ 15 ] . But they surface this information as dashboard analytics for human operators, not as a compact intermediate representation that a distillation pipeline can consume. A skill-mining analyst needs per-step cost ranked and labeled with redundancy flags and failure markers, packaged into a format small enough to fit many sessions into a single context window. This paper contributes on two fronts. First, ClawTrace is an open-source tracing platform that instruments agent sessions through eight event hooks, capturing every LLM call, tool use, and sub-agent spawn. It links child agent sessions back to their parent spans and accounts for cached tokens at their actual billed rate. From these raw events, ClawTrace compiles a TraceCard for each session: a compact ( {\sim} 1.5 kB) YAML summary listing per-step USD cost, token counts by type, redundant tool-call clusters, and failed-or-repaired steps. TraceCards are designed as a reusable intermediate representation that any downstream pipeline, not only CostCraft, can consume for cost-aware analysis. The ingest API accepts plain JSON with no framework dependency, so any agent harness that posts events to the endpoint gets valid TraceCards without changes to ClawTrace. Second, CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches. Preserve patches keep behaviors that led to success. Prune patches remove expensive steps that did not affect the outcome; each prune patch must name the specific high-cost step it targets and provide a counterfactual argument for why removal is safe. Repair patches fix failure modes using oracle evidence. This three-way split replaces Trace2Skill’s [ 13 ] success-versus-error split with a correctness-versus-efficiency split, and the merge step ranks causally grounded rules above unsupported ones. We evaluate on 30 held-out SpreadsheetBench [ 12 ] tasks and 30 cross-benchmark SkillsBench [ 11 ] tasks. In our single-seed evaluation, ablations show that cost attribution and prune patches each carry independent weight: removing either one sharply increases quality regressions. The cross-benchmark evaluation reveals that prune and preserve rules transfer differently, a distinction prior work does not make. We scope these claims to a methods paper with small-scale validation on a single backbone and release all code, TraceCards, and evolved skills to support reproducibility and follow-up research. 2 Related Work Agent skills and benchmarks. Anthropic [ 18 ] introduced agent skills as structured packages that augment an LLM agent without weight updates. Xu and Yan [ 24 ] distinguish skills from atomic tools and one-off plans, defining them as reusable modules with applicability conditions. SkillsBench [ 11 ] provides the first systematic benchmark: curated skills raise mean pass rate by 16.2 percentage points, but self-generated skills offer no average benefit. SWE-Skills-Bench [ 5 ] evaluates skill injection in software-engineering tasks and reports performance drops under context mismatch. SpreadsheetBench [ 12 ] provides 912 real-world tasks with deterministic cell-match grading and is our primary evaluation benchmark. Trajectory mining and skill distillation. Trace2Skill [ 13 ] is the closest prior work. It dispatches parallel analyst sub-agents under a success-versus-error split, with the error analyst using a multi-turn ReAct loop with oracle access. We adopt its analyst structure but replace the two-way split with a three-way split that adds prune patches for removing waste from successful trajectories. CoEvoSkills [ 26 ] couples a Skill Generator with a co-evolving Surrogate Verifier, achieving 71.1% pass rate on SkillsBench through per-task iterative optimization rather than cross-task distillation. EvoSkill [ 1 ] iteratively diagnoses failures and validates skill updates with ground-truth supervision. AutoSkill [ 25 ] maintains a skill lifecycle from user chat trajectories. SkillWeaver [ 27 ] generates web API skills through structured exploration. These systems address different settings from our offline cross-task distillation. Experience memory and observability. Voyager [ 20 ] accumulates skills through open-ended interaction. Reflexion [ 19 ] refines decisions through verbal self-reflection. ReasoningBank [ 16 ] stores trajectory-derived insights in a retrieval bank; Trace2Skill shows that distillation into a compact skill document outperforms retrieval by 13.8 percentage points. SkillRL [ 23 ] co-evolves skills and policies via reinforcement learning; we study frozen-model, training-free adaptation only. On the observability side, LangSmith, Langfuse, Phoenix, and Arize build on OpenTelemetry [ 15 ] conventions to provide step-level logs and per-span token costs. Dong et al. [ 4 ] survey the emerging AgentOps landscape and identify cost tracking and

Read the complete text