When Skills Don’t Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity Conference: The First Workshop on Agent Skills, co-located with ACM CAIS 2026; May 26, 2026; San Jose, CA, USA CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Planning under uncertainty CCS: Security and privacy Vulnerability management
Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu · 2026
Read When Skills Don’t Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity Conference: The First Workshop on Agent Skills, co-located with ACM CAIS 2026; May 26, 2026; San Jose, CA, USA CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Planning under uncertainty CCS: Security and privacy Vulnerability management by Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20023.txt.
Introduction
Samuel Jacob Chacko and three co-workers re-analyse a controlled study of a tool-using capture-the-flag agent to ask when procedural skill packs actually help. In offensive cybersecurity, where such packs are thin, the no-skill to full-skill gap is small and statistically weak. They argue the missing factor is feedback bandwidth: when tools return strict, schema-checked, fast observations, the environment itself supplies the correction skills normally provide. In some settings, such as a timing side channel, added skills even degrade performance.
Catalog record
| CREATOR | Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23839 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20023
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Skills Don’t Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity Conference: The First Workshop on Agent Skills, co-located with ACM CAIS 2026; May 26, 2026; San Jose, CA, USA CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Planning under uncertainty CCS: Security and privacy Vulnerability management Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu Abstract . Agent Skills, structured packages of procedural knowledge loaded into an LLM agent at inference time, are widely reported to improve task pass rates by an average of 16.2 percentage points across diverse domains. Yet the same benchmarks show wide variance, with 16 of 84 tasks suffering negative deltas when Skills are introduced. The community has not yet articulated a clean mechanism for when Skills help and when they are merely redundant overhead. We re-analyze a recently published 180-run controlled study of an MCP-grounded autonomous Capture-the-Flag (CTF) agent under four documentation conditions of increasing richness (591, 12865, 17253, and 36001 tokens) and show that these conditions correspond almost exactly to a No-Skills, Experiential-Skills, Curated-Skills, and Comprehensive-Skills ablation. In offensive cybersecurity, a domain not deeply covered by existing Skills benchmarks, the marginal benefit of Skills collapses. The spread between the no-Skills and full-Skills conditions is only 8.9 pp ( p = 0.71 p=0.71 , χ 2 \chi^{2} ; p = 0.25 p=0.25 , Cochran–Armitage trend test; five of six pairwise Cohen’s h h values fall below the 0.2 0.2 small-effect threshold). We argue that the missing variable is environment-feedback bandwidth . When an agent’s tool layer returns strict, schema-validated, low-latency observations, the environment itself supplies the procedural correction signal that Skills are normally needed to provide. As a result, the marginal benefit of curated Skills diminishes substantially, and, in some cases (e.g., our timing side-channel setting), actively degrades performance. We articulate a falsifiable hypothesis, sketch its design implications for compound AI systems, and will release the reanalysis pipeline to support replication. Position Short Paper 1. Introduction Agent Skills ( 2 ; 1 ) have emerged as a unifying abstraction for the procedural knowledge that LLM agents consult at inference time. A Skill is a folder containing a SKILL.md file with metadata and instructions, optionally bundled with scripts and references, that an agent loads on demand. The format was released as an open standard in late 2025 and has since been adopted across more than thirty agent platforms ( 6 ) , with community marketplaces hosting tens of thousands of community-authored Skills. The premise is intuitive: foundation models supply broad capability, and Skills supply the domain-specific recipes the model otherwise lacks. The first systematic empirical evidence on whether Skills work at scale arrived with SkillsBench ( 7 ) , a benchmark of 84 tasks across 11 domains, evaluated across 7 agent–model configurations and 7,308 trajectories. Two findings define the current state of the field. First, curated Skills add a substantial average of +16.2 percentage points (pp) to pass rates. Second, that average hides extreme variance: domain effects range from +51.9 pp (healthcare) down to +4.5 pp (software engineering), and 16 of 84 individual tasks show negative deltas when Skills are provided. The authors of SkillsBench are explicit that the field lacks a mechanistic account of when Skills help, when they are inert, and when they degrade performance. This paper offers one such account, grounded in a re-analysis of a recently published 180-run controlled study of an autonomous CTF agent ( 5 ) that was designed and run before the SKILL.md specification was published, but whose four documentation conditions correspond, structurally and operationally, to a clean Skills ablation. The agent is built on the Model Context Protocol (MCP) ( 3 ) , a schema-validated tool interface that returns strictly typed observations from a containerized arsenal of reverse-engineering, web exploitation, and binary-analysis tools. This schema-enforcement principle is analogous to grammar-constrained decoding ( 4 ) , which has been shown to substantially reduce hallucinated outputs in structured prediction. We argue that this MCP-grounded execution stack is the moderating variable that has been missing from Skills evaluation: rich, deterministic, low-latency tool feedback substitutes for a substantial fraction of the procedural guidance that curated Skills are normally needed to supply. We make three contributions: • A new domain entry. We extend the empirical landscape of Skills evaluation into offensive cybersecurity (binary exploitation, web exploitation, reverse engineering, cryptography), a domain underrepresented in current Skills benchmarks. Our re-analysis covers 180 trajectories across 15 challenges and a four-level documentation ablation. • A null result at the sample size studied. On this domain, structured procedural knowledge, spanning attack templates, distilled lessons, and a comprehensive Skills bundle, adds at most +8.9 pp over an MCP-only baseline (95% CI for the difference: [-6.8,+24.6] pp). None of the differences reach statistical significance ( p=0.71 , \chi^{2} ; p=0.25 , Cochran–Armitage trend test; five of six pairwise Cohen’s h values fall below the 0.2 small-effect threshold, with No-Skills vs. Comprehensive at h=0.23 ). We do not claim a causal demonstration of zero effect; we claim only that any effect in this domain is below the resolution of a 180-run controlled study. The Comprehensive-Skills condition is also outperformed by a Curated-Skills condition on at least one challenge class. • The feedback-bandwidth hypothesis. We articulate a falsifiable mechanistic hypothesis: the marginal benefit of Skills is inversely related to the bandwidth of deterministic environment feedback available to the agent. We sketch the implications for compound AI system design and propose a concrete experimental program to test the hypothesis at scale. The paper proceeds as follows. Section 2 positions the four documentation conditions of the source study as a Skills ablation. Section 3 presents the re-analyzed numbers, including a case-study where the Comprehensive-Skills condition underperforms a narrower Curated-Skills condition. Section 4 articulates the feedback-bandwidth hypothesis and its testable predictions. Section 5 situates the result in the Skills literature and draws implications for practitioners building MCP-grounded agents. Section 6 addresses limitations. 2. Reframing the Study as a Skills Ablation The source study ( 5 ) evaluated an MCP-grounded LLM agent on 15 CTF challenges (memory corruption, reverse engineering, web exploitation, cryptography) under four conditions defined by the type and volume of context-attached procedural documentation. Each condition was run on every challenge with three independent trials, yielding 15\times 4\times 3=180 trajectories. The model (Claude Sonnet 4.5 with extended thinking) and the tool layer (Nmap, Ghidra, Angr, GDB, exposed via MCP servers with strict JSON-schema validation) were held constant; only the procedural documentation varied. Importantly, each challenge is a long-horizon task ( 8 ) that requires sequential tool invocation, hypothesis revision, and multi-phase exploitation. Trajectories average 17–20 minutes per run, making 15 challenges a substantive evaluation despite the modest count. We map these conditions onto the standard Skills taxonomy: • No-Skills (Minimal, 55 lines). Only MCP server definitions and tool schemas. No SKILL.md -like procedural guidance. This is the No-Skills condition in the SkillsBench taxonomy ( 7 ) . • Experiential-Skills (Lessons, 1,478 lines). Lessons distilled from prior debugging sessions (heap exploitation pitfalls, web-framework patterns, etc.) plus a triage classifier. Structurally, this is a self-distilled, trajectory-mined Skills bundle, procedural knowledge after the fact . • Curated-Skills (Templates, 1,976 lines). A structured multi-phase solving protocol with attack templates indexed by