Toward User Comprehension Supports for LLM Agent Skill Specifications Conference: ACM CAIS Workshop on Design, Evaluation, and Optimization of Procedural Knowledge for LLM Agents; May 26, 2026; San Jose, CA, USA CCS: Security and privacy Usability in security and privacy CCS: Computing methodologies Artificial intelligence
Zikai Alex Wen · 2026
Read Toward User Comprehension Supports for LLM Agent Skill Specifications Conference: ACM CAIS Workshop on Design, Evaluation, and Optimization of Procedural Knowledge for LLM Agents; May 26, 2026; San Jose, CA, USA CCS: Security and privacy Usability in security and privacy CCS: Computing methodologies Artificial intelligence by Zikai Alex Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19362.txt.
Introduction
Zikai Wen asks whether skill specifications actually help users grasp what a language-model agent skill does. Audits usually hunt for malicious skills, while this study checks whether specs state a skill's inputs, outputs, and scope. Across 878 cybersecurity skills, cues for how a skill works were common, yet only 19.0 percent showed an example task or outcome, and only 2.3 percent covered all four comprehension anchors. A small telemetry subset suggests examples ease first local checks, so specs should be judged as user-facing disclosures, not just code containers.
Catalog record
| CREATOR | Zikai Alex Wen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20781 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toward User Comprehension Supports for LLM Agent Skill Specifications Conference: ACM CAIS Workshop on Design, Evaluation, and Optimization of Procedural Knowledge for LLM Agents; May 26, 2026; San Jose, CA, USA CCS: Security and privacy Usability in security and privacy CCS: Computing methodologies Artificial intelligence Zikai Alex Wen Abstract . Users often interpret and select agent skills through their SKILL.md specifications. To protect users, existing audits mainly focus on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we used rule-based coding to measure textual cues for four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Cues for operational basis were common, but only 19.0% of specifications exhibited cues for an example task, sample, or expected outcome, and only 2.3% exhibited cues for all four anchors. We further examined a small DNS/C2 telemetry subset (n = = 6) to illustrate why missing examples may matter. Examples appeared to make first local checks easier to construct, while no-example skills typically required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions. \@ACM@balancefalse Rare User-Facing Demonstrations Across 878 Cybersecurity Skills 1. Introduction Agent skills have emerged as a distribution format for procedural knowledge that LLM agents can load on demand. Public repositories now distribute thousands of YAML and Markdown skill packages for compatible agents ( 8 ) . Cybersecurity is one of the popular categories, covering secure code review, CWE triage, DNS tunneling detection, and threat-intelligence extraction. The stakes are high because users may install these skills to perform tasks they cannot independently verify. In this setting, the user-facing SKILL.md file does double duty. It is an instruction file that tells an agent how to perform a task, but it is also the document that people read when deciding whether a skill fits their task and risk tolerance. A skill can describe how to review source code, analyze DNS logs, or produce a structured security report. If the specification does not convey what the skill is for, what evidence it expects, and what result the agent may produce, users can run skills they should not, or trust outputs that exceed the skill’s actual coverage. Recent LLM agent skill audits ( 20 ; 11 ; 14 ) have examined SKILL.md files for adversarial behavior, including prompt injection, unsafe tool use, and malicious behavior, with the goal of protecting users from unsafe or malicious skills. Nonetheless, an audit for malicious behavior is not the same as a comprehension aid. Showing that a skill is non-malicious does not ensure that users can run it with appropriate expectations. We drew on description-to-permission fidelity research, which compared user-facing app descriptions against technical access ( 10 ; 12 ; 16 ) , as a methodological analogue. For agent skills, the relevant comparison is not between descriptions and permissions, but between specification text and operational capability. We investigated a concrete precondition for user comprehension. Can a user form a usable expectation about the skill and construct a first local check from the specification itself? This precondition aligns with prior findings on usable security disclosures, which succeed only when users can attend to, understand, and act on them ( 2 ; 15 ; 6 ) . A skill that supports comprehension does not merely state “detect DNS tunneling”. It may name the expected input artifact, state the output shape, disclose relevant limits, and provide a small example, fixture, test case, or expected output that makes the behavior concrete. A less supportive skill may still work, but the user may have to inspect helper code or infer hidden assumptions. Our contribution is threefold. Firstly, we characterized four user-facing comprehension supports in agent skill specifications and quantified their prevalence across 878 cybersecurity SKILL.md files. Secondly, we examined a local DNS/C2 telemetry subset and observed that example demonstrations made first local checks more legible from the specification. Thirdly, we used these findings to identify emerging questions about skill-specification design and user comprehension supports. 2. Related Work Our work builds on three lines of prior research. Existing skill-safety audits targeted the same artifact we study (i.e., SKILL.md ), but focused on adversarial behavior, such as hidden instructions, unsafe tool use, or attempts to leak data. User-facing comprehension was largely outside their scope. In contrast, research on security and privacy disclosures shows that user comprehension is critical and should be treated as an essential evaluation criterion as well. Building on this insight, we adapted the logic of description-to-permission fidelity from app ecosystems to ask whether specifications expose the operational anchors users need to evaluate a skill. 2.1. Skill-Safety Audits Recent projects ( 20 ; 11 ; 14 ) have emerged to audit agent skills for security risks in their specifications and helper code. For example, Agent Audit ( 20 ) and skill-audit ( 11 ) statically analyzed SKILL.md files for adversarial behavior. In this context, adversarial behavior refers to specification content that tries to steer the agent away from the user’s intent or security policy, for example by embedding prompt-injection instructions, requesting dangerous shell or network actions, concealing side effects, or exfiltrating secrets. DynAuditClaw ( 14 ) extended this approach to dynamic execution by cloning the agent installation into an isolated container and judging adversarial scenarios against a three-axis taxonomy. Our study complements this body of work by examining whether a non-adversarial skill specification provides users with sufficient information to construct an initial capability check. 2.2. Disclosure Comprehension Research on security and privacy disclosures shows that notices are most useful when they support user attention, comprehension, and decision-making. Felt et al. ( 2 ) found that Android permission prompts received little attention and were poorly understood by many users. Schaub et al. ( 15 ) systematized the design space for privacy notices and emphasized usable, useful notice for different audiences and contexts. Labels can improve access to information, but confusing terminology and information structure can impair comprehension ( 6 ; 21 ; 7 ) . In this work, we treated SKILL.md as an analogous notice artifact for agent capabilities. 2.3. Description-to-Permission Fidelity WHYPER ( 10 ) introduced an NLP pipeline for identifying whether Android app descriptions explained the need for sensitive permissions, framing the task as a way to bridge the semantic gap between user expectations and application functionality. AutoCog ( 12 ) generalized this into “description-to-permission fidelity” and measured low-fidelity descriptions across 45,811 Android applications. TKPERM ( 16 ) later transferred permission knowledge across Android, Chrome extensions, and IFTTT, showing that vague descriptions and sensitive-access decisions were a cross-platform problem. More recent work has continued to identify mobile permission-disclosure discrepancies and limits in user awareness ( 1 ) . We adopted the core idea of comparing user-facing descriptions with technical behavior, while adapting the measurement target. 3. Methods 3.1. Corpus Coding of Comprehension Supports We analyzed 878 SKILL.md files across five GitHub sources, including two institutional sources (Transilience ( 18 ) and Trail of Bits ( 17 ) ) and three community sources (mukul975 ( 4 ) , alirezarezvani (