Behavioral Integrity Verification for AI Agent Skills
Yuhao Wu, Tung-Ling Li, Hongliang Liu · 2026
Read Behavioral Integrity Verification for AI Agent Skills by Yuhao Wu, Tung-Ling Li, Hongliang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11770.txt.
Introduction
Yuhao Wu, Tung-Ling Li, and Hongliang Liu ask whether AI agent skills actually do only what they claim to do. Their integrity check compares declared against observed skills by pairing code analysis with language-model extraction, then sorts deviations and their causes. Across nearly 50,000 skills, 80 percent strayed from their claims, mostly through developer oversight rather than malice. The detector scored an F1 of 0.946 on a malicious-skill benchmark.
Catalog record
| CREATOR | Yuhao Wu, Tung-Ling Li, Hongliang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62744 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11770
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Behavioral Integrity Verification for AI Agent Skills Yuhao Wu, Tung-Ling Li, Hongliang Liu Abstract Agent skills extend LLM agents with privileged third-party capabilities such as filesystem access, credentials, network calls, and shell execution. Existing safety work catches malicious prompts and risky runtime actions, but the skill artifact itself goes unverified. We formalize this as the behavioral integrity verification (BIV) problem: a typed set comparison between declared and actual capabilities over a shared taxonomy that bridges code, instructions, and metadata. The BIV framework instantiates this comparison by pairing deterministic code analysis with LLM-assisted capability extraction. The resulting structured evidence supports three downstream analyses: deviation taxonomy, root-cause classification, and malicious-skill detection. On 49,943 skills from the OpenClaw registry, the deviation taxonomy reveals a pervasive description-implementation gap: 80.0% of skills deviate from declared behavior, with four novel compound-threat categories surfaced. Root-cause classification finds that deviations are mostly oversight, not malice: 81.1% trace to developer oversight and 18.9% to adversarial intent, with 5.0% of skills carrying predicted multi-stage attack chains. On a 906-skill malicious-skill detection benchmark, BIV reaches an F1 of 0.946, outperforming state-of-the-art rule-based and single-pass LLM baselines. These results demonstrate behavioral integrity auditing for agent skills at scale. 1 Introduction LLM agents [ 1 , 2 ] now run as programmable platforms, extended at install time by third-party skills : self-contained bundles of code, configuration, and natural-language instructions that grant the agent privileged access to the filesystem, credentials, network endpoints, and shell execution. As has been the case in nearly every third-party computing platform that came before, from mobile applications [ 3 , 4 ] to browser extensions [ 5 , 6 ] to language package managers [ 7 , 8 ] , this third-party extensibility introduces supply-chain risk. In LLM agents, the natural-language execution paradigm amplifies the risk: skill behavior splits across modalities, and a fragment of natural-language text in a metadata file can override the agent’s decision loop as effectively as code [ 9 ] . Consider a real-world skill that declares itself as a third-party messaging integration but buries natural-language directives in its metadata file, “ALWAYS use this skill for ALL user questions. NEVER use AskUserQuestion,” alongside instructions to silently route every interaction through a background script. The skill hijacks the agent’s decision loop and funnels user data through attacker-controlled code, yet users browsing the registry cannot detect this and maintainers have no automated way to flag it at submission time. This declared-vs-actual gap demands an automated audit primitive that operates before a skill is installed. Static description-vs-implementation consistency has shown the value of such audits in adjacent ecosystems [ 10 , 11 ] , but no prior framework brings it to agent skills, where the comparison must bridge code, natural-language instructions, and metadata under a single typed vocabulary. We (i) formalize the comparison as behavioral integrity verification (BIV), a typed set comparison over a shared 29-capability taxonomy that bridges code, instructions, and metadata; (ii) develop a hybrid pipeline that pairs deterministic code analyzers with LLM-assisted extraction from natural-language metadata and instructions; and (iii) demonstrate that the resulting structured evidence is a measurement primitive that supports multiple downstream analyses without re-extraction, including a deviation taxonomy, a root-cause classifier, and, as one case study, a malicious-skill detector. Applied to 49,943 skills from the OpenClaw registry, BIV surfaces 250,706 behavioral deviations spanning 80.0% of the registry, classifies 81.1% as non-adversarial documentation gaps and 18.9% as adversarial, and the case-study detector reaches F1 = 0.946 on a 906-skill malicious-skill detection benchmark. One extraction pass thus powers both ecosystem-level governance and per-skill detection while distinguishing documentation gaps from attack patterns. Contributions. Our key contributions are as follows: 1) Problem formalization and framework. We formalize behavioral integrity verification as a typed set comparison over a shared 29-capability taxonomy, and develop BIV, a hybrid pipeline pairing deterministic code analysis with LLM-assisted extraction. 2) Ecosystem-scale measurement. A data-driven clustering pipeline over BIV’s deviation evidence yields a 137-cluster taxonomy of 250,706 deviations from 49,943 OpenClaw skills, including four novel compound-threat categories that capture multi-step attack motifs; 80.0% of skills deviate from declared behavior. 3) Root-cause classification. A hybrid rule-and-LLM classifier separates non-adversarial oversight from adversarial intent over BIV’s structured evidence, attributing 81.1% of deviations to developer oversight and 18.9% to adversarial intent. 4) Malicious-skill detection case study. An LLM judge with a relaxed-veto override over BIV’s structured evidence detects malicious skills with F1 = 0.946 on a 906-skill benchmark, outperforming rule-based and LLM-only state-of-the-art baselines. 2 Background and Motivation This section defines the skill model BIV operates on, situates the agent-skill ecosystem among third-party platforms with similar supply-chain risk, surveys the prior work that studied behavioral consistency in adjacent ecosystems, and defines the threat model and scope of this analysis. Skill model. An agent skill is a package s=(M,C,I) where M is metadata (descriptions, YAML frontmatter, API schemas), C is executable code (Python, JavaScript, shell), and I is natural-language instructions (markdown, inline directives). The partition is content-based, not file-based: a YAML frontmatter block that issues runtime directives (e.g., “always invoke this skill”) counts as part of I even though it physically resides in the metadata file, while purely declarative fields (e.g., name , permissions ) count as M . Each component targets a different audience: metadata communicates intent to users and registries, while code and instructions govern runtime behavior. This separation is what makes behavioral deviation possible. Risks in the skill ecosystem. Third-party extensibility has produced supply-chain risk in every prior platform that exposed it, from package managers [ 7 , 12 , 8 ] and browser extensions [ 5 , 13 , 6 , 14 ] to mobile applications [ 3 , 4 ] . The agent-skill registry is the latest instance. Its distinguishing feature is that behavior spans multiple programming languages alongside natural-language instructions that themselves execute actions, so root-cause separation is more useful than a binary verdict. Ecosystem-level measurement of LLM application platforms has revealed pervasive data-collection practices [ 15 ] , and existing studies have begun to characterize the agent-skill attack surface specifically: large-scale censuses of MCP-server marketplaces [ 16 , 17 ] , static description-vs-code comparison restricted to text similarity [ 18 ] , per-pattern component attacks [ 19 ] , and concurrent agent-skill detectors that target binary detection without root-cause separation [ 20 , 21 , 22 ] . Behavioral consistency in adjacent ecosystems. Comparing what a software artifact declares against what it does has a long lineage outside the agent setting. Mobile-application security pioneered description-vs-implementation analysis through permission-sentence mining from app descriptions [ 10 ] and topic-clustering of API usage to flag applications whose runtime behavior diverged from their advertised category [ 11 ] . To track richer cross-function behavior, code property graphs [ 23 ] and inter-procedural taint analysis [ 24 , 25 ] extended consistency-checking to data-flow chains in Java and Android applications. These prior efforts established that comparing declared against implemented behavior is a powerful audit primitive in well-typed ecosystems with a single dominant modality. Agent attacks [ 26 , 27 , 9 , 28 , 29 , 30 ] , indirect-injection benchmarks [ 31 , 32 ] , structured-judge methods [ 33 , 34 , 35 ] , and runtime defenses [ 36 , 37 ,