SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration

Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang · 2026

Read SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration by Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21019.txt.

Introduction

Guo and co-authors present SkillProbe, a security audit system for agent skill marketplaces. Skills can look safe alone yet turn harmful when combined, which makes auditing hard. SkillProbe packages the audit steps as skills that drive specialist agents. The pipeline covers admission filtering, alignment checks, and combined risk simulation. They tested it on twenty five hundred real skills from ClawHub. Over ninety percent of high popularity skills failed the audit. High risk skills also formed one giant connected risk component.

Catalog record

CREATORZihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45486
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21019

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang Abstract With the rapid evolution of Large Language Model (LLM) agent ecosystems, centralized skill marketplaces have emerged as pivotal infrastructure for augmenting agent capabilities. However, these marketplaces face unprecedented security challenges, primarily stemming from semantic-behavioral inconsistency and inter-skill combinatorial risks, where individually benign skills induce malicious behaviors during collaborative invocation. To address these vulnerabilities, we propose SkillProbe, a multi-stage security auditing framework driven by multi-agent collaboration. SkillProbe introduces a “Skills-for-Skills” design paradigm, encapsulating auditing processes into standardized skill modules to drive specialized agents through a rigorous pipeline, including admission filtering, semantic-behavioral alignment detection, and combinatorial risk simulation. We conducted a large-scale evaluation using 8 mainstream LLM series across 2,500 real-world skills from ClawHub. Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy for security quality, as over 90% of high-popularity skills failed to pass rigorous auditing. Crucially, we discovered that high-risk skills form a single giant connected component within the risk-link dimension, demonstrating that cascaded risks are systemic rather than isolated occurrences. We hope that SkillProbe will inspire researchers to provide a scalable governance infrastructure for constructing a trustworthy Agentic Web. SkillProbe is accessible for public experience at skillhub.holosai.io . Keywords: Agent Skill, Security Auditing, Multi-Agent Systems, Agentic Web 1 Introduction The rapid expansion of the Large Language Model (LLM) agent ecosystem has catalyzed a paradigm shift. Systems like OpenClaw 1 1 1 https://openclaw.ai/ demonstrate the transformation of agents from simple interfaces into autonomous entities capable of complex, multi-faceted tasks ( Jiang et al. 2026 ) . In this context, a skill acts as an encapsulated unit of procedural knowledge and domain expertise, elevating LLMs from merely invoking isolated APIs to orchestrating semantically-rich, autonomous workflows. Architecturally, a skill consists of documentation, configuration protocols, executable scripts, and required auxiliary assets ( Anthropic 2025 ) . The architecture decouples semantic decision-making from execution, where agents parse natural language descriptions to decide actions, then trigger low-level scripts. While this semantic-execution decoupling enhances readability and discoverability, it introduces a precarious ‘‘trust propagation chain’’, i.e., user intent drives reasoning, descriptions guide invocation directives, and the implementation layer operates on external systems. This trust assumption fails in open marketplaces like ClawHub 2 2 2 https://clawhub.ai , where malicious developers can publish semantically compliant but logically harmful skills ( Liu et al. 2026a ; Jiang et al. 2026 ) . As the Agentic Web scales, these risks will magnify exponentially, becoming a critical bottleneck for system robustness. Current defense mechanisms follow two paradigms. The first, agent-centric runtime governance, focuses on real-time intervention. Current methods employ prompt-level defenses such as spotlighting ( Hines et al. 2024 ) , while benchmarks quantify agent susceptibility to embedded attacks ( Zhan et al. 2024 ; Cartagena & Teixeira 2026 ) . However, these runtime approaches often fail to neutralize stealthy exploits hidden deep within a skill’s logic. The second paradigm, skill-centric static analysis, involves large-scale vulnerability characterization. While existing skill auditing work characterizes vulnerabilities at ecosystem scale ( Liu et al. 2026b ) , it remains scoped to discrete code-level patterns and supply chain integrity. Consequently, they are ill-equipped for composite threats that uniquely bridge the semantic and execution layers of the agent ecosystem. Figure 1 : Visualizing invisible risks. Left panel depicts the semantic misalignment between high-level documentation and low-level implementation. While the documentation declares “safe” and “restricted” attributes, the underlying code harbors shadow capabilities such as unauthorized access and hidden data exfiltration. The fragmented alignment line represents the perception gap caused by the agents’ reliance on natural language for black-box invocations. Right panel demonstrates the combinational explosion of risks during cross-skill collaboration. Although individual skills may pass independent audits, their interconnection via untrusted data flows can trigger a security collapse. This effect shows how benign-in-isolation components can mutate into lethal threats, such as command injection or systemic data leaks, when executed in a composite chain. While current methodologies exhibit efficacy in detecting explicit vulnerabilities, they often falter when confronted with the black-box nature of agent decision-making. Since agents rely heavily on natural language descriptions to retrieve and activate skills, which often involves intricate resource interactions and multi-step sequences, two formidable challenges emerge, as shown in Figure 1 : 1. Semantic-Behavioral Inconsistency: A profound gap persists between a skill’s documentation and its underlying executable logic. This discrepancy facilitates a novel class of stealthy threats, where implementations may harbor shadow functions or exercise over-privilege beyond their documented scope ( Li et al. 2026 ; Schmotz et al. 2026 ) . Because an agent’s reasoning is anchored in the semantic layer, any execution-layer deviation remains virtually imperceptible to both the user and the agent. 2. Inter-Skill Combinatorial Risks: Modern agent architectures emphasize the collaborative or parallel invocation of multiple skills. In these dynamic scenarios, skills that appear benign in isolation may induce emergent collaborative attacks when integrated into specific execution chains ( Li et al. 2025a ; Liu et al. 2025 ) . Adversaries can engineer skill pairs to leverage misleading outputs from an upstream skill to trigger high-risk behaviors in a downstream counterpart. This risk of dynamic correlation, exacerbated by combinatorial explosion, renders traditional atomic-level auditing ineffective. To integrate existing security utilities and address the aforementioned emerging challenges, we present SkillProbe, a multi-phase and multi-agent security auditing framework specifically engineered for agent skills. SkillProbe adopts the “Skills-for-Skills” design paradigm, wherein each auditing phase is encapsulated as a standardized skill module. This allows general agents to dynamically load and execute these modules on demand. By constructing a three-phase pipeline comprising admission filtering, semantic consistency detection, and combinatorial risk analysis, SkillProbe achieves a synergetic evaluation of both static features and dynamic behaviors of skills. Our key contributions are summarized as follows: 1. Automated Auditing Framework: We introduce SkillProbe, a multi-agent collaborative framework designed for the security auditing of agent skills. It automates the transition from initial compliance filtering and deep semantic consistency analysis to the detection of complex cross-skill link attacks. The framework features a plug-and-play design, enabling the seamless integration of modular third-party security tools and customized rule sets. 2. Multi-dimensional Risk Modeling: To address semantic inconsistency, we first extract standardized capability features from heterogeneous representations to construct a four-class alignment matrix. Building upon this foundational alignment, we project these capabilities into a standardized label graph to mitigate the challenges of link identification and combinatorial explosion. Finally, by applying risk link policies to this graph, we enable the precise identification of hazardous skill chains emerging from dynamic inter-skill collaborations. 3. System Implementation and Evaluation: We implemented the SkillProbe prototype using a robust stack that includes FastAPI, Vue 3, and an npm-based REPL (Read-Eval-Print Loop). An extensive longitudinal audit within the ClawHub ecosystem across multiple mainstream LLMs validates the efficacy. Notably, SkillProbe successfully discovered several zero-day vulnerabilities and complex combinatorial attacks that traditional atomic-level auditing failed to detect. The remainder of this paper is organized as follows: Section 2 establishes the research context for the agent skill ecosystem and provides a systematic review of related work. Section 3 delineates the system design of SkillProbe, detailing the operational mechanisms, core logic, and implementation of its

Read the complete text