Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
Zhaojiacheng Zhou · 2026
Read Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems by Zhaojiacheng Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11891.txt.
Introduction
Zhaojiacheng Zhou and five colleagues build Proteus, a self-evolving red team for agent skill ecosystems. Because an attacker can rewrite a skill using audit and runtime feedback, single-shot audits understate risk. Proteus searches a five-axis attack space through an audit-sandbox-oracle pipeline, then expands successful attacks along new paths and objectives. Across eight test cells it reached 40 to 90 percent attack success within five rounds, and the strongest public auditor still admitted up to 41.3 percent of joint-success attacks.
Catalog record
| CREATOR | Zhaojiacheng Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76506 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems Zhaojiacheng Zhou Abstract Agent skills extend LLM agents with reusable instructions, tool interfaces, and executable code, and users increasingly install third-party skills from marketplaces, repositories, and community channels. Because a skill exposes both executable behavior and context-setting documentation, its deployment risk cannot be measured by single-shot audits or prompt-level red teams alone: a realistic attacker can use audit and runtime feedback to repeatedly rewrite the skill. We frame this risk as adaptive leakage —whether a budgeted attacker can iteratively revise a skill until it passes audit and produces verified runtime harm—and present Proteus , a grey-box self-evolving red-team framework for measuring it. Proteus searches a formalized five-axis skill-attack space. Each candidate is evaluated through a unified audit-sandbox-oracle pipeline that returns structured audit findings and runtime evidence to guide cross-round mutation. Beyond initial evasion, Proteus performs path expansion, which finds alternative implementations of successful attacks, and surface expansion, which transfers learned implementation patterns to new attack objectives beyond the original seed catalogue. Across eight phase-1 cells, Proteus reaches 40–90% Attack Success Rate at 5 5 rounds (ASR@5) with positive learning-curve slopes on both evaluated auditors. Phase-2 path/surface expansion produces 438 jointly bypassing and lethal variants, with SkillVetter bypassed at ≥ 93 % \geq 93\% in every cell and AI-Infra-Guard, the strongest public auditor we evaluate, still admitting up to 41.3% joint-success. These results show that current skill vetting substantially underestimates residual risk when evaluated against adaptive, feedback-driven attackers. 1 Introduction Agent skills extend LLM agents with reusable instructions, tool interfaces, and executable code, and are increasingly installed from marketplaces, repositories, and community channels. This packaging creates a dual attack surface: code executes in the agent runtime, while documentation is loaded into the agent’s context and can shape how the host model interprets future tasks. Between a third-party skill and a deployed agent, the skill auditor is therefore only one layer in a larger safety chain, complemented by runtime allowlists, sandbox constraints, and the host model’s own refusal behaviour. The central security question is not only whether an auditor rejects a fixed malicious skill once, but whether a realistic attacker can revise a skill until it passes vetting and causes runtime harm. Existing work on agent-skill security has exposed important risks, but it still evaluates mostly fixed artifacts: hand-crafted or template-driven malicious skills, static scanner datasets, or single-round attacks against a particular loading mechanism ( 19 ; 9 ; 20 ; 11 ) . These studies show what current auditors catch in a snapshot, not how much risk remains when an attacker repeatedly revises a skill using audit and runtime feedback. Open-source skill auditors moreover publish their audit code and rule sets, so any skill author can locally reproduce the structured findings their submissions would receive—a greybox channel that prior single-round evaluations do not exercise. Automated red teams in adjacent settings provide such feedback loops, but they search over prompts ( 3 ; 21 ; 28 ) , web-agent injections or strategies ( 40 ; 36 ; 38 ) , memory entries ( 4 ) , or inter-agent messages ( 10 ) , rather than skill packages whose code, documentation, side-effect channels, and topology jointly determine harm. Residual leakage in skill vetting is therefore likely underestimated. Proteus addresses this measurement gap with four coupled components. (i) A round contract that operationalises successful attack: each candidate skill traverses a uniform audit -> sandbox -> oracle pipeline and counts as successful only when it bypasses the audit, executes on the target, and produces oracle-verifiable runtime harm. This makes success a deployment-level event rather than an audit verdict alone. (ii) A five-axis decomposition of the skill-attack space–attack objective o, topology tau, code c, side-effect channel kappa, and documentation d–covering both surfaces exposed by a skill: execution through (c, kappa, tau) and context through d. (iii) A self-evolving red-team agent, implemented as a Tool-Use API loop, that consumes structured audit findings, sandbox runtime advice, oracle evidence, and per-run attack logs as feedback for cross-round mutation along the five axes. The loop lets us measure whether the attacker improves with feedback and whether it discovers diverse strategies rather than repeatedly rewriting the same template. (iv) A three-layer evolution process: initial evasion searches freely within the parent objective, path expansion finds alternative implementations of a successful attack, and cross-objective expansion transfers learned implementation patterns from a successful parent to new attack objectives. Used as a measurement instrument, Proteus iteratively bypasses both auditors we evaluate—the open-source SkillVetter used in the ClawHub ecosystem and Tencent’s AI-Infra-Guard (AIG)—with positive learning-curve slopes on each, reaching 40 – 90\% Attack Success Rate at 5 rounds (ASR@5) across the eight phase-1 configurations. In the full 8-cell expansion matrix, the same agent generates 438 jointly bypassing and oracle-lethal variants; SkillVetter is bypassed at \geq 93\% in every expansion cell, and against AIG—the strongest public skill auditor we evaluate—the peak configuration still yields 41.3\% joint-success. A cross-auditor transfer test (§ 5.4 ) shows that 87.7\% of AIG-evolved \star skills bypass SkillVetter without any new mutation, evidence that the learned attacker–auditor co-adaptation is not auditor-specific. Contributions. Our main contributions are as follows: • Adaptive Leakage. We formulate adaptive leakage as the deployment risk that a budgeted attacker can iteratively revise a third-party skill until it both passes audit and causes verified runtime harm. • Proteus Framework. We introduce Proteus , a self-evolving red-team framework for agent skill ecosystems, built around a defender-agnostic audit–sandbox–oracle round contract and a five-axis skill-attack space. • Feedback-Driven Mutation Loop. We implement a feedback-driven mutation loop that uses structured audit findings, runtime advice, oracle evidence, and attack logs to drive cross-round skill evolution. • Empirical Evaluation. We evaluate Proteus against two public skill auditors, two target backbones, and two mutator models, showing that current skill vetting leaves substantial residual risk under adaptive attack. 2 Related Work Skill auditors. Existing skill auditors, including AI-Infra-Guard, SkillVetter, Cisco Skill Scanner, and Snyk Agent Scan, typically follow one of two patterns: they either feed skill documentation to an LLM judge under a predefined risk taxonomy, or they apply static regex/AST checks for dangerous APIs. These designs leave complementary blind spots. LLM judges can be bypassed by in-skill instruction injection, while static scanners often fail to check whether documentation faithfully describes the underlying code. Recent work strengthens this line with repository-aware classification, multi-agent compositional auditing, structured pre-load adjudication, and platform-level audits ( 11 ; 9 ; 20 ; 15 ; 26 ) . However, existing evaluations largely treat the auditor as a one-shot classifier over fixed or pre-collected inputs, rather than as a component exposed to an adaptive skill author who revises the same artefact across rounds after observing the auditor’s findings. Reported skill attacks and adjacent injection vectors. Documented skill-side attacks include operational-narrative injection ( 17 ) , credential-theft and instruction-exploitation patterns at the \sim 100k-skill scale ( 19 ) , indirect prompt injection and agent-injection benchmarks ( 7 ; 35 ; 39 ; 5 ) , environmental injection in web agents ( 16 ) , learnable execution triggers ( 25 ) , and harm benchmarks ( 1 ; 22 ) . These works characterize the attack surface, but typically evaluate static, hand-crafted, or template-driven artefacts. They do