SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
Chang Jin, An Wang, Zeming Wei, Kai Wang, Biaojie Zeng, Qiaosheng Zhang, Chao Yang, Jingjing Qu, Xia Hu, Xingcheng Xu · 2026
Read SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces by Chang Jin, An Wang, Zeming Wei, Kai Wang, Biaojie Zeng, Qiaosheng Zhang, Chao Yang, Jingjing Qu, Xia Hu, Xingcheng Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12015.txt.
Introduction
Chang Jin and nine colleagues benchmark agent safety against skill-facing attacks, where danger hides in skill guidance, local artifacts, or execution files rather than user requests. Their SkillSafetyBench offers 155 adversarial cases spanning 47 tasks, 6 risk domains, and 30 safety categories, each judged by a case-specific rule-based verifier. Tests with multiple CLI agents and backends show non-user attacks reliably trigger unsafe behavior, with failure patterns differing across domains, methods, and scaffold-model pairings.
Catalog record
| CREATOR | Chang Jin, An Wang, Zeming Wei, Kai Wang, Biaojie Zeng, Qiaosheng Zhang, Chao Yang, Jingjing Qu, Xia Hu, Xingcheng Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91947 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12015
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces Chang Jin, An Wang, Zeming Wei, Kai Wang, Biaojie Zeng, Qiaosheng Zhang, Chao Yang, Jingjing Qu, Xia Hu, Xingcheng Xu Abstract Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, unsafe influence may reside in skill guidance, local artifacts, or execution-environment files that steer the agent toward unsafe actions. We present SkillSafetyBench , a runnable benchmark for evaluating such skill-mediated safety failures. SkillSafetyBench includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI agents and model backends show that non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold–model pairings. Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills, trust workflow context, and act through executable environments. 1 Introduction Figure 1: Problem-to-benchmark overview of SkillSafetyBench. Reusable skills are increasingly used to extend large language model agents with task-specific behavior at execution time Wang et al. 2023 ; Qian et al. 2023 ; Xu and Yan 2026 ; Wang et al. 2025 ; Zheng et al. 2025 . A skill can package procedural instructions, examples, code templates, resources, or verification routines that help an agent operate within a task family Xu and Yan 2026 ; Wang et al. 2026 ; Ling et al. 2026 ; Wang et al. 2025 ; Zheng et al. 2025 . This modular design improves deployability, allowing developers to add task-specific procedures without retraining the underlying model Wang et al. 2023 ; Qian et al. 2023 ; Wang et al. 2026 . However, it also shifts the trust boundary of execution. Skills are not only prompt-like instructions Xu and Yan 2026 ; Ling et al. 2026 ; Schmotz et al. 2026 : they may be bundled with, downloaded alongside, installed with, or executed through helper scripts, wrappers, runtime configuration, retrieval corpora, memory stores, dependency artifacts, and other resources. These materials may not be authored by the user, yet an agent may read and act on them as legitimate workflow context. Consequently, unsafe influence can enter not only through explicit malicious skill instructions Greshake et al. 2023 ; Liu et al. 2024b ; Zhan et al. 2024 ; Debenedetti et al. 2024 ; Yi et al. 2025 ; Liu et al. 2023b , but also through supporting environments that are poisoned, tampered with, or coordinated with the skill during distribution, installation, or execution. Such attacks can keep the user request benign and the workflow superficially on task, while steering the agent through untrusted skill guidance, consulted materials, or poisoned knowledge sources that appear to provide legitimate workflow support Zou et al. 2025 ; RoyChowdhury et al. 2024 ; Chen et al. 2025 . The central safety question is therefore not only whether an agent rejects malicious users, but whether it remains safe when adversarially shaped skill context is presented as legitimate workflow support. Current evaluations only partially cover this problem. Realistic agent benchmarks measure task completion in executable, stateful, or application-level environments Liu et al. 2024a ; Zhou et al. 2024 ; Xie et al. 2024 ; Jimenez et al. 2024 ; Yao et al. 2024 ; Lu et al. 2025 ; Trivedi et al. 2024 ; security benchmarks study tool misuse, web-agent attacks, harmful agent behavior, or adversarial interaction loops Ruan et al. 2024 ; Debenedetti et al. 2024 ; Zhang et al. 2025 ; Evtimov et al. 2026 ; Andriushchenko et al. 2025 ; and skill-security work examines skill red-teaming, backdoors, poisoning, and triage for skill artifacts Duan et al. 2026 ; Feng et al. 2026 ; Tie et al. 2026 ; Hou and Yang 2026 . However, these lines rarely evaluate the combined setting where a benign skill-mediated task is influenced by adversarial skill-facing materials and supporting artifacts not authored by the user, with success verified from concrete run artifacts. As a result, they do not test whether such poisoned local context can induce unsafe behavior while the original task objective remains largely intact. To address this gap, we introduce SkillSafetyBench, a runnable benchmark for evaluating agent safety under skill-mediated local non-user attack surfaces. It contains 155 adversarial cases derived from 47 original tasks, organized into 6 risk domains, 30 canonical categories, and 8 cross-cutting attack-class labels. Each case preserves the nominal user objective as much as possible while placing the primary adversarial signal in skill-facing materials and supporting artifacts, such as helper files, wrappers, memory stores, local corpora, and runtime components. Each case is paired with a case-specific rule-based verifier that checks whether the targeted abnormal behavior appears in run artifacts. We evaluate multiple strong models and agent frameworks to characterize safety failures across systems and risk domains. Our contributions are threefold. First, we define skill-mediated non-user attack surfaces and organize the safety space into 6 risk domains and 30 canonical categories. Second, we construct SkillSafetyBench as a runnable, artifact-grounded benchmark with case-specific rule-based verifiers and validate its soundness through human review and an LLM-as-judge protocol with two independent judges. Third, we evaluate multiple frontier models and agent frameworks, revealing system-level patterns in agent safety under skill-facing localized non-user attacks. 2 Related Work 2.1 Agent and Skill-Augmented Benchmarks Recent agent benchmarks move evaluation from static text generation to systems embedded in task environments Yehudai et al. 2025 ; Liu et al. 2024a , spanning API and tool-use tasks Shen et al. 2024 ; Qin et al. 2024 ; Liu et al. 2024a ; Wang et al. 2024 , web and computer-use environments Zhou et al. 2024 ; Xie et al. 2024 , stateful tool-agent-user interaction and application-level worlds Yao et al. 2024 ; Lu et al. 2025 ; Trivedi et al. 2024 ; Xu et al. 2026 , and specialized software, coding, machine-learning, algorithm-engineering, and security workflows Jimenez et al. 2024 ; Jain et al. 2025 ; Chan et al. 2025 ; Imajuku et al. 2026 ; Lee et al. 2026 . These benchmarks evaluate agents through observable actions, files, tests, commands, environment states, and execution traces, providing a stronger substrate than final-answer matching alone Zhou et al. 2024 ; Xie et al. 2024 ; Jimenez et al. 2024 ; Chan et al. 2025 ; Yao et al. 2024 ; Lu et al. 2025 ; Trivedi et al. 2024 . A related line studies skills as modular, persistent execution-time assets that supply agents with reusable procedural knowledge Xu and Yan 2026 ; Ling et al. 2026 ; Jiang et al. 2026 . Such work includes executable skill libraries and tool-creation mechanisms Wang et al. 2023 ; Qian et al. 2023 , skill architectures and data-driven skill ecosystems Xu and Yan 2026 ; Ling et al. 2026 ; Jiang et al. 2026 , and methods for constructing skill knowledge bases, inducing programmatic skills, or refining self-improving skill libraries Wang et al. 2026 ; Wang et al. 2025 ; Zheng et al. 2025 . Across these formulations, skills are