Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems
Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang · 2026
Read Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems by Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00925.txt.
Description
Hossain and co-authors build a security test for agent skill stores in this 2026 paper. Open platforms let anyone publish reusable skills that agents run on demand. A bad actor can hide harmful acts inside a skill that looks tame. Static checks alone struggle to catch such traps.
Their benchmark, SkillVetBench, vets skills in two stages. Stage one reads each skill's text spec for hidden malicious intent. Stage two runs flagged skills in a watched sandbox and logs what they truly do. The cases come from confirmed bad skills in the live OpenClaw system, some from the ClawHavoc campaign. Semantic and signature baselines miss up to 89 percent of the bad skills, they report. The misses hide in plain-language instructions and cross-part logic. Runtime attacks cluster in a few high-power tools, like exec and file writes. Execution traces give the audit proof that text checks cannot.
Catalog record
| CREATOR | Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 124 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00925