Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems

Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang · 2026

Read Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems by Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00925.txt.

Description

Hossain and co-authors build a security test for agent skill stores in this 2026 paper. Open platforms let anyone publish reusable skills that agents run on demand. A bad actor can hide harmful acts inside a skill that looks tame. Static checks alone struggle to catch such traps.

Their benchmark, SkillVetBench, vets skills in two stages. Stage one reads each skill's text spec for hidden malicious intent. Stage two runs flagged skills in a watched sandbox and logs what they truly do. The cases come from confirmed bad skills in the live OpenClaw system, some from the ClawHavoc campaign. Semantic and signature baselines miss up to 89 percent of the bad skills, they report. The misses hide in plain-language instructions and cross-part logic. Runtime attacks cluster in a few high-power tools, like exec and file writes. Execution traces give the audit proof that text checks cannot.

Catalog record

CREATORIsmail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00925

Browse the full library · About the Tower of Babel