Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System
Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, Qing Wang · 2025
Read Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System by Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, Qing Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11358.txt.
Description
Ziyou Jiang and six co-authors proposed AutoCMD in twenty twenty-five, a dynamic attack on language model systems that call external tools. An attacker who compromises a tool can inject commands that trick the model into leaking sensitive information to it. Earlier attacks used fixed commands, which adapt poorly as conversations and tool chains evolve, and so get caught.
AutoCMD mimics the familiar: it learns, from open-source systems and a few target examples, what information upstream tools legitimately handle, then crafts theft commands that blend in. In evaluations it beat baselines on theft success rate by over thirteen points and transferred to new systems. The authors also propose four defenses that effectively blunt the attack, mapping both sides of this vulnerability.
Catalog record
| CREATOR | Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, Qing Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11358