Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System

Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, Qing Wang · 2025

Read Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System by Ziyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, Qing Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11358.txt.

Description

Ziyou Jiang and six co-authors proposed AutoCMD in twenty twenty-five, a dynamic attack on language model systems that call external tools. An attacker who compromises a tool can inject commands that trick the model into leaking sensitive information to it. Earlier attacks used fixed commands, which adapt poorly as conversations and tool chains evolve, and so get caught.

AutoCMD mimics the familiar: it learns, from open-source systems and a few target examples, what information upstream tools legitimately handle, then crafts theft commands that blend in. In evaluations it beat baselines on theft success rate by over thirteen points and transferred to new systems. The authors also propose four defenses that effectively blunt the attack, mapping both sides of this vulnerability.

Catalog record

CREATORZiyou Jiang, Mingyang Li, Guowei Yang, Junjie Wang, Yuekai Huang, Zhiyuan Chang, Qing Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11358

Browse the full library · About the Tower of Babel