On the Feasibility of Hijacking MLLMs’ Decision Chain via One Perturbation
Changyue Li, Jiaying Li, Youliang Yuan, Jiaming He, Zhicong Huang, Pinjia He · 2025
Read On the Feasibility of Hijacking MLLMs’ Decision Chain via One Perturbation by Changyue Li, Jiaying Li, Youliang Yuan, Jiaming He, Zhicong Huang, Pinjia He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20002.txt.
Description
Changyue Li, Jiaying Li, Youliang Yuan, Jiaming He, Zhicong Huang, and Pinjia He study a new attack on multimodal language models. Such models are being placed in stateless systems like driving and robotics, where each decision stands alone. The authors ask whether one single crafted perturbation could hijack many separate decisions at once.
Their Semantic-Aware Universal Perturbation acts like a router: it senses the input's meaning and steers it toward attacker-chosen targets. Analysis of the latent space guides their SORT optimization method, and a new dataset with fine-grained semantics supports the tests. Across three representative models, the attack reaches a strong success rate over five targets from a single frame, showing the threat is real in principle.
Catalog record
| CREATOR | Changyue Li, Jiaying Li, Youliang Yuan, Jiaming He, Zhicong Huang, Pinjia He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20002