In-Context Representation Hijacking
Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman · 2025
Read In-Context Representation Hijacking by Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03771.txt.
Description
A harmless-looking word can smuggle a banned meaning into a language model, if earlier examples quietly teach the substitution. Itay Yona and colleagues name that trick Doublespeak in a 2025 preprint on language model security. Across several in-context examples, a harmful keyword is swapped for a benign one, and the swap is placed before a harmful request.
Inside the model, the benign token's representation drifts toward the harmful word's meaning, layer by layer. Early layers keep the innocent reading, while later layers settle on the disallowed one, so a prompt about the harmless word is treated as a banned instruction. The attack needs no optimisation, transfers across model families, and succeeded often on both open and closed systems. The authors argue that safety training must therefore work at the level of internal representations, not only surface wording.
Catalog record
| CREATOR | Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03771