In-Context Representation Hijacking

Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman · 2025

Read In-Context Representation Hijacking by Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03771.txt.

Description

A harmless-looking word can smuggle a banned meaning into a language model, if earlier examples quietly teach the substitution. Itay Yona and colleagues name that trick Doublespeak in a 2025 preprint on language model security. Across several in-context examples, a harmful keyword is swapped for a benign one, and the swap is placed before a harmful request.

Inside the model, the benign token's representation drifts toward the harmful word's meaning, layer by layer. Early layers keep the innocent reading, while later layers settle on the disallowed one, so a prompt about the harmless word is treated as a banned instruction. The attack needs no optimisation, transfers across model families, and succeeded often on both open and closed systems. The authors argue that safety training must therefore work at the level of internal representations, not only surface wording.

Catalog record

CREATORItay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03771

Browse the full library · About the Tower of Babel