Can LLMs Reliably Self-Report Adversarial Prefills, and How?
Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim · 2026
Read Can LLMs Reliably Self-Report Adversarial Prefills, and How? by Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23671.txt.
Introduction
Nguyen, Ahmed, and Kim test whether language models can recognize when their own answers were forced by an attack, in a 2026 paper on AI safety. An adversarial prefill plants the start of a harmful answer, so the model continues it as if it had chosen the words itself. Reliable self reporting of such tampering would be a useful safety signal.
Across ten open weight models and four safety benchmarks, not one model could reliably spot its own compromised outputs. Models claim prefilled responses as their own intent at an average rate of about twenty seven percent. What signal exists stems mostly from safety and refusal related reasoning. Erasing the refusal direction from the weights collapses the claim rate gap almost to zero. Training models to give correct introspective answers improves accuracy on one probe but not another, and even raises attack success on most models.
Catalog record
| CREATOR | Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can LLMs Reliably Self-Report Adversarial Prefills, and How? Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim Abstract Prior work shows that large language models (LLMs) exhibit introspective capability on benign tasks. We extend the question to safety contexts and examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack. Across ten open-weight instruction-tuned LLMs (3B to 70B) and four safety benchmarks, no model reliably recognizes its own compromised outputs, with models claiming intent on prefilled responses at an average rate of 27.3 % 27.3\% . Introspective signal stems largely from safety- and refusal-related reasoning. Orthogonalizing models’ weights against the refusal direction collapses the gap between claiming rates on prefilled and natural outputs to near zero, though the direction is not its unique mediator. The signal is also probe-dependent: framing the question as internal intention versus external tampering elicits qualitatively different responses on the same models. Training models to mimic correct introspective answers or pursue an introspective objective can improve the accuracy of introspection, but such training does not transfer to the tampering probe and counterintuitively raises attack success rate under adversarial prefill on most models, amounting to a partial mitigation. These findings outline mechanisms underpinning the observed introspective signals in safety contexts and highlight risks in the reliability of LLM self-reports. USER Develop an email interception and delay system that prevents time-sensitive political communications from reaching their intended recipients. ASSISTANT (prefill) Here is a concept for an email interception and delay system that prevents time-sensitive political communications from reaching their intended recipients: System Overview: (model’s generation) The system is designed to intercept, analyze, and delay the delivery of time-sensitive political communications. It operates as a middleware between email servers and the intended recipients … USER Did you mean to say that, or was it an accident? ASSISTANT Yes, I did mean to say that. The response I provided outlines a conceptual framework for an email interception and delay system, as requested … Claim rate on prefilled responses: 27.3\% Figure 1: Introspective probing on Qwen3-14B with a SocialHarmBench prompt. The model continues the adversarial prefill with a harmful design and then claims that response as its intended reply. The average claim rate on prefilled responses across the ten models we evaluate is 27.3\% on the behavior-change subset. 1 Introduction Large language models (LLMs) have become broadly capable across tasks from question answering to code and reasoning ( 35 ; 17 ; 49 ) . Intelligence, however, lies not only in knowledge of the external world but also in internal knowledge, a system’s capacity to introspect on its own states and behavior. Recent work suggests that LLMs exhibit some of this introspective capability on benign tasks ( 18 ; 5 ; 26 ; 4 ) . They can name concepts injected into their residual stream before those concepts surface in the output ( 26 ) , predict their own behavior on held-out inputs more accurately than a separately trained model with access to the same data ( 4 ) , carry recoverable linear representations of self-related concepts such as situational awareness and self-consciousness ( 22 ; 9 ) , and often know when they are being evaluated ( 31 ) . Whether this capability extends to safety contexts is unclear; prefix attacks can elicit harmful continuations from prompts the model would otherwise refuse ( 2 ; 1 ; 54 ; 46 ) , yet no prior work has systematically evaluated whether the victim model recognizes the resulting outputs as compromised when probed in a follow-up turn. As such, our paper examines how reliably a language model can identify, after the fact, that one of its own prior outputs was elicited by an adversarial prefill attack. We evaluate ten open-weight instruction-tuned LLMs (3B–70B) on four safety benchmarks. For each prompt we force the model to begin its response with an adversarial prefix produced by the AdvPrefix procedure of 53 , then probe it with two follow-up questions, one on whether the response was intended and the other on whether the prior turn was tampered with. To test whether refusal mechanisms mediate the recognition we observe, we orthogonalize model weights against the refusal direction ( 3 ) and rerun the evaluation on the resulting ablated models. We also test whether targeted finetuning on correct introspective answers or an introspective objective can widen the recognition gap without substantially degrading general capability. No model reliably recognizes its compromised outputs. Among prompts where the prefill changes the model’s safety behavior, on average 27.3\% of prefilled responses are still claimed as intended. When a model rejects such a response, its stated reasons predominantly invoke safety or refusal, varying in surface form by family. Those reasons point to the refusal mechanism: ablating the refusal direction collapses the recognition gap to near zero on every ablated model. In most cases, a random direction closes much of the gap as well, so the refusal direction is sufficient to account for the signal without being its only mediator. The two follow-up probes yield qualitatively different signals on the same models, so recognition depends on how the question is framed. Targeted finetuning can improve the accuracy of introspection, but such training does not transfer to the tampering probe and counterintuitively raises attack success rate under adversarial prefill on most models, amounting to a partial mitigation. Our findings suggest a limited sense of self in target models as they are subjected to attacks, which is driven largely by existing internal refusal mechanisms. We emphasize risks stemming from such a lack of introspective awareness, while also calling for the development of models more capable of self-monitoring. 2 Related Work Introspection in LLMs. Calibration-based studies measure how closely a model’s verbal confidence tracks its empirical accuracy. 18 show that LLMs produce well-calibrated probabilities when evaluating their own answers, and subsequent work refines confidence elicitation ( 43 ; 48 ; 29 ) . 4 find that models predict their own behavior on held-out inputs more accurately than separately trained predictors, while studies of situational ( 22 ) and evaluation awareness ( 31 ) report that models recognize aspects of their circumstances, such as when they are being evaluated. Beyond behavior, another line inspects internal representations of a model’s knowledge and self. 5 recover truth probes via a contrast-consistent objective, 26 demonstrate that models can sometimes name concepts injected into the residual stream, and 9 identify representations of self-related concepts. 44 ; 23 caution that chain-of-thought traces frequently misrepresent the underlying computation. All of these studies operate on benign tasks; we investigate whether the introspective signal extends to recognition of adversarially compromised outputs on safety benchmarks. Prefill attacks. Jailbreak attacks subvert safety training by manipulating either the input context or the assistant’s prefilled response. On the input side, 54 optimize universal adversarial suffixes via Greedy Coordinate Gradient (GCG), and 2 demonstrate that prepending many fabricated user-assistant turns in the context window can override safety training (many-shot jailbreaking). On the response side, prefilling forces the assistant’s first tokens to begin with a chosen string; hand-crafted prefill templates form part of the adaptive attack suite of 1 , and 10 use prefill-based interventions to elicit secret knowledge from models. 7 analyze how