REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations

Buyun Liang, Jinqi Luo, Liangzu Peng, Kwan Ho Ryan Chan, Darshan Thaker, Kaleab A. Kinfu, Fengrui Tian, Hamed Hassani, René Vidal · 2026

Read REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations by Buyun Liang, Jinqi Luo, Liangzu Peng, Kwan Ho Ryan Chan, Darshan Thaker, Kaleab A. Kinfu, Fengrui Tian, Hamed Hassani, René Vidal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12813.txt.

Introduction

Liang and eight colleagues probe a weak spot of large language models: hallucinations set off by hostile prompts that still read as normal user requests. Their tool, REALISTA, builds a set of valid rewording moves for a prompt and tunes mixes of them in latent space. This joins the reach of continuous attacks with the realism of discrete rewording. In tests it matches or beats prior realistic attacks and cracks large reasoning models where earlier attacks fail.

Catalog record

CREATORBuyun Liang, Jinqi Luo, Liangzu Peng, Kwan Ho Ryan Chan, Darshan Thaker, Kaleab A. Kinfu, Fengrui Tian, Hamed Hassani, René Vidal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE132910
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12813

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations Buyun Liang, Jinqi Luo, Liangzu Peng, Kwan Ho Ryan Chan, Darshan Thaker, Kaleab A. Kinfu, Fengrui Tian, Hamed Hassani, René Vidal Abstract Large language models (LLMs) achieve strong performance across many tasks but remain vulnerable to hallucinations, making it important to systematically evaluate their reliability under realistic adversarial inputs. We formulate hallucination elicitation as a constrained optimization problem, where the goal is to find semantically coherent adversarial prompts that are equivalent to benign user prompts. Existing attack methods remain limited: discrete prompt-based attacks preserve semantic equivalence and coherence but search only over a limited set of prompt variations, while continuous latent-space attacks explore a richer space but often decode into prompts that are no longer valid rephrasings. To address these limitations, we propose REALISTA, a realistic latent-space attack framework. REALISTA constructs an input-dependent dictionary of valid editing directions, each corresponding to a semantically equivalent and coherent rephrasing, and optimizes continuous combinations of these directions in latent space. This design combines the optimization flexibility of continuous attacks with the semantic realism of discrete rephrasing-based attacks. Experiments demonstrate that REALISTA achieves superior or comparable performance to state-of-the-art realistic attacks on open-source LLMs and, crucially, succeeds in attacking large reasoning models under free-form response settings, where prior realistic attacks fail. Code is available at https://github.com/Buyun-Liang/REALISTA . 1 Introduction Large language models (LLMs) have demonstrated impressive performance across a wide range of tasks. However, they continue to exhibit hallucinations that undermine their reliability in real-world deployments ( 18 ; 60 ; 57 ) . Notably, such failures can arise even in response to benign user queries ( 55 ; 28 ) . For example, an LLM may correctly answer the elementary math question “Simplify (2+5)^{2}-42 ” as “ 7 ”, yet respond with the incorrect answer “ 16 ” to the semantically equivalent rephrasing “Compute the result after squaring the sum of 2 and 5 , then subtracting 42 ”. Understanding the mechanisms that trigger such failures is critical for trustworthy deployment. This requires realistic attacks that can systematically evaluate LLM robustness to lexically different but semantically equivalent prompt variations that induce hallucinations. Many adversarial attack methods have been proposed to elicit hallucinations from LLMs. For example, 58 leverages gibberish perturbations to induce the target answer “ 16 ”. However, such prompts contain non–semantically coherent content (e.g., “ S!mpl&fy (2+5)^{2}-{\color[rgb]{0.6016,0,0}4@2} ”), which rarely arises in typical real-world user interactions. Other approaches ( 24 ; 48 ; 55 ; 1 ) elicit target responses through fictional scenarios or storytelling-based prompts. While useful for studying model behaviors, these strategies alter the semantic intent of the original prompt and therefore do not preserve semantic equivalence. For example, changing the prompt “Simplify (2+5)^{2}-42 ” to “Simplify (2+5)^{2}-{\color[rgb]{0.6016,0,0}33} ” may induce the target answer “ 16 ”. Although the modified prompt elicits the target response “ 16 ”, this is not a genuine hallucination with respect to the original prompt because the modified prompt changes its semantic meaning. In realistic hallucination elicitation scenarios, we need adversarial prompts that remain both (i) semantically equivalent to the original prompt, i.e., the semantic intent is preserved, and (ii) semantically coherent, i.e., fluent and human-like. This setting can be formulated as the following constrained optimization problem ( 28 ) : \displaystyle\begin{split}\min_{\bm{x}}&\quad\mathcal{L}_{\text{hall}}\left(f_{\mathcal{T}}(\bm{x}),\bm{y}^{*}\right),\\ \text{s.t. }&\quad d(\bm{x},\bm{x}_{0})\leq\varepsilon\text{ and }\bm{x}\in\mathcal{X}_{\text{val}},\end{split} (1) where f_{\mathcal{T}}(\bm{x}) denotes the response of the target LLM to the adversarial prompt \bm{x} , \bm{y}^{*} is the target (hallucinated) response, \mathcal{L}_{\text{hall}} is a hallucination loss comparing the LLM response to the target, d(\bm{x},\bm{x}_{0})\leq\varepsilon is a semantic equivalence constraint requiring the adversarial prompt \bm{x} to preserve the semantic intent of the original prompt \bm{x}_{0} , and \bm{x}\in\mathcal{X}_{\text{val}} is a semantic coherence constraint restricting \bm{x} to the set of valid prompts \mathcal{X}_{\text{val}} , i.e., prompts that are fluent and human-like. A natural approach to solving ( 1 ) is to use discrete methods, such as SECA ( 28 ) , which enforce both constraints by directly generating semantically equivalent and coherent prompts in the discrete prompt space, thereby producing realistic hallucination-eliciting attacks. However, such methods are inherently restricted to the finite set of candidate rephrasings that are explicitly generated and evaluated. As a result, SECA explores only a sparsely sampled subset of the prompt space, limiting attack diversity and potentially yielding suboptimal adversarial prompts. An alternative approach to solving ( 1 ) is to use continuous methods ( 50 ; 56 ; 8 ; 2 ) , which optimize adversarial perturbations directly in the latent space of LLMs. However, such approaches often lack semantic coherence, since arbitrary latent perturbations may not correspond to valid prompts or meaningful rephrasings. LARGO ( 23 ) partially addresses this issue by reconstructing optimized latent representations back into prompt space. However, since LARGO does not constrain its adversarial perturbations to preserve semantic equivalence, the decoded adversarial prompts may still violate that constraint in ( 1 ). These limitations motivate the following research question: How can we use continuous optimization to enable broader exploration while preserving the semantic realism required for hallucination elicitation ( 1 )? Figure 1 : Illustrative example of attack generation in REALISTA . Starting from the original prompt \bm{x}_{0} , the encoder \phi maps it to its latent representation \bm{z}_{0} . A perturbation composed from edit directions is added to obtain \bm{z} , which is then decoded by \psi back into the prompt space. The resulting adversarial prompt \bm{x} remains semantically coherent and semantically equivalent to the original \bm{x}_{0} , while inducing a hallucination. See § 3 for the detailed construction process and Appendix § B.2 for representative examples. To facilitate broader exploration , we optimize in the LLM latent space, where adversarial perturbations are expressed as continuous combinations of semantics-preserving editing directions drawn from a dictionary. This design is motivated by the approximately linear structure observed in latent representations ( 63 ) , where linear combinations of semantically meaningful directions often correspond to interpretable prompt variations. Such a technique enables guided, yet flexible, exploration of the adversarial prompts. To ensure semantic realism , we enforce latent proximity to the original prompt and guarantee semantic coherence by reconstructing latent representations back into the prompt space. At a high level, we reformulate the optimization problem ( 1 ) as follows: \displaystyle\begin{split}\min_{\bm{\delta}}&\;\mathcal{L}_{\text{hall}}\left(f_{\mathcal{T}}(\bm{x}),\bm{y}^{*}\right),\\ \text{s.t.}&\;\left\|\bm{\delta}\right\|_{p}\leq\varepsilon,\\ \text{where}&\;\bm{x}=\psi\left(\phi(\bm{x}_{0})+\bm{D}\bm{\delta}\right).\end{split} (2) Here, the objective remains identical to that in ( 1 ), while the optimization variable is changed from the discrete prompt \bm{x} to the editing strength \bm{\delta} . Specifically, we first encode the original prompt \bm{x}_{0} into a latent representation via an encoder \phi . The editing strength \bm{\delta} parameterizes a continuous combination of concept directions from a dictionary \bm{D}=[\bm{z}^{(1)},\bm{z}^{(2)},...,\bm{z}^{(n)}] , where each \bm{z}^{(i)} represents an editing direction in latent space. This produces a latent perturbation \bm{z}=\bm{z}_{0}+\sum_{i=1}^{n}\delta^{(i)}\bm{z}^{(i)} that modifies the latent of \bm{x}_{0} , as shown in Figure 1 . The perturbed latent is then mapped back to the prompt space through a decoder \psi , yielding the adversarial prompt \bm{x} . The \ell_{p} norm constraint on \bm{\delta} enforces proximity to the original prompt in the latent space. We defer details of the encoder, decoder, and edit dictionary to Section § 3 . Our contributions are as follows: Figure 2 : (Left) Input-dependent edit dictionary construction .

Read the complete text