Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models
Lexiang Xiong, Qi Li, Jingwen Ye, Xinchao Wang · 2026
Read Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models by Lexiang Xiong, Qi Li, Jingwen Ye, Xinchao Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15557.txt.
Introduction
Xiong and co-authors build a diagnostic framework that traces hallucinations inside vision-language models as they unfold. Instead of treating a false caption as a static error, they model generation as a path through a low-dimensional cognitive state space. Probes project that path into a readable form. Their key finding links geometric oddness of the path to high information surprisal. Detection then becomes anomaly spotting in that space. Tests on POPE, MME, and MS-COCO reach state-of-the-art results, even with weak or dirty calibration data.
Catalog record
| CREATOR | Lexiang Xiong, Qi Li, Jingwen Ye, Xinchao Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43614 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models Lexiang Xiong, Qi Li, Jingwen Ye, Xinchao Wang Abstract Vision-Language Models (VLMs) frequently ‘hallucinate’—generate plausible yet factually incorrect statements—posing a critical barrier to their trustworthy deployment. In this work, we propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model’s computational cognition. Our framework is grounded in a normative principle of computational rationality, allowing us to model a VLM’s generation as a dynamic cognitive trajectory . We design a suite of information-theoretic probes that project this trajectory onto an interpretable, low-dimensional Cognitive State Space . Our central discovery is a governing principle we term the geometric-information duality : a cognitive trajectory’s geometric abnormality within this space is fundamentally equivalent to its high information-theoretic surprisal. Hallucination detection is thus elegantly re-framed as a geometric anomaly detection problem. Evaluated across diverse settings—from rigorous binary QA (POPE) and comprehensive reasoning (MME) to unconstrained open-ended captioning (MS-COCO)—our framework achieves state-of-the-art performance. Crucially, it operates with high efficiency under weak supervision and remains highly robust even when calibration data is heavily contaminated. This approach enables a causal attribution of failures, mapping observable errors to distinct pathological states: perceptual instability (measured by Perceptual Entropy, H Evi H_{\text{Evi}} ), logical-causal failure (measured by Inferential Conflict, S Conf S_{\text{Conf}} ), and decisional ambiguity (measured by Decision Entropy, H Ans H_{\text{Ans}} ). Ultimately, this opens a path toward building AI systems whose reasoning is transparent, auditable, and diagnosable by design. 1 Introduction Figure 1: An example of computational cognitive dissonance in Idefics2, where a cascade of failures leads to a coincidentally correct answer. (1) Perceptual Failure: The model hallucinates a ‘motorcycle’ in the evidence chain, an object not present in the image (a cyclist is visible). Our framework captures this as high Perceptual Instability (see panel (c)). (2) Logical Failure: The model then contradicts its own faulty evidence, concluding the final answer is ‘No’. This breakdown of self-consistency is diagnosed as extremely high Inferential Conflict (see panel (d)). This case study demonstrates the limitation of accuracy-only evaluations and highlights our framework’s ability to perform a stage-by-stage differential diagnosis of a VLM’s cognitive process, identifying complex, multi-stage failure trajectories. Consider a striking paradox in Vision-Language Models (VLMs) 52 ; 27 ; 43 ; 26 ; 28 : when asked “Is there a motorcycle in the image?”, a model might confidently hallucinate evidence—“In the image, there is a motorcycle parked”—yet inexplicably conclude, “Therefore, the final answer is No.” We term this cascade of errors computational cognitive dissonance , as illustrated in Figure 1 . What makes this specific case so dramatic is that a double failure (perceiving a non-existent object, then logically contradicting that very perception) leads to a coincidentally correct final answer. This phenomenon exposes a critical insight that severely limits VLM deployment in high-stakes domains 20 ; 25 ; 2 ; 24 ; 47 : hallucinations are rarely monolithic errors that can be diagnosed by a single metric like “accuracy” or “self-consistency.” Instead, they are often complex, multi-stage pathologies where distinct failures—such as perceptual drift and logical bypass—compound and interact within a single cognitive trajectory. Current approaches to hallucination detection generally treat the generation process as an indivisible, monolithic event. They either evaluate the semantic consistency of final outputs via multiple sampling 31 ; 11 or probe for a binary ‘truthfulness’ representation within internal states 1 ; 5 . While foundational, these reductionist views conflate fundamentally different failure modes. They struggle to distinguish whether a hallucination stems from an initial failure to ground concepts in the image ( perceptual drift ) or from an illogical jump that bypasses extracted facts ( inferential bypass ). Our central thesis is that hallucination is a process-level failure that must be diagnosed within a structured model of cognition. To address this, we introduce a normative principle of computational rationality 15 ; 34 for VLMs, formalized as a Markovian information flow: Image ( \mathcal{I} ) \xrightarrow{\text{Perception}} Textual Evidence ( \mathcal{T}_{\text{evi}} ) \xrightarrow{\text{Inference}} Final Answer ( \mathcal{A} ). This principle asserts that for a rational agent, the final answer \mathcal{A} is conditionally independent of the image \mathcal{I} given the evidence \mathcal{T}_{\text{evi}} , implying the conditional mutual information I(\mathcal{A};\mathcal{I}|\mathcal{T}_{\text{evi}}) must be zero. Critics might argue that requiring an explicit evidence chain limits the applicability of such a framework. However, we employ Chain-of-Thought (CoT) not as a strict operational constraint, but as a crucial diagnostic probe in explainable AI (XAI) —akin to a medical contrast agent. By forcing the model to externalize its latent reasoning, we make the implicit cognitive trajectory observable and mathematically diagnosable. To diagnose this cognitive process, we design a suite of probes. While S_{\text{Conf}} directly measures violations of our core principle, Perceptual Entropy ( H_{\text{Evi}} ) and Decision Entropy ( H_{\text{Ans}} ) quantify the stability of the process’s initial and final stages, providing a complete diagnostic picture. These probes act as natural coordinates to project the high-dimensional trajectory onto an interpretable 3D Cognitive State Space : • Perceptual Instability ( H_{\text{Evi}} ): Measured via Perceptual Entropy, this probes the uncertainty at the perception stage ( \mathcal{I}\rightarrow\mathcal{T}_{\text{evi}} ). • Logical-Causal Failure ( S_{\text{Conf}} ): Measured via Inferential Conflict, this directly quantifies the information leakage that violates our core principle. • Decisional Ambiguity ( H_{\text{Ans}} ): Measured via Decision Entropy, this probes the final uncertainty at the trajectory’s terminal stage. Collectively, these probes summarize each cognitive trajectory as a Cognitive State Vector within this space. This perspective reveals a powerful geometric-information duality : a trajectory’s geometric abnormality within this space is fundamentally an expression of its high information-theoretic surprisal. Our experimental results (see Figure 3 ) provide strong empirical evidence for this duality. Normative cognitive trajectories consistently evolve towards stable, low-energy basins of attraction, forming a dense submanifold. Hallucinations, conversely, are high-energy deviations that are perturbed off this manifold, appearing as geometric anomalies. This duality serves as the theoretical bridge that translates the semantic problem of hallucination into a rigorous geometric anomaly detection task 41 . This novel reframing achieves state-of-the-art detection performance while offering significant practical advantages. Unlike multi-sample methods, our approach requires only a single generation pass plus a highly efficient non-autoregressive replay (detailed in Section 3 ). Furthermore, it operates under weak supervision (requiring only ground-truth answers, not fine-grained hallucination labels) and remains highly resilient even when calibration data is contaminated with up to 30% noise. To demonstrate its versatility, we rigorously validate our framework across a spectrum of tasks: from the controlled adversarial subset of POPE 28 , to the comprehensive reasoning categories of MME 13 , and finally to unconstrained open-ended captioning on MS-COCO 29 . In summary, our primary contribution is not merely a new state-of-the-art hallucination detector, but a principled diagnostic framework that provides a new lens through which to understand VLM failures. Our contributions are: • We propose a diagnostic framework that reframes hallucination from a static flaw to a dynamic analysis of a VLM’s cognitive trajectory, grounded in a normative principle of computational rationality. • We design a suite of information-theoretic probes that act as natural coordinates to project the generative process onto an interpretable Cognitive State Space , enabling a stage-by-stage differential