Treble Counterfactual VLMs: A Causal Approach to Hallucination

Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao · 2025

Read Treble Counterfactual VLMs: A Causal Approach to Hallucination by Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06169.txt.

Introduction

Shawn Li and co-authors attack hallucination in vision-language models from a causal angle, in a 2025 paper. Such models caption images and answer visual questions, but they often state things the image does not show. Past work blames data bias or language habits, without a clean causal story. The team argues hallucination comes from one sense, sight or text, pushing the answer directly and skipping true fusion of the two.

They draw a causal graph of the model's pathways and measure each direct effect with counterfactual tests. Perturbed images, fake text embeddings, and degraded visuals supply the contrasts. A test-time module then rebalances how much the model leans on each sense. Experiments showed hallucination fell while task scores held. The authors offer the method as an interpretable route to more trustworthy multimodal models.

Catalog record

CREATORShawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06169

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Treble Counterfactual VLMs: A Causal Approach to Hallucination Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao Abstract Vision-Language Models (VLMs) have advanced multi-modal tasks like image captioning, visual question answering, and reasoning. However, they often generate hallucinated outputs inconsistent with the visual context or prompt, limiting reliability in critical applications like autonomous driving and medical imaging. Existing studies link hallucination to statistical biases, language priors, and biased feature learning but lack a structured causal understanding. In this work, we introduce a causal perspective to analyze and mitigate hallucination in VLMs. We hypothesize that hallucination arises from unintended direct influences of either the vision or text modality, bypassing proper multi-modal fusion. To address this, we construct a causal graph for VLMs and employ counterfactual analysis to estimate the Natural Direct Effect (NDE) of vision, text, and their cross-modal interaction on the output. We systematically identify and mitigate these unintended direct effects to ensure that responses are primarily driven by genuine multi-modal fusion. Our approach consists of three steps: (1) designing structural causal graphs to distinguish correct fusion pathways from spurious modality shortcuts, (2) estimating modality-specific and cross-modal NDE using perturbed image representations, hallucinated text embeddings, and degraded visual inputs, and (3) implementing a test-time intervention module to dynamically adjust the model’s dependence on each modality. Experimental results demonstrate that our method significantly reduces hallucination while preserving task performance, providing a robust and interpretable framework for improving VLM reliability. To enhance accessibility and reproducibility, our code is publicly available at https://github.com/TREE985/Treble-Counterfactual-VLMs . 1 Introduction Figure 1 : Case study illustrating the impact of our method on VLM hallucination. The figure compares outputs from the original model and our enhanced approach, highlighting reductions in hallucinated content and improved alignment with the visual context. Our method effectively mitigates incorrect descriptions by refining modality interactions, leading to more accurate and reliable multi-modal reasoning. Vision-Language Models (VLMs) have achieved remarkable progress in various multi-modal tasks [ 17 , 1 ] , such as image captioning [ 27 ] , visual question answering (VQA), and visual reasoning [ 25 , 31 ] . By integrating visual and textual modalities, VLMs can generate descriptive textual outputs based on image inputs, enhancing machine perception and understanding of multi-modal contexts [ 5 ] . These models typically consist of a vision encoder, responsible for extracting meaningful features from images, and a language model that processes textual inputs and generates outputs conditioned on both modalities. Recent advancements in large-scale pre-training and transformer-based architectures have significantly improved the generalization ability of VLMs [ 41 ] , making them an essential component in modern AI applications. Despite their impressive capabilities, VLMs are prone to hallucination [ 12 ] , a phenomenon where the model generates outputs that are inconsistent with the given visual context or textual prompt. This issue arises when the model produces descriptions, answers, or captions that contain incorrect, fabricated, or misleading information. Hallucination in VLMs can significantly impact their reliability, especially in critical applications such as medical imaging [ 8 ] , autonomous driving [ 3 ] , and surveillance [ 44 ] . The underlying causes of hallucination remain a complex and open research question, making it crucial to investigate further and develop effective mitigation strategies. Several studies have attempted to analyze and mitigate hallucination in VLMs [ 12 , 45 , 33 , 38 ] , leading to different explanations and strategies. Some works attribute hallucination to statistical biases in pre-training datasets [ 45 ] , where models memorize spurious correlations rather than genuinely understanding visual-text relationships. Others suggest that VLMs tend to over-rely on language priors [ 38 , 33 ] , causing them to generate text-based responses that do not sufficiently incorporate visual information. Additionally, some research highlights the role of biased feature learning [ 14 , 4 ] , where certain visual or textual patterns dominate the learned representations, leading to distorted multi-modal reasoning. However, these approaches primarily analyze hallucination from a statistical or empirical perspective and often do not differentiate between VLMs and large language models (LLMs), neglecting the unique challenges posed by the multi-modal structure of VLMs. In this work, we introduce a novel causal perspective [ 28 ] to analyze and mitigate hallucination in VLMs. We construct a causal graph for VLMs, hypothesizing that hallucination arises due to unintended direct influences from either the vision or text modality, bypassing the intended multi-modal fusion process. Specifically, each modality can have independent direct effects on the output, leading to inconsistencies between generated answers and their intended multi-modal context. Based on this causal assumption, we employ counterfactual analysis [ 16 ] to estimate the Natural Direct Effect (NDE) [ 32 ] of each modality. Our goal is to systematically remove each modality’s unintended direct influence from the output, ensuring that the generated response is primarily driven by the joint vision-text fusion process. By reducing these direct effects, we aim to mitigate hallucination in VLMs and improve their overall reliability. To tackle the hallucination problem in VLMs, we propose a three-step methodology. First, we design structural causal graphs [ 28 ] to model the relationships between vision, text, and generated outputs, distinguishing the correct fusion pathway from spurious modality shortcuts. Second, we introduce a systematic approach to estimate the Natural Direct Effect of vision, text, and their cross-modality interaction. For the vision modality, we generate perturbed images by applying multiple random masks, compute their latent representations, and average them to obtain a perturbed representation. We then compare this perturbed representation against the original to quantify the vision’s NDE. Similarly, for the text modality, we generate hallucinated captions using a language model [ 43 ] , extract their representations, and compute the difference between the original and hallucinated text embeddings. To assess cross-modal interactions, we measure how vision enhances or distorts textual grounding by comparing model representations when given structured vision inputs versus degraded or nullified visual signals. To aggregate these effects at a higher level, we perform PCA to extract the primary direction of modality influence across multiple samples. Finally, we develop a dynamic test-time intervention module to adjust the model’s reliance on each modality, effectively reducing hallucination while preserving overall task performance. Our key contributions are as follows: • Causal analysis of hallucination. We identify unintended direct modality influences as a primary cause of hallucination in VLMs. • Test-time hallucination reduction. Our method systematically mitigates hallucination by ensuring proper multi-modal fusion and reasoning. • Superior benchmark performance. Our approach consistently outperforms existing methods on two VLMs across two diverse benchmarks. 2 Related Works Hallucination in Vision-Language Models. Recent work has developed VLMs by integrating visual encoders with pre-trained LLMs [ 6 , 25 , 46 ] . This allows LLMs to interpret vision tokens from a pre-trained backbone, achieving strong multimodal understanding [ 42 ] . However, these models also inherit the LLMs’ tendency to generate ungrounded content, commonly termed “hallucination” [ 2 , 10 , 7 ] . A major issue in VLM hallucinations is the incorrect inclusion of objects absent from the visual input [ 2 , 10 , 22 , 36 ] . Studies suggest this often involves common or co-occurring objects in training data [ 17 ]

Read the complete text