Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models
Xinmiao Hu 1 Chun Wang 1 Ruihe An 1 ChenYu Shao 1 Xiaojun Ye 1 Sheng Zhou 1 Liangcheng Li 1 1 Zhejiang University {xinmiao_hu, zjuheadmaster, 3220106416, 3230105139, yexiaojun, zhousheng_zju, liangcheng_li}@zju.edu.cn · 2025
Read Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models by Xinmiao Hu 1 Chun Wang 1 Ruihe An 1 ChenYu Shao 1 Xiaojun Ye 1 Sheng Zhou 1 Liangcheng Li 1 1 Zhejiang University {xinmiao_hu, zjuheadmaster, 3220106416, 3230105139, yexiaojun, zhousheng_zju, liangcheng_li}@zju.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19474.txt.
Introduction
Xinmiao Hu and six co-authors attack a familiar flaw in 2025: vision-language models that describe objects which are not in the picture. The errors trace to biased training data, where objects that often appear together become tangled in the model's memory. Shown a kitchen, it may report a toaster that was never there.
Their fix borrows from causal thinking. A projector in the visual pathway and an intervention module near the language model's end both work to break false links between co-occurring things. Tests show far fewer invented objects with no loss on standard benchmarks. Visual checks confirm that object representations sit farther apart and cleaner.
Catalog record
| CREATOR | Xinmiao Hu 1 Chun Wang 1 Ruihe An 1 ChenYu Shao 1 Xiaojun Ye 1 Sheng Zhou 1 Liangcheng Li 1 1 Zhejiang University {xinmiao_hu, zjuheadmaster, 3220106416, 3230105139, yexiaojun, zhousheng_zju, liangcheng_li}@zju.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models Xinmiao Hu 1 Chun Wang 1 Ruihe An 1 ChenYu Shao 1 Xiaojun Ye 1 Sheng Zhou 1 Liangcheng Li 1 1 Zhejiang University {xinmiao_hu, zjuheadmaster, 3220106416, 3230105139, yexiaojun, zhousheng_zju, liangcheng_li}@zju.edu.cn Abstract Multimodal Large Language Models (MLLMs) perform well on visual understanding tasks, but they suffer from object hallucination and generating inconsistent or nonexistent object descriptions due to the dataset biases. This paper explores how dataset biases affect object hallucinations in MLLMs on feature space, revealing that biased object co-occurrences lead to entangled semantic representations across modalities. Thus, the model mistakenly activates the representation of an object that often appears with the input object but is absent, leading to hallucinations. To address this, we propose a causality-based disentanglement framework using causal intervention to deconfound the biases introduced by frequent object co-occurrences in the data. Specifically, we introduce a Causal-Driven Projector in the visual pathway and a Causal Intervention Module in the final LLM transformer layer, which work together to mitigate spurious correlations. Our experiments show a notable reduction in hallucinations while preserving strong performance across several benchmarks, along with visualization analysis confirming improved representation separability. Our code is available at https://github.com/IgniSavium/Causal-LLaVA . 1 Introduction Multimodal Large Language Models (MLLMs) like LLaVA [ 37 , 36 ] , MiniGPT-4 [ 67 ] and InstructBLIP [ 6 ] have demonstrated their remarkable ability in visual understanding tasks, such as image captioning [ 20 ] , visual question answering (VQA) [ 1 ] , and complex multimodal reasoning [ 15 ] . Despite these achievements, MLLMs still suffer from hallucination, where models generate content inconsistent with visual inputs, such as describing nonexistent objects, incorrect attributes, or implausible relationships [ 35 , 3 ] . Among these issues, object hallucination, where models describe inconsistent or nonexistent objects, represents a significant challenge that has drawn substantial attention in the field. Because of the co-occurrence biases on datasets, models will mistakenly infer one object’s presence due to its frequent co-occurrence with another object [ 30 ] . For example, an MLLM might hallucinate chairs around a table when processing visual inputs containing only the table itself, due to learned biased co-occurrence patterns that chairs are always around a table statistically from datasets (see Figure 1 left). Such hallucinations fundamentally undermine MLLMs’ reliability in downstream reasoning, leading to erroneous semantic interpretations, breaking multi-step inference chains built on visual context [ 14 ] , and impairing grounding for sequential vision-based agents in dynamic environments [ 7 ] . These failures not only erode user trust but also pose serious risks in safety-critical applications requiring accurate perception. Figure 1: left: A hallucination example where an MLLM incorrectly describes non-existent objects (chairs and a coffee cup) based on co-occurrence patterns. middle: Representations of original LLaVA’s final LLM Transformer layer. The dining table (green dot, id=67) is surrounded by its top-10 most frequently co-occurring objects (red dots). right: Disentangled representations in our framework, with previously clustered objects now drifting significantly apart. While previous work [ 30 ] has only shown that co-occurrence frequencies are statistically linked to hallucination rates, without exploring the deeper reasons behind this, we investigate the dynamic formation of these biases throughout the representation learning process. This helps us gain a more nuanced understanding of how statistical patterns in the training data gradually shape the model’s internal representations, allowing us to identify critical stages where biases are most likely to be amplified or mitigated. We first examine the distribution of textual and visual representations within the semantic space of MLLMs. In Figure 1 (middle and right), dining table and its top-10 most frequently co-occurring objects on MSCOCO [ 32 ] are visualized via average visual object representations in the original LLaVA’s final LLM Transformer layer using principal component analysis (PCA). We observe that after the trained projector and in the lower half and final layers of the Transformer , the entanglement of hidden states shows strong correlation with the co-occurrence frequency of objects in the instruction dataset (see Figure 1 middle). Based on these findings, we conclude that such spurious correlations arise directly from the biased representation learning process. This representational entanglement causes the model to align the semantic representations of frequently co-occurring objects, thus one object occurs will raise the probability of the other related object occurring with the risk of ignoring the real visual information. Statistically biased co-occurrence and associations embedded in the semantic space of MLLMs are a primary contributor to object hallucinations. We identify these entangled representations as a key underlying cause. They stem from two main sources: (1) object co-occurrence patterns present in the training data, and (2) the model’s inclination to associate objects based on superficial correlations rather than genuine causal relationships [ 21 , 19 , 11 ] . While many existing approaches attempt to address the first issue through data reweighting [ 34 , 22 , 52 ] , they often overlook the second and more fundamental challenge of understanding how the model internally learns and represents semantic relations. To enable MLLMs to comprehend essential semantic and logical structures, it is crucial to establish unbiased representations. Causal inference, which uncovers causal relationships from observational data, offers a principled way to bridge empirical patterns and underlying causality [ 43 ] . Building on this insight, we propose a causality-driven framework with two parts: a Causal-Driven Projector that decouples co-occurring object features, and Causal Intervention Modules that purify hidden states in top Transformer layers. This two-part design breaks the spread of bias during vision-language learning (see Figure 1 right), helping the model tell objects apart and reducing hallucinations from data bias." Empirical evidence demonstrates that our method effectively suppresses hallucinations without compromising performance across various comprehensive benchmarks, with the integration of our Disentanglement Modules into LLaVA yielding a 22.6% increase in the MME-Perception score [ 8 ] . In conclusion, this paper makes the following contributions: 1. We reveal that object hallucinations in MLLMs are rooted in entangled semantic representations formed during training, driven by co-occurrence biases in data. Our analysis focuses on how these representations evolve within the model, offering a new perspective at the representational level. 2. We design a causality-based disentanglement architecture that separates object representations to reduce hallucinations from commonly confused combinations. This module can be easily adapted to different multimodal large language model architectures. 3. We empirically validate that our approach not only effectively reduces hallucinations but also preserves strong performance across multiple benchmarks. The code and framework will be released publicly for further research. 2 Related Work 2.1 Hallucination in MLLMs Hallucination in Multimodal Large Language Models (MLLMs) refers to generating content inconsistent with input images [ 3 ] . The causes of hallucinations in MLLMs are multifaceted, including over-reliance on parametric or linguistic knowledge [ 65 , 56 ] , biases in training data (distribution imbalances and annotation irrelevance) [ 12 , 62 ] , insufficient attention to contextual information in large language models (LLMs) [ 25 , 13 ] , and resolution-constrained vision encoders that struggle to capture fine-grained semantics [ 31 , 46 ] , all of which collectively exacerbate hallucination phenomena. Prior efforts to mitigate hallucinations have explored diverse strategies. Wang