Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

Zhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen · 2026

Read Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating by Zhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24024.txt.

Introduction

Cheng and three co-authors trace vision-language hallucinations to route competition, where text can dominate the choice despite attention. Their training-free fix splits each attention head into visual and text routes, estimating token-level effects with a one-forward, one-gradient approximation. That reveals within-head route conflict and prior-dominant heads, so only the text route is suppressed while the visual route stays intact. Across five benchmarks in discriminative and generative settings, the method cuts hallucination errors with little harm and modest inference overhead.

Catalog record

CREATORZhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110575
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24024

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating Zhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen Abstract Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead. † † affiliationnotice: Equal contribution 1 Introduction LVLMs have become a core interface for visual understanding, enabling systems that can answer questions about images and generate grounded descriptions ( 59 ; 3 ) . However, a central obstacle to reliable deployment is hallucination: the model produces fluent, plausible content that is not supported by the image ( 30 ) . Importantly, this is not merely “poor writing” or generic inaccuracy; rather, it often reflects a failure of grounding, where the model fills in missing visual details using language priors ( 12 ; 55 ; 67 ) . Such prior-driven fabrications directly undermine trustworthiness and can be harmful in safety-critical or decision-support settings ( 54 ) . Figure 1 : When language priors override visual inputs, the baseline model tends to produce hallucinated predictions (left). To address this, a conflict-aware intervention applies text-route gating to suppress language priors (middle), enabling the model to correctly rely on visual evidence after intervention (right). At a high level, hallucination often reflects a mismatch between visual evidence and language priors: the model may attend to the image, yet the final token choice is still dominated by the textual pathway, favoring what is plausible over what is visible. Training-free mitigation has become attractive for deployment. Among approaches that act at inference time without additional training, two representative paradigms emerge: (i) output-level decoding control, and (ii) proxy-guided internal intervention. The first paradigm modifies generation behavior through decoding-time heuristics or constraints (e.g., Visual Contrastive Decoding (VCD) ( 26 ) , OPERA ( 21 ) , MaskCD ( 9 ) ). While effective, these methods primarily act as external policies and provide limited insight into which internal components drive prior-dominant errors. The second paradigm performs internal intervention by selecting modules using attention-based proxies ( 5 ; 62 ) , most notably VAR-style measures ( 24 ; 65 ; 25 ) that quantify the attention share on visual tokens. Yet such proxies are inherently correlational: higher visual attention does not necessarily translate to larger causal contribution to the output ( 23 ; 42 ) , and softmax normalization can spuriously change the visual share by weakening textual competition ( 48 ) . Moreover, proxy-based selection is typically paired with coarse head rescaling ( 24 ; 61 ) , which inevitably suppresses both routes and may harm genuinely grounded reasoning. To move beyond attention-based proxies, we expose a head-internal visual route and text route , so that we can intervene on either route at inference time (Fig. 1 ). For the current token y^{*} , we measure decision-aligned route effects by do interventions: \Delta^{\mathrm{vis}}_{l,h}=\ell_{l,h}(1,1)-\ell_{l,h}(0,1) and \Delta^{\mathrm{txt}}_{l,h}=\ell_{l,h}(1,1)-\ell_{l,h}(1,0) . These effects directly reveal route conflict : a head is conflicting when \mathrm{sign}(\Delta^{\mathrm{vis}}_{l,h})\neq\mathrm{sign}(\Delta^{\mathrm{txt}}_{l,h}) , with Conflict-A (+,-) and Conflict-B (-,+) . Within conflicting heads, we identify prior-dominant ones by VRI_{l,h}=\frac{|\Delta^{\mathrm{vis}}_{l,h}|}{|\Delta^{\mathrm{vis}}_{l,h}|+|\Delta^{\mathrm{txt}}_{l,h}|+\epsilon} (smaller VRI_{l,h} indicates more text-driven behavior). At decoding time, we estimate \Delta^{\mathrm{vis}}_{l,h} and \Delta^{\mathrm{txt}}_{l,h} with a one-forward/one-gradient approximation, select Top- k smallest VRI_{l,h} within the conflict sets , and suppress only their text routes using a smooth rank-based schedule, with stronger suppression for Conflict-B than Conflict-A (visual routes remain intact). We evaluate our method on five benchmarks spanning discriminative and generative settings. Results indicate that the proposed method can reduce hallucination-related errors across models, with limited impact on overall multimodal performance. Contributions. (1) We introduce route-level, decision-aligned causal effects for attention heads and a corresponding Visual Reliance Index. (2) We derive an efficient first-order estimator that enables per-step head selection during decoding. (3) We propose conflict-aware text-route gating that targets prior-dominant, decision-conflicting heads without suppressing visual routes. (4) We theoretically analyze VAR-style proxies, showing attention mass can be misaligned with decision-relevant visual grounding. 2 Related Work Hallucination in LVLMs. Hallucination in LVLMs has received growing attention because it undermines visual grounding ( 28 ; 53 ; 13 ; 57 ) . Unlike textual LLMs, LVLMs exhibit prominent object hallucination ( 40 ) , where the model mentions objects or attributes not supported by the image. Mitigation methods span training-based alignment/fine-tuning ( 29 ; 19 ; 20 ) and training-free inference-time control ( 60 ; 47 ; 7 ; 63 ; 46 ) . On the training-free side, methods broadly fall into output-level decoding control and intervention-style approaches that act on internal computation at inference time ( 41 ) . For example, PAI ( 33 ) encourages stronger reliance on visual evidence, and latent-space steering methods such as VTI ( 32 ) modify hidden representations to suppress hallucination tendencies. Our work is most closely related to these training-free intervention paradigms, but differs in where and how we act: we compute token-level within-head route effects and apply route-specific gating at decoding time to reduce language-prior influence. Causal Interventions for Mechanistic Attribution. Mechanistic interpretability ( 11 ; 43 ) increasingly emphasizes interventional attribution, where one perturbs internal states and measures the induced behavioral change ( 16 ; 15 ) . Prior work on pruning and ablations suggests substantial redundancy in multi-head attention, but indiscriminate removals are often a blunt instrument and do not directly localize causal pathways ( 35 ; 51 ) . To pinpoint causal structure, Causal Mediation Analysis (CMA) ( 49 ; 56 ) and activation patching (a.k.a. causal tracing) ( 64 ) perform targeted interventions under controlled inputs to estimate component effects ( 50 ; 16 ; 34 ; 56 ) , and path-level methods ( 37 ) further attribute behaviors to specific information-flow paths ( 17 ) . Since head contributions can be highly interactive and non-modular, Causal Head Gating (CHG) learns differentiable head gates and searches for sufficient head sets directly from data, reducing reliance on hand-crafted prompt pairs ( 36 ) . Building on this interventional line, we extend head-level interventions to within-head routes: unlike CHG’s learned head gates, we estimate route-level effects to drive decoding-time route-specific gating that reduces language-prior dominance and maintains visual grounding. 3 Causal Route Effects Figure 2 : Overview of our CRG framework. Step 1 computes modal causal effects for each attention head by separating visual and textual components and measuring their contributions. Step 2 identifies conflicts by comparing the signs of visual and textual effects, distinguishing Agreement vs. Conflict (A/B), and selects the top- k conflicting heads for intervention. Step 3

Read the complete text