Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs
Chenchen Lin, Sanbao Su, Rachel Luo, Yuxiao Chen, Yan Wang, Marco Pavone, Fei Miao · 2026
Read Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs by Chenchen Lin, Sanbao Su, Rachel Luo, Yuxiao Chen, Yan Wang, Marco Pavone, Fei Miao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03100.txt.
Introduction
Chenchen Lin and six co-authors reduce hallucination in multimodal language models by making better use of the vision encoder's many layers. Such models usually read only one late-layer feature from a frozen image encoder, wasting the hierarchy of visual cues stored across layers, so the model leans on language habits and invents details. Their module, TGIF, treats the encoder's layers as depth-wise experts.
TGIF predicts a prompt-dependent blend of those layer features, so the visual input adapts to what the user actually asked. It needs no updates to the vision encoder and adds little overhead. Plugged into LLaVA-1.5-7B, it improved results across hallucination, text-recognition, and visual question answering benchmarks, while holding or improving performance on several general benchmarks as well.
Catalog record
| CREATOR | Chenchen Lin, Sanbao Su, Rachel Luo, Yuxiao Chen, Yan Wang, Marco Pavone, Fei Miao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs Chenchen Lin, Sanbao Su, Rachel Luo, Yuxiao Chen, Yan Wang, Marco Pavone, Fei Miao Abstract Multimodal large language models (MLLMs) typically rely on a single late-layer feature from a frozen vision encoder, leaving the encoder’s rich hierarchy of visual cues under-utilized. MLLMs still suffer from visually ungrounded hallucinations, often relying on language priors rather than image evidence. While many prior mitigation strategies operate on the text side, they leave the visual representation unchanged and do not exploit the rich hierarchy of features encoded across vision layers. Existing multi-layer fusion methods partially address this limitation but remain static, applying the same layer mixture regardless of the query. In this work, we introduce TGIF (Text-Guided Inter-layer Fusion), a lightweight module that treats encoder layers as depth-wise “experts” and predicts a prompt-dependent fusion of visual features. TGIF follows the principle of direct external fusion, requires no vision-encoder updates, and adds minimal overhead. Integrated into LLaVA-1.5-7B, TGIF provides consistent improvements across hallucination, OCR, and VQA benchmarks, while preserving or improving performance on ScienceQA, GQA, and MMBench. These results suggest that query-conditioned, hierarchy-aware fusion is an effective way to strengthen visual grounding and reduce hallucination in modern MLLMs. Our code will be available at: https://github.com/Linchenchen/TGIF . Figure 1 : Comparison of layer fusion designs in MLLMs. (a) MLP Connector: Uses only the penultimate layer of the vision encoder, mapping global visual tokens through a simple projection. (b) Dense Connector: Aggregates multi-layer visual features via concatenation or downsampling before projection, enriching semantics but with a fixed fusion pattern. (c) MMFuser: Retrieves shallow-layer features using deep-layer queries (Q–K/V attention) to capture local details. (d) Proposed TGIF: Introduces a text-guided layer router that dynamically reweights features from multiple layers based on the input query, enabling adaptive, context-aware visual fusion. 1 Introduction Multimodal large language models (MLLMs) have recently achieved impressive progress on visual question answering, captioning, and open-ended multimodal dialogue by combining the reasoning ability of large language models (LLMs) with the perceptual capacity of pretrained vision encoders 25 ; 18 ; 9 . Most state-of-the-art systems adopt a modular design: a frozen vision encoder (e.g., CLIP ViT), a lightweight connector, and a powerful LLM decoder. The connector projects visual embeddings into the LLM’s token space, enabling the model to jointly process image and text tokens for downstream reasoning. Despite these advances, modern MLLMs still frequently produce confident but visually ungrounded descriptions, a phenomenon broadly referred to as hallucination 31 ; 23 . In the vision–language setting, hallucination typically manifests as objects, attributes, or relations that are plausible under language priors but inconsistent with the input image. This issue is especially severe for detail-oriented tasks (e.g., OCR, small object recognition), where high-level semantic features alone are insufficient to support fine-grained grounding. Prior work tackles hallucination from two main angles. Training-based approaches improve alignment via additional instruction tuning, contrastive fine-tuning, or RLHF 13 ; 42 , but they are often data- and compute-intensive. Training-free approaches instead modify decoding or inference without retraining the model. Methods such as VCD 15 , VTI 26 , OPERA 14 , FarSight 34 , and PerturboLLaVA 5 mitigate hallucination via contrastive calibration, latent intervention, causal masking, or over-trust penalties. However, these strategies primarily operate on the text side: they adjust the decoder or filter responses post hoc, while the underlying visual representation typically remains a single, fixed layer of the vision encoder passed through an MLP projector. Meanwhile, transformer-based vision encoders such as CLIP are known to build a rich hierarchy of visual abstractions across layers: shallow layers preserve edges, textures, and local geometry, whereas deeper layers encode high-level semantics aligned with text 30 ; 11 . Recent works have begun to exploit this hierarchy by fusing features from multiple depths. DenseConnector 38 concatenates or downsamples features from selected layers, and MMFuser 4 retrieves shallow-layer details using deep features as queries. A recent systematic study further shows that direct, external fusion of multi-layer visual features at the input stage yields the most stable performance across architectures 20 . However, existing fusion schemes are static : the mixture of layers is fixed once chosen, independent of the question, and tends to overemphasize globally aligned semantics even when the query requires local evidence. In this work, we argue that hallucination in MLLMs is fundamentally linked to how visual features are selected and exposed to the LLM. We propose TGIF ( T ext- G uided I nter-layer F usion), a dynamic routing framework that treats the layers of a frozen vision encoder as a pool of specialized “experts” and adaptively fuses them based on the input query. Concretely, TGIF introduces a lightweight router inside the multimodal projector that takes text (and optionally a global image feature) as input and outputs a soft distribution over all ViT layers. These learned weights are then used to form a query-conditioned fused visual representation, which is projected into the LLM token space using a standard MLP connector (Fig. 2 ). Following the principle of direct external fusion 20 , TGIF keeps the vision encoder frozen and preserves the token budget, while allowing the layer mixture to vary per prompt. We instantiate TGIF on top of LLaVA-1.5 16 with CLIP-ViT-L/14 and Vicuna-7B, and evaluate it across three benchmark families: hallucination, OCR, and general VQA. On hallucination-focused evaluations, TGIF achieves state-of-the-art performance among 7B-scale LLaVA variants, improving POPE accuracy from 86.85% to 87.91% and boosting HallusionBench All Accuracy from 46.90% to 49.94%, outperforming both decoding-based baselines (VCD, OPERA, VTI, FarSight, PerturboLLaVA) and larger 13B models. On OCRBench, TGIF improves the final score from 297 to 313 (+16), and provides consistent gains on TextVQA. At the same time, it maintains or improves overall performance on ScienceQA, GQA, and MMBench, indicating that better grounding does not come at the expense of high-level reasoning. Our main contributions are three-fold: • We identify a key limitation of current multimodal LLMs: visual tokens are typically drawn from a single, late-layer representation, which is poorly suited for detail-sensitive grounding and exacerbates hallucination under strong language priors. • We propose TGIF , a text-guided inter-layer fusion module that dynamically reweights CLIP layers per query, following the direct external fusion principle while remaining parameter- and token-efficient. We explore both text-only and multimodal routers, and introduce an entropy-based load-balancing loss to prevent expert collapse. • We demonstrate that TGIF substantially improves hallucination robustness and fine-grained visual perception on POPE, HallusionBench, OCRBench, and TextVQA, while preserving competitive performance on general reasoning benchmarks. Qualitative analysis of router behavior further shows that TGIF learns semantically meaningful depth-selection patterns across task types. 2 Related Work 2.1 Multimodal Large Language Models Multimodal large language models (MLLMs) integrate the reasoning ability of LLMs with the perceptual capacity of pretrained vision encoders. Most follow a modular design with a frozen vision encoder, a lightweight connector, and a powerful LLM decoder. The connector projects visual embeddings into the text space, enabling cross-modal alignment essential for grounded reasoning. Early designs used simple MLP projectors 25 , while later approaches like BLIP-2 18 and InstructBLIP 9 employ query-based modules for salient visual token extraction. 2.2 Hallucination in MLLMs MLLMs frequently generate confident yet ungrounded content—a phenomenon known as hallucination 31 ; 23 . Training-based