EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models
Andrés Villa 1, Juan León Alcázar 1, Motasem Alfarra 1, Vladimir Araujo 2 · 2025
Read EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models by Andrés Villa 1, Juan León Alcázar 1, Motasem Alfarra 1, Vladimir Araujo 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02699.txt.
Introduction
Andres Villa and co-authors attack a familiar flaw of image aware chat models: they invent things that are not in the picture. Most fixes polish the language side of the model or blend in extra vision parts. Their method, EAGLE, instead retrains the vision encoder itself after its first training round. A reworked version of the original matching task teaches it to ground words in the right image regions.
The tuned encoder drops into existing chat models with no fresh instruction training. Across several hard test sets, invented content fell a great deal. The team stresses that the fix sits in one place, the eyes of the model, so it travels across designs.
Catalog record
| CREATOR | Andrés Villa 1, Juan León Alcázar 1, Motasem Alfarra 1, Vladimir Araujo 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-02699
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models Andrés Villa 1, Juan León Alcázar 1, Motasem Alfarra 1, Vladimir Araujo 2 Abstract Large language models and vision transformers have demonstrated impressive zero-shot capabilities, enabling significant transferability in downstream tasks. The fusion of these models has resulted in multi-modal architectures with enhanced instructional capabilities. Despite incorporating vast image and language pre-training, these multi-modal architectures often generate responses that deviate from the ground truth in the image data. These failure cases are known as hallucinations. Current methods for mitigating hallucinations generally focus on regularizing the language component, improving the fusion module, or ensembling multiple visual encoders to improve visual representation. In this paper, we address the hallucination issue by directly enhancing the capabilities of the visual component. Our approach, named EAGLE , is fully agnostic to the LLM or fusion module and works as a post-pretraining approach that improves the grounding and language alignment of the visual encoder. We show that a straightforward reformulation of the original contrastive pre-training task results in an improved visual encoder that can be incorporated into the instructional multi-modal architecture without additional instructional training. As a result, EAGLE achieves a significant reduction in hallucinations across multiple challenging benchmarks and tasks. Figure 1 : EAGLE visual encoders reduce hallucinations in IT-VLMs. We present three example scenarios, each featuring a question about an image input to a specific IT-VLM with its original visual encoder (left box in pink) and the corresponding EAGLE-tuned visual encoder (right box in orange). “IB7”, “IB13” and “IBT5” refer to InstructBLIP with Vicuna7B, Vicuna13B, and FlanT5xl, respectively. EAGLE substantially reduces hallucinations, providing more visually grounded and reliable descriptions. i) In the first example (left), IB13 with the EAGLE-tuned visual encoder can clearly identify fine-grained elements such as a fence, house, and trees. ii) In the second example (center), EAGLE helps IB7 to accurately recognize a dog, even in an unusual viewpoint and context. iii) In the third example (right), EAGLE enhances object localization, allowing the model to precisely identify the laptop’s position. These examples illustrate EAGLE’s effectiveness in improving visual grounding across complex, multi-object scenes. 1 Introduction Large-scale pre-trained architectures have significantly advanced the fields of Natural Language Processing (NLP), and Computer Vision. In the language domain, Large Language Models (LLMs) have demonstrated remarkable zero-shot performance in multiple NLP tasks Brown et al. 2020 ; Devlin et al. 2018 ; Liu et al. 2019 ; Raffel et al. 2020 ; Chiang et al. 2023 ; Chung et al. 2024 ; Touvron et al. 2023 . This success has resulted in a shift in NLP research, moving from task-specific pre-training to task-agnostic representation learning. In the image domain, visual transformers (ViT) Dosovitskiy et al. 2020 have driven significant advances in zero-shot transferability Radford et al. 2021 ; Sun et al. 2023 ; Oquab et al. 2023 ; Zhai et al. 2023 ; Chen et al. 2020 ; Grill et al. 2020 ; He et al. 2020 ; Cherti et al. 2023 , and enabled new state-of-the-art results on downstream tasks. Building upon the improvements attained by transformer architectures in vision and NLP, Instruction Tuning Vision and Language Models (IT-VLM) have emerged as the ensemble of an LLM and a large-scale ViT fine-tuned with instructional data. IT-VLMs leverage the zero-shot capabilities of both modalities by learning a small fusion sub-network known as adapter or connector Li et al. 2023a ; Liu et al. 2023a ; Dai et al. 2024 ; Zhu et al. 2023 ; Peng et al. 2024 ; Bai et al. 2023 ; Dong et al. 2024 ; Research 2024 . Although the fusion module is far smaller than the other components, it enables IT-VLM models to achieve notable zero-shot performance in multi-modal tasks, such as VQA, Image Captioning, and Image Classification. The open-set natural language responses of IT-VLM allow for the generation of erroneous outputs unrelated to image ground-truth, these false positive errors are known as hallucinations Villa et al. 2024 ; Li et al. 2023b ; Tong et al. 2024 ; Guan et al. 2024 . This phenomenon is often a result of language biases in the decoder module, as the decoding component would generate outputs consistent with the current stream of generated tokens while ignoring the semantic information in the visual modality Villa et al. 2024 . Existing methods for reducing hallucinations in IT-VLMs primarily focus on the language model (LLM) component or the adapter module. These methods include optimizing the adapter module’s training strategy Peng et al. 2024 ; Bai et al. 2023 , improving the adapter module’s architecture Li et al. 2023a ; Dai et al. 2024 ; Liu et al. 2023a , leveraging extensive data to pre-train robust LLMs Chen and Wang 2022 ; Bai et al. 2023 ; Liu et al. 2023a , and enhancing instructional training data through commercial LLMs Liu et al. 2023a ; Liu et al. 2023b . In contrast, we address the hallucination problem from an orthogonal direction by focusing exclusively on the visual component of the IT-VLM. Previous approaches to enhance the visual representation in IT-VLMs Tong et al. 2024 ; Kar et al. 2024 introduce additional visual encoders, which prevent scalability and require further fine-tuning of the IT-VLM. We depart from that practice and develop a method that directly enforces visual grounding on the Vision Transformer (ViT) for a subset of common object classes while preserving the global feature descriptor. This refined ViT effectively encodes fine-grained object class distributions and seamlessly integrates into an IT-VLM model without additional fine-tuning or adaptation. Empirical results confirm that our ViT module consistently reduces hallucinations across diverse benchmarks, demonstrating compatibility and effectiveness across IT-VLM architectures. In Figure 1 , we provide some qualitative examples that show the baseline (left pink box) and improved response (right orange box) of IT-VLMs. Notably, when our ViT is incorporated, we observe significantly more factual and visually grounded responses. In this paper, we introduce E nhanced Visu A l G rounding Minimizes Hallucinations in Instructional Mu L timodal Mod E ls (EAGLE), a pre-training strategy that significantly reduces the hallucinations in IT-VLMs. Unlike other methods, EAGLE does not target a specific IT-VLM architecture, or a particular LLM. In fact, EAGLE is fully agnostic to the other IT-VLM components. At training time, EAGLE optimizes the ViT component in isolation from the LLM and the fusion module; the improved visual transformer is then plugged into the trained IT-VLM at inference time without any further modification. Without any bells and whistles, our tuned ViTs reduce the hallucinations in common IT-VLMs. Across three well-established hallucination benchmarks and 6 different IT-VLMs models, we show direct improvement by simply replacing their default ViT with our optimized version. Contributions. Our contributions are two-fold: i) We mitigate the hallucination problem in IT-VLMs by directly enhancing the grounding of the visual encoder, tuning it to capture fine-grained visual details. We obtain 11.2% relative improvement in the MMVP benchmark and 6.3% relative improvement in the MERLIM benchmark ii) EAGLE is a straightforward and effective approach that reduces hallucinations in IT-VLMs without requiring any additional alignment or tuning. Therefore, EAGLE visual modules can be directly integrated into an IT-VLM. To ensure reproducibility and to foster future research, upon acceptance, we