PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model
Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, Chris Thomas · 2025
Read PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model by Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, Chris Thomas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12206.txt.
Introduction
Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, and Chris Thomas cure a bad habit of image-describing models: inventing objects that are not there. Such hallucinations, they find, grow as the model's inner attention drifts away from the image. Deeper layers lean ever less on visual tokens. Simply boosting all visual attention helps only so much.
The team singles out two token families that carry the visual story. Local tokens hold grounded facts about objects actually present. Summary tokens hold the image's overall gist. Each family needs its own boost size, learned by experiment. Their framework, PAINT, adjusts attention weights for just these tokens during generation. It plugs into existing models without retraining. On a standard captioning set, hallucination rates fell clearly. The fix targets the tokens that matter instead of flooding attention everywhere.
Catalog record
| CREATOR | Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, Chris Thomas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12206
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model Kazi Hasan Ibn Arif, Sajib Acharjee Dip, Khizar Hussain, Lang Zhang, Chris Thomas Abstract Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities in understanding and describing visual content, achieving state-of-the-art performance across various vision-language tasks. However, these models often generate descriptions containing objects or details that are absent in the input image, a phenomenon commonly known as hallucination. Our work investigates the key reasons behind this issue by analyzing the pattern of self-attention in transformer layers. We find that hallucinations often arise from the progressive weakening of attention weight to visual tokens in the deeper layers of the LLM. Some previous works naively boost the attention of all visual tokens to mitigate this issue, resulting in suboptimal hallucination reduction. To address this, we identify two critical sets of visual tokens that facilitate the transfer of visual information from the vision encoder to the LLM. Local tokens encode grounded information about objects present in an image, while summary tokens capture the overall aggregated representation of the image. Importantly, these two sets of tokens require different levels of weight enhancement. To this end, we propose PAINT ( P aying A ttention to IN formed T okens), a plug-and-play framework that intervenes in the self-attention mechanism of the LLM, selectively boosting the attention weights of local and summary tokens with experimentally learned margins. Evaluation on the MSCOCO image captioning dataset demonstrate that our approach reduces hallucination rates by up to 62.3% compared to baseline models while maintaining accuracy. Code is available at https://github.com/hasanar1f/PAINT 1 Introduction Large Vision Language Models (LVLMs) attract considerable interest due to advancements in pre-trained models that bridge visual and textual embedding spaces [ 32 , 16 , 35 , 11 , 38 , 6 ] . This focus spans both new architectural designs [ 10 , 46 , 13 , 26 , 18 , 44 , 23 , 12 ] and the creation of comprehensive benchmarking datasets [ 30 , 40 , 20 ] . Similar to text only LLMs, LVLMs experience hallucination generating descriptions for nonexistent objects in images, which affects accuracy in critical areas like medical imaging [ 41 , 37 , 29 , 27 , 19 ] . Figure 1 : Imbalanced self-attention weights observed in LLaVA-1.5. Image tokens (labeled in red ) receive lower attention weights than text tokens, especially in shallow layers (e.g., 16, 32). A typical LVLM consists of a pretrained Vision Transformer (ViT) [ 8 , 43 ] and a pretrained Large Language Model (LLM) [ 5 , 45 , 3 ] with a lightweight MLP projector. First, the input image is transformed into visual tokens through the ViT and project layer. Then, the LLM process the visual and text tokens together to generate the response. LVLMs often generate plausible but incorrect information about the image, commonly known as hallucination. To investigate this phenomena, we visualize the attention values (before applying softmax) for both visual and text tokens during the generation process, as shown in Fig 1 . The results show that in shallow layers (e.g., Layer 1), attention is distributed relatively uniformly across both visual and text tokens, suggesting initial comprehensive processing of all input modalities. In intermediate layers (e.g., Layer 16), the model begins to show preferential attention to specific token clusters, particularly focusing on summary tokens that capture high-level semantic information. Finally, in deeper layers (e.g., Layer 32), attention becomes heavily concentrated on a small subset of tokens, primarily text-based summary tokens, while attention to visual tokens diminishes significantly. In summary, we observe a progressive weakening of attention weights in LVLMs. This pattern contributes to the hallucination observed in current MLLMs [ 15 ] . To mitigate hallucination, a recent work, PAI [ 28 ] suggests to boost up the attention weights of visual tokens in LLM layers. However, this uniform attention enhancement leads to suboptimal hallucination reduction. Our investigation reveals that there exist two critical sets of visual tokens that facilitate the transfer of visual information from the vision encoder to the LLM. Local tokens encode grounded information about objects present in an image, while summary tokens capture the overall aggregated representation of the image. Importantly, these two sets of tokens require different levels of attention enhancement. To this end, we propose PAINT ( P aying A ttention to IN formed T okens), a plug-and-play self-attention modification in the LLM, that selectively boosts the attention weights of top N local and summary tokens with some margins \alpha and \beta . The value of the margins as well as N are identified through our experiments. We evaluated our method on the MSCOCO [ 24 ] dataset, demonstrating that our approach reduces hallucination rates by up to 62.3% compared to the previous method, PAI [ 28 ] , while maintaining strong task performance. In summary, we make three major contributions in this study. (1) Through experiments on vision encoders, we identify local tokens that store grounded information and summary tokens that store aggregated information from images, which are then transferred to the LLM. (2) We propose a plug-and-play technique that requires no additional training and intervenes in the self-attention layers of LVLMs, boosting the attention of local and summary tokens with learned margins to mitigate hallucination. (3) We evaluate our method on standard hallucination benchmarks, demonstrating that it outperforms state-of-the-art approaches. 2 Previous Works Many recent studies have explored various techniques to mitigate hallucinations in Large Vision Language Models (LVLMs) [ 31 , 21 , 34 , 17 , 42 , 36 , 47 , 28 , 14 , 1 , 9 ] . Among them, the PAI (Paying Attention to Image) [ 28 ] stands out for its simple yet effective hallucination mitigation. At its core, PAI [ 28 ] uniformly enhance the self-attention weights of visual tokens by a constant factor \alpha . Additionally, PAI [ 28 ] employs logit refinement to balance visual and textual contributions. While PAI [ 28 ] improves upon previous baselines, it has several limitations. The primary issue is that it applies a uniform boost to all visual tokens, disregarding their individual significance. This indiscriminate enhancement can amplify noise and degrade model performance. Additionally, the static enhancement factor \alpha does not adapt to different contexts, and the framework treats visual tokens as independent entities, ignoring spatial and semantic relationships. 3 Methodology Figure 2 : CLS-to-patch attention in ViT [ 8 ] . Initial layers (e.g., 0-1) highlight local patches containing fine-grained visual details ( local tokens ), while deeper layers (e.g., 22-23) emphasize summary tokens that aggregate global information ( summary tokens ). We propose Paying Attention to INformed Tokens (PAINT), a hallucination mitigation method. Unlike previous methods that uniformly treat all visual tokens the same, we identify two kinds of visual tokens, known as local and summary tokens (detailed in section 3.1 ), and boost their attention with \alpha and \beta , respectively, as discussed in section 3.3 . PAINT is illustrated in Fig 3 . 3.1 Local and Summary Tokens Inspired by previous studies [ 7 , 2 ] , we observe a distinct phenomenon in how ViT encodes the input image. The initial layers of ViT