Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
Haohan Zheng, Zhenguo Zhang · 2025
Read Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens by Haohan Zheng, Zhenguo Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02419.txt.
Description
Haohan Zheng and Zhenguo Zhang wrote a 2025 paper on false objects in the output of vision language models. Such models can name things that are not in the image at all. Past work blamed a tilt toward text, since the models grow out of language systems and can lean on the prompt over the picture.
The authors find a wider fault they call modality bias: at times the model drops either the image or the text from its focus. That split focus breaks its grasp of the user request. Their fix adjusts the attention weights on text and image tokens to rebalance the two streams. A contrast decode step further cuts the pull of stored knowledge. The method needs no retraining.
Catalog record
| CREATOR | Haohan Zheng, Zhenguo Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02419