Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens

Haohan Zheng, Zhenguo Zhang · 2025

Read Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens by Haohan Zheng, Zhenguo Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02419.txt.

Description

Haohan Zheng and Zhenguo Zhang wrote a 2025 paper on false objects in the output of vision language models. Such models can name things that are not in the image at all. Past work blamed a tilt toward text, since the models grow out of language systems and can lean on the prompt over the picture.

The authors find a wider fault they call modality bias: at times the model drops either the image or the text from its focus. That split focus breaks its grasp of the user request. Their fix adjusts the attention weights on text and image tokens to rebalance the two streams. A contrast decode step further cuts the pull of stored knowledge. The method needs no retraining.

Catalog record

CREATORHaohan Zheng, Zhenguo Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02419

Browse the full library · About the Tower of Babel