CLAIM : Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention

Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin · 2025

Read CLAIM : Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention by Zekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11073.txt.

Description

Zekai Ye and co-authors stop vision-language models from inventing objects more often when the question is not in English. These models hallucinate in every language, but the errors grow in non-English queries. Fixes usually need costly retraining. Their 2025 paper proposes CLAIM, an intervention that works at run time by borrowing English's better visual attention.

The method finds the attention heads whose cross-modal focus differs by language, measures the shift from English to the target language, and nudges attention outputs during generation to close the gap. On two standard hallucination benchmarks, scores improved by about fourteen and twenty-two percent on average across languages, with Spanish gaining up to thirty. The divergence between languages peaked in the model's middle layers. The authors offer a near-training-free patch for a fairness flaw that retraining budgets rarely reach.

Catalog record

CREATORZekai Ye, Qiming Li, Xiaocheng Feng, Libo Qin, Yichong Huang, Baohang Li, Kui Jiang, Yang Xiang, Zhirui Zhang, Yunfei Lu, Duyu Tang, Dandan Tu, Bing Qin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11073

Browse the full library · About the Tower of Babel