When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs

Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li · 2026

Read When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs by Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00344.txt.

Description

Beidi Zhao and team diagnose attention distraction in retrieval augmented large vision language models. Retrieval augmentation leads knowledge based visual question answering, yet failures were blamed on too little focus on context. The authors identify a distinct failure mode: when retrieved context suffices, that text dims visual attention globally, shifting image focus away from question relevant regions. That leads to failures on questions that the model could answer correctly even without that text. Their training free fix, MAD-RAG, separates visual grounding from context integration via a dual question formulation.

Catalog record

CREATORBeidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45391
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.00344

Browse the full library · About the Tower of Babel