When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li · 2026
Read When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs by Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00344.txt.
Description
Beidi Zhao and team diagnose attention distraction in retrieval augmented large vision language models. Retrieval augmentation leads knowledge based visual question answering, yet failures were blamed on too little focus on context. The authors identify a distinct failure mode: when retrieved context suffices, that text dims visual attention globally, shifting image focus away from question relevant regions. That leads to failures on questions that the model could answer correctly even without that text. Their training free fix, MAD-RAG, separates visual grounding from context integration via a dual question formulation.
Catalog record
| CREATOR | Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45391 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00344