Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation
Jiayu Yao, Shenghua Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Yuyao Ge, Zhecheng Li, Xueqi Cheng · 2025
Read Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation by Jiayu Yao, Shenghua Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Yuyao Ge, Zhecheng Li, Xueqi Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11063.txt.
Introduction
Jiayu Yao and seven co-authors expose a position bias in AI systems that answer questions using retrieved text and images, in a 2025 paper. Such systems fetch evidence, then reason over it. The team asked a simple question: does where a piece of evidence sits in the pile change the answer?
Across text-only, image-only, and mixed tasks, accuracy traced a U shape: evidence at the start or end was used well, evidence in the middle was neglected. Mixing images with text made the bias worse, and it grew as more items were retrieved. The authors built an index to measure the effect and traced attention patterns inside the model to watch it happen. Retrieval systems, it turns out, play favorites like human readers skimming a list.
Catalog record
| CREATOR | Jiayu Yao, Shenghua Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Yuyao Ge, Zhecheng Li, Xueqi Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11063
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation Jiayu Yao, Shenghua Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Yuyao Ge, Zhecheng Li, Xueqi Cheng Abstract Multimodal Retrieval-Augmented Generation (RAG) systems have become essential in knowledge-intensive and open-domain tasks. As retrieval complexity increases, ensuring the robustness of these systems is critical. However, current RAG models are highly sensitive to the order in which evidence is presented, often resulting in unstable performance and biased reasoning, particularly as the number of retrieved items or modality diversity grows. This raises a central question: How does the position of retrieved evidence affect multimodal RAG performance? To answer this, we present the first comprehensive study of position bias in multimodal RAG systems. Through controlled experiments across text-only, image-only, and mixed-modality tasks, we observe a consistent U-shaped accuracy curve with respect to evidence position. To quantify this bias, we introduce the Position Sensitivity Index ( P S I p PSI_{p} ) and develop a visualization framework to trace attention allocation patterns across decoder layers. Our results reveal that multimodal interactions intensify position bias compared to unimodal settings, and that this bias increases logarithmically with retrieval range. These findings offer both theoretical and empirical foundations for position-aware analysis in RAG, highlighting the need for evidence reordering or debiasing strategies to build more reliable and equitable generation systems. 1 Introduction The growing demand for multimodal interaction has driven the development of multimodal Retrieval-Augmented Generation (RAG) systems, which integrate heterogeneous data sources (text, images, audio) to achieve comprehensive information understanding Abootorabi et al. 2025 . This technological advancement has enabled breakthroughs across diverse domains: academic research leverages frameworks like Taichu-mRAG, OmniSearch, VARAG Faysse et al. 2024 and GraphRAG Edge et al. 2025 for knowledge discovery, while industrial applications employ DocPrompting Zhou et al. 2023 , UniFashion Zhao et al. 2024 , RAG-Driver Yuan et al. 2024 and Img2Loc Zhou et al. 2024b for real-world problem solving. In professional fields, systems like RULE Xia et al. 2024b and MMed Xia et al. 2024a enhance medical diagnostics. In other scenarios, SoccerRAG Strand et al. 2024 and MMRA Wu et al. 2024 research demonstrates the social computing and entertainment applications. These successes highlight multimodal RAG’s potential in open-domain question answering and knowledge-intensive scenarios. Figure 1: Left An example of RAG for open question answering, where the prompt contains the question and top-k sorted documents. Mid Different positions of the search results may lead to different answers. Right Accuracy analysis when correct document ranks at position i ( i\in k,k=20 ) - gray blocks display 10 random experiments, red line indicates mean accuracy. However, the reliability of multimodal RAG systems faces critical challenges as applications expand. Current systems exhibit vulnerability when faced with complex retrieval problems: excessive or insufficient retrieved content often induces hallucination. Ensuring that generated outputs faithfully adhere to the provided context is crucial for overall system robustness Bi et al. 2024 , yet even with optimal corpus selection, the positional arrangement of retrieved results significantly impacts answer reliability when fed to generation models. This instability aligns with emerging researches on systematic position bias in contemporary Large Language Models (LLMs) and Vision-Language Models (VLMs) Tan et al. 2024 ; Zhang et al. 2024 . These models disproportionately focus on the start and end positions of input sequences while neglecting middle content, a phenomenon we term "middle-loss". However, the existing research presents two critical gaps. First, current studies primarily address single-modal RAG systems (e.g. text-based), lacking systematic investigation of multimodal scenarios. Second, conventional evaluation metrics (e.g. NDCG, MRR) fail to quantify positional sensitivity, with no interpretable framework established for bias analysis in multimodal contexts. To address these limitations, we obtained the following research results through controlled experiments and interpretability analysis: • We show that the multimodal RAG system has position bias (Figure 1 ), and in text, image and mixed-modality scenarios, the generate model will give different degrees of attention to the evidence at different positions, forming a U-shaped accuracy curve. • We propose a position sensitivity index PSI_{p} to quantify the amplitude of position bias. Through benchmarking across modalities and retrieval scales ( k ), we demonstrate that cross-modal interactions and larger retrieval sizes may amplify bias severity. Furthermore, we conduct mathematical modeling between the retrieval scale and the bias amplitude, providing guidance for balancing the retrieval range and robustness in deployment. • Through systematic attention visualization analysis, we experimentally validate the above conclusions and uncover layer-specific sparsity patterns in cross-modal attention across decoder hierarchies. 2 Position Bias To better understand how the position of retrieved evidences affect reasoning in multimodal RAG systems, we begin with a set of controlled experiments designed to probe position bias across diverse input modalities. This section presents both the experimental setup and key empirical findings, revealing a consistent pattern of position-induced performance fluctuation. By systematically perturbing the position of gold evidence while holding content constant, we are able to isolate and quantify the impact of sequence order on model behavior across text-only, image-only, and mixed-modality contexts. To simulate real-world open-domain question answering and knowledge-intensive scenarios, we conducted controlled experiments across three modality configurations: text-only, image-only, and image-text mixed. As depicted in Figure 2 , our framework emulates practical RAG workflows where systems process multiple evidence documents (text passages, charts, or multimodal pairs) alongside user queries to generate answers. Through systematic dataset perturbation, we preserve semantic content while manipulating gold evidence positions relative to distractors, with 10 randomized experiments per configuration to ensure statistical robustness. Through this design, we isolate positional effects from content biases, enabling precise characterization of context ordering sensitivity. Figure 2: Order sensitivity evaluation tasks in three modality configurations: (a) Text-only, (b) Image-only, (c) Mixed-modality. 2.1 Benchmark Data For text-based QA simulation, we leverage the MS-MARCO passage ranking benchmark Nguyen et al. 2016 , constructing triplets comprising a query, one gold passage containing the answer, and two topically relevant but non-answer distractor passages. Gold passage permutation across three positions (Top/Mid/End) mimics real-world multi-document retrieval scenarios. In VQA tasks, we use ChartQA Masry et al. 2022 to create chart triplets (one answer-containing gold chart and two distractors) with identical positional variations to evaluate pure image processing. For image-text mixed-modality evaluation, we use the VEGA dataset Zhou et al. 2024a . Each sample in this dataset includes a question, several image-text pairs, and a golden answer based on a specific pair. In our experiment, for each query, we selected the correct image-text pairs and randomly chose two pairs of unrelated images and paragraphs as interference pairs. Then, we set the same position transformation to evaluate whether the multimodal RAG model shows sequence sensitivity when reasoning about interlaced visual and textual evidence. Models Our evaluation protocol preserves validity through frozen model parameters and deactivated fine-tuning functions, ensuring comparisons reflect inherent architectural biases rather than training artifacts. For the retrieval stage, we implement cross-modal retrieval using the VisRAG-Ret Yu et al. 2025 . In the generation stage, we systematically evaluate three cutting-edge open-source instruction-tuned models (Qwen2-VL-7B-Instruct Wang et al. 2024a , Llama-3.2-11B-Vision-Instruct AI@Meta 2024 , MINICPM-v2.6 OpenBMB 2024 ) and the closed-source GPT-4o OpenAI 2024 , which support multi-image input integration and cross-modal attention interactions. 2.2 Empirical Patterns of Positional Bias Our