Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang, Kewen Wang, Ran Gong, Jinhan Zhang, Chihang Wang · 2026
Read Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering by Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang, Kewen Wang, Ran Gong, Jinhan Zhang, Chihang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27724.txt.
Introduction
Chen and seven colleagues propose MED-VRAG, an iterative multimodal retrieval system for medical question answering that reasons over document page images instead of OCR text. It couples patch-level page embeddings to a sharded MapReduce filter, scaling to about 350,000 pages with fast first-stage retrieval. A vision-language model then refines its query and banks evidence across up to three reasoning rounds. Across four medical benchmarks, MED-VRAG reaches 78.6 percent average accuracy, with retrieval adding about 6 points over no retrieval.
Catalog record
| CREATOR | Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang, Kewen Wang, Ran Gong, Jinhan Zhang, Chihang Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26536 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang, Kewen Wang, Ran Gong, Jinhan Zhang, Chihang Wang Abstract Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual content (tables, figures, structured layouts) of original document pages. We propose MedVRAG , an iterative multimodal RAG framework that retrieves and reasons over PMC document page images instead of OCR’d text. The system pairs ColQwen2.5 patch-level page embeddings with a sharded MapReduce LLM filter, scaling to ∼ \sim 350K pages while keeping Stage-1 retrieval under 30 ms via an offline coarse-to-fine index ( C = 8 C{=}8 centroids per page, ANN over centroids, exact two-way scoring on the top- R R shortlist). A vision-language model (VLM) then iteratively refines its query and accumulates evidence in a memory bank across ≤ \leq 3 reasoning rounds, with a single iteration costing ∼ \sim 15.9 s and the full three-round pipeline ∼ \sim 47.8 s on 4 × \times A100. Across four medical QA benchmarks (MedQA, MedMCQA, PubMedQA, MMLU-Med), MedVRAG reaches 78.6% average accuracy. Under controlled comparison with the same Qwen2.5-VL-32B backbone, retrieval contributes a + 5.8 +5.8 -point gain over the no-retrieval baseline; we also note a + 1.8 +1.8 -point edge over MedRAG + GPT-4 (76.8%), with the caveat that this is a cross-paper rather than head-to-head comparison. Ablations isolate + 1.0 +1.0 from page-image vs. text-chunk retrieval, + 1.5 +1.5 from iteration, and + 1.0 +1.0 from the memory bank. 1 Introduction Knowledge-intensive multimodal question answering—answering questions whose evidence lives in document pages with mixed text, tables, and figures—stresses both retrieval and reasoning under non-trivial compute budgets. Medical QA is a particularly demanding instance: the underlying biomedical literature is large ( \sim 350K relevant PMC pages in our setup), pages routinely encode dosage tables and diagnostic flowcharts that linearize poorly, and many questions require synthesizing evidence from multiple sources, motivating an iterative retrieve-reason loop rather than a single pass. We use medical multiple-choice QA as the testbed but design the system around the broader goal of efficient multimodal knowledge-intensive QA. Large language models (LLMs) achieve strong accuracy on medical licensing exams—GPT-4-base 86.1% on MedQA under 5-shot prompting ( 14 ) , Med-PaLM 2 86.5% on MedQA from a MedQA-specific instruction-finetuned variant ( 17 ) —but remain limited by static parametric knowledge and hallucination ( 7 ) ; clinical decision support ( 16 ) would require both grounded evidence and bounded latency. RAG ( 10 ) addresses the grounding side by retrieving from external corpora; in medicine, MedRAG ( 19 ) , Self-BioRAG ( 6 ) , i-MedRAG ( 20 ) , and RAG2 ( 18 ) have demonstrated substantial gains. The medical RAG baselines we compare against, however, share one assumption: retrieval over text. Biomedical PDFs are OCR’d or parsed into chunks, and only the extracted text is indexed. This systematically discards visual content—dosage tables, diagnostic flowcharts, anatomical diagrams—that frequently conveys information more effectively than its linearized form. Single-pass retrieval is also limited for multi-hop questions where mechanism, contraindications, and monitoring guidelines reside on distinct pages. We propose MedVRAG (Medical Visual RAG), which addresses both issues: (i) page-level multimodal retrieval indexes original 350K PMC document pages with a vision-language embedding model rather than text chunks; (ii) iterative reasoning with memory lets a VLM autonomously refine its query and accumulate findings across up to three rounds. To our knowledge MedVRAG is among the first multimodal RAG frameworks for biomedical multiple-choice QA operating on PMC page images with iterative VLM reasoning—general-domain page-image RAG ( 4 ) targets document VQA, and the medical RAG systems above operate on text. Contributions. (1) A two-stage retrieval pipeline combining ColQwen2.5 patch-level page embeddings with a sharded MapReduce LLM filter that scales to 350K biomedical pages at \sim 30 ms Stage-1 latency through an offline coarse-to-fine index ( C{=}8 centroids/page, ANN over centroids, exact two-way scoring on the top- R shortlist). (2) An iterative VLM reasoner with a structured memory bank that issues query refinements and accumulates per-round findings across up to three rounds; per-iteration latency \sim 15.9 s on 4 \times A100. (3) A controlled head-to-head comparison against same-backbone text-only RAG, no-RAG, and progressively-ablated variants on four standard medical QA benchmarks, with per-stage latency, per-iteration accuracy, and a four-class error taxonomy. 2 Related Work Medical RAG. MedRAG ( 19 ) systematically benchmarks RAG on five corpora and four retrievers; Self-BioRAG ( 6 ) adds self-reflection to decide when to retrieve; i-MedRAG ( 20 ) issues iterative follow-up queries; RAG2 ( 18 ) adds rationale-guided retrieval. All operate on text chunks. Multimodal page-image retrieval. ColPali ( 4 ) introduced multi-vector page-image retrieval over a vision-language backbone, with later variants (ColQwen2.5) replacing the encoder. Concurrent general-domain page-image RAG efforts explore document-VQA-style retrieval, but to our knowledge none targets medical benchmarks or couples retrieval with iterative reasoning over a memory bank. Vision-language models for medical content. LLaVA-Med ( 11 ) and Med-Flamingo ( 13 ) are biomedical VLMs trained for visual QA over images such as radiographs; Qwen2.5-VL ( 1 ) demonstrates strong general document and layout understanding that we leverage for biomedical pages without further fine-tuning. 3 Method Figure 1 : MedVRAG pipeline. A medical question is encoded by ColQwen2.5 and matched against a FAISS index over \sim 350K PMC pages; an LLM filters candidates by summary (sharded MapReduce); a VLM iteratively reasons over the highest-ranked 10 page images plus top-20 summaries from the N_{2}{=}100 filtered set, refining queries and accumulating findings in a memory bank across \leq 3 rounds. System overview. MedVRAG consists of four components (Fig. 1 ): (1) a page-level embedding and indexing module over the PMC corpus; (2) a two-stage retrieval pipeline (embedding similarity, then LLM filtering); (3) an iterative reasoning engine driven by a VLM; and (4) a memory bank that accumulates findings across rounds. We describe each component below. Knowledge base. We render each PMC OA article PDF page-by-page at 300 DPI; near-duplicate pages are dropped (cosine >0.97 ). The final corpus is \sim 350K pages from \sim 18K articles. Text overlap with each benchmark’s source articles is excluded by DOI/PMID; for MMLU-Med (no source DOIs) we additionally check verbatim question text appears in zero retained pages. Page-level embedding. Each page is encoded with the ColPali multi-vector page-image retriever ( 4 ) —specifically the ColQwen2.5 community variant that swaps PaliGemma for a Qwen2.5-VL backbone—into n patch-level vectors. We PCA-reduce these to d{=}128 (from 768) for index size; a short text summary ( \sim 120 tokens) is also generated offline by Qwen2.5-VL-7B for use in Stage 2. Stage 1 (embedding retrieval). Given a question q encoded into m token vectors, we score each page by a two-way late-interaction sum S(q,p)=\tfrac{1}{m}\sum_{i}\max_{j}q_{i}^{\top}v_{j}\;+\;\tfrac{1}{n}\sum_{j}\max_{i}q_{i}^{\top}v_{j} and return the top N_{1}{=}2{,}000 pages. A coarse-to-fine strategy ( C{=}8 centroids per page, ANN over the centroid index, then exact two-way scoring on the top candidates) keeps Stage 1 latency under 30 ms. Stage 2 (LLM filtering). Since N_{1}{\times}\bar{\ell}_{\rm sum}\approx 240 K tokens exceeds the 32K context, we use sharded MapReduce over Qwen3-30B-A3B: 8 parallel map calls each score B{=}256 summaries ( \sim 31K tokens) and emit top-25; one reduce call ranks the surviving 200 into the final N_{2}{=}100