MIRA: A Novel Framework for Fusing Modalities in Medical RAG Jinhong Wang, Tajamul Ashraf, Zongyan Han, Jorma Laaksonen, Rao Mohammad Anwer Abstract . Multimodal Large Language Models ( MLLM ) have significantly advanced AI-assisted medical diagnosis but often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation ( RAG ) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce Multimodal Intelligent Retrieval and Augmentation ( MIRA ) framework designed to optimize factual accuracy in MLLM . MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code, is released at https://github.com/mbzuai-oryx/MIRA . 1. Introduction Medical Visual Question Answering ( MedVQA ) plays a crucial role in AI-driven healthcare by enabling accurate diagnosis, disease detection, treatment optimization, and clinical decision-making ( Ben Abacha et al. 2021 ; Ben Abacha et al. 2019 ; Liu et al. 2023b ; Ben Abacha et al. 2021 ) . The integration of multimodal data, such as medical images and corresponding textual reports, presents unique challenges like domain shifts ( Nazi and Peng 2024 ) , limited labeled data ( Gu et al. 2021 ; Ashraf and Bashir 2025 ) , and modalities ( Lin et al. 2022 ) , etc, which require a balance between visual reasoning and textual coherence. Existing models, including LLaVA-Med ( Li et al. 2024 ) and Huatuo-Vision-7B ( Wang et al. 2023 ) , struggle with hallucinations, rigid reasoning, and unimodal biases, often misinterpreting subtle imaging findings and failing in open-ended queries. Although Multimodal Large Language Models ( MLLMs ) ( AlSaad et al. 2024 ; Demirhan and Zadrozny 2023 ; Moor et al. 2023 ) have shown resilience to internal knowledge that often leads to hallucinations and factual inaccuracies that pose risks to patient safety ( Park et al. 2024 ) . Representative analyses of these constraints are shown in Fig. 1 , and these issues compromise their reliability in medical use. And there’s calls for more reliable and precise models to perform MedVQA . To address this issue, Retrieval-Augmented Generation ( RAG ) ( Xia et al. 2024a ; Xia et al. 2024b ) has emerged as a promising solution by enriching model responses with verified external medical literature ( Xia et al. 2024c ) . This approach enhances factual grounding, reduces hallucinations, and improves the reliability of AI-driven clinical insights ( Xia et al. 2024a ) . Figure 1. Overview of key constraints in automatic medical question answering, highlighting the challenges related to factuality, domain specificity, and clinical relevance that must be addressed for reliable system performance. Despite its potential, RAG introduces two critical challenges in medical settings: (1) Retrieval imbalance , where the system disproportionately relies on specific sources, potentially overlooking relevant yet underrepresented data, leading to biased or incomplete information synthesis; and (2) Over-reliance , where the model excessively trusts the retrieved information without adequate verification, increasing the risk of propagating factual inaccuracies and medical misinformation. Recently, various methods ( Xia et al. 2024c ; Xia et al. 2024a ; Thawkar et al. 2023 ) have attempted to address these issues by incorporating retrieval calibration and preference tuning to mitigate over-reliance. While these strategies improve factual accuracy to some extent, they have notable limitations. Specifically, prior approaches are predominantly designed for radiology-related tasks and often struggle to generalize across diverse medical domains such as pathology, dermatology, and cross-specialty diagnostic reasoning as shown in Fig. 1 . Prior approaches ( Xia et al. 2024a ; Xia et al. 2024c ) decouple retrieval from reasoning, leading to fragmented outputs without dynamic modality fusion or real-time evidence integration. Furthermore, these frameworks typically treat vision and text as unified modalities, neglecting the need for modality-specific processing and adaptive fusion. More critically, they rely on static databases for retrieval, lacking mechanisms for real-time evidence integration, which is essential for ensuring up-to-date medical knowledge. These limitations hinder their effectiveness in complex clinical scenarios requiring dynamic, multimodal reasoning. To address the challenges in multimodal medical reasoning, we introduce Multimodal Intelligent Retrieval and Augmentation ( MIRA ), a novel RAG framework that enhances clinical decision-making through structured retrieval, validation, and reasoning. Our approach integrates two key innovations: (1) an RTRA pipeline that employs iterative “rethink-rearrange” cycles to refine outputs by validating them against retrieved multimodal medical knowledge, ensuring factual accuracy and interpretability, and (2) a Multimodal RAG (MRAG) strategy that combines offline (prebuilt medical databases) and online (real-time web sources) retrieval to ground responses in dynamically updated clinical evidence. The MRAG pipeline jointly retrieves textual and visual information from sources like NIH ChestX-ray14 and real-time medical guidelines, while the RTRA cycle ensures clinical consistency through a three-stage validation process: (i) initial response generation, (ii) self-critique via Chain-of-Thought (CoT), and (iii) final answer refinement. Our vision-language backbone, built on CLIP-ViT-Large and SigLIP , is trained on 500K medical image-text pairs, enabling robust multimodal alignment. Our contributions include a novel Vision-Language Model ( VLM ) extending LLaVA with dynamic modality fusion, the first multimodal RAG framework for MedVQA , and state-of-the-art performance across multiple benchmarks, achieving 9 times faster inference than 72B models, making it efficient for real-world clinical applications. Our main contributions are as follows: • We introduce MIRA , the first retrieval-augmented generation (RAG) framework that seamlessly integrates structured multimodal retrieval with adaptive reasoning , surpassing existing methods in clinical decision-making accuracy and efficiency. • Unlike static retrieval paradigms, our context-rethink module employs an iterative “rethink-rearrange” cycle for dynamic k -selection , ensuring precision in evidence selection, while Chain-of-Thought (CoT) reasoning enhances factual consistency in MedVQA. • Our architecture pioneers a dual-pathway retrieval mechanism with specialized vision and language encoders, enabling fine-grained image-text alignment . The integration of a curated citation module ensures interpretability, setting a new standard for transparency in medical AI. • Our model establishes new state-of-the-art (SOTA) results on MedVQA, significantly reducing factual errors through online search augmentation and adaptive CoT-based verification . With 9× faster inference than 72B models, it sets a new benchmark for real-time, high-precision medical reasoning. 2. Related Work Multimodal Large Language Models (MLLMs): Traditional LLMs are limited to textual input, restricting their ability to process and understand multimodal information. Multimodal LLMs extend their capabilities by incorporating additional modalities such as images, videos, and audio, enabling a richer and more holistic understanding of input data ( Thirunavukarasu et al. 2023 ; Zhang et al. 2023b ; AlSaad et al. 2024 ; Peng et al. 2023 ) . A typical MLLM architecture consists of three primary components: a modality encoder, a pre-trained LLM, and a modality generator, interconnected through learned projectors to facilitate seamless information transformation across different modalities ( Demirhan and Zadrozny 2023 ; Moor et al. 2023 ) . These architectures have been successfully applied in various domains, including healthcare, where multimodal integration has enhanced clinical decision support, diagnosis, and medical research applications. Medical Large Vision-Language Models (Med-LVLMs): Medical Large Vision-Language Models (Med-LVLMs) are specialized architectures designed to process and interpret medical images alongside textual data. These models typically integrate an LLM with a dedicated vision module that extracts relevant information from medical images and converts it into a representation compatible with the LLM’s processing capabilities. Given a medical image x_{v} and a clinical query x_{t} , the combined input is represented as x=(x_{v},x_{t}) . The model then autoregressively predicts the probability distribution of the text output y based on the multimodal input. Med-LVLMs have been applied in tasks such as radiology report generation, pathology image analysis, and multi-modal medical question answering ( Zhang et al. 2023a ; Peng et al. 2023 ; Moor et al. 2023 ) . Unlike these, MIRA integrates iterative reasoning (RTRA) and multimodal retrieval-augmented generation (MRAG) to ensure that outputs are validated against retrieved evidence, addressing the issue of hallucination and improving accountability. By introducing a dynamic multimodal RAG pipeline that adapts to real-time evidence, MIRA enhances both the versatility and precision of medical vision-language models. Retrieval-Augmented Generation (RAG): Retrieval-Augmented Generation (RAG) has emerged as an effective paradigm to address LLMs’ limitations in domain-specific question answering ( Xia et al. 2024a ; Thawkar et al. 2023 ; Xia et al. 2024c ; Xia et al. 2024d ) . By combining an LLM with an external knowledge retriever, RAG allows models to access relevant information dynamically, improving accuracy and factual consistency in generated responses. This approach has been successfully applied to tasks such as question answering, fact verification, and knowledge-intensive text generation, achieving state-of-the-art performance in open-domain scenarios ( Ben Abacha et al. 2021 ; Ben Abacha et al. 2019 ; Liu et al. 2023b ) . Recent advances in multimodal RAG have further expanded its capabilities by incorporating images, videos, and structured data into retrieval mechanisms ( Chen et al. 2022 ; Lin et al. 2023 ; Karpukhin et al. 2020 ) . Multimodal RAG frameworks have been explored in medical applications, where the integration of medical text with radiological and histopathological images has improved diagnostic accuracy and clinical decision support ( Xia et al. 2024c ; Lin et al. 2022 ; Deng et al. 2023 ) . While existing works like MMed-RAG ( Xia et al. 2024a ) and RULE ( Xia et al. 2024c ) address factuality challenges, they rely on static retrieval mechanisms. MIRA extends this paradigm by incorporating online retrieval and iterative reasoning, ensuring that outputs are dynamically refined against cross-modal evidence, thereby enhancing adaptability and reliability in real-world clinical scenarios. Evaluation of Retrieval-Augmented Generation (RAG): Evaluating RAG models involves evaluating both the retrieval and the generation components. Frameworks like RAGAs ( Es et al. 2023 ) propose metrics such as Faithfulness, Answer Relevance, and Context Relevance to provide a comprehensive evaluation. Benchmarking datasets such as TruthfulQA ( Hager et al. 2024 ) and MMLU ( Devlin et al. 2019 ) have been widely used to measure retrieval performance and factual accuracy. Although automated evaluation methods offer scalability and alignment with human judgments, they remain approximations. Human annotations remain the gold standard, particularly in domain-specific applications like medical question answering, where accuracy is critical. For multimodal evaluation, GPT-4V has shown a strong alignment with human assessments in vision language tasks ( Zhang et al. 2023b ) , further highlighting the potential of large multimodal models in knowledge-intensive applications. Despite these advancements, existing RAG-based evaluation frameworks often struggle with dynamic multimodal reasoning and ensuring real-time validation of retrieved knowledge. MIRA addresses this limitation by integrating iterative reasoning with online retrieval, allowing the model to refine outputs against cross-modal evidence continuously. This not only improves factual accuracy and faithfulness but also enhances the adaptability of the system to real-world clinical settings. By leveraging retrieval-aware generation and multimodal verification, our approach sets a new benchmark in robust, evidence-grounded medical AI. 3. Methodology In this section, we introduce MIRA, a novel framework designed to enhance multimodal reasoning in medical applications, particularly MedVQA. Unlike conventional RAG systems, MIRA optimizes the retrieval, fusion, and generation of both textual and visual data to improve factual accuracy and interoperability as illustrated in Fig. 2 . Built upon the LLaVA architecture, MIRA integrates a high-resolution vision encoder for detailed medical image understanding and introduces dynamic modality fusion to adaptively balance textual and visual contributions based on the query type. Furthermore, it employs a multimodal RAG pipeline, retrieving both text and images from structured and unstructured medical databases using a joint embedding space. Finally, MIRA enhances reasoning through a Rethink-Rearrange (RTRA) mechanism, leveraging Chain-of-Thought (CoT) reasoning for iterative refinement, ensuring responses are clinically accurate and well-supported by retrieved evidence. We will now discuss each component of MIRA in detail. 3.1. Data preprocessing Multimodal Input. The MIRA framework receives two primary inputs: a textual query Q and an image I . These inputs are processed through separate pathways—textual and visual—and then combined in a manner that allows for sophisticated multimodal reasoning and retrieval-augmented generation (RAG). The inputs are transformed into high-dimensional embeddings through the use of a text and vision encoder. The embeddings serve as the basis for data retrieval and fusion during the reasoning process. Query Rewrite Module. Before the input query Q is passed to the text encoder, it undergoes an initial transformation via the Query Rewrite module. This module aims to enhance the semantic alignment and contextual clarity of the input query. Specifically, given an input query Q , the module applies a learned function \mathcal{R}(\cdot) to rewrite and refine the query Q , resulting in a modified query Q^{\prime} . The rewriting function can be expressed as: Q^{\prime}=\mathcal{R}(Q),\quad\mathcal{R}:\mathbb{R}^{n_{\text{words}}}\rightarrow\mathbb{R}^{n^{\prime}_{\text{words}}}, where \mathcal{R}(\cdot) is a transformation that adjusts the query’s linguistic structure while preserving its semantic meaning. The rewritten query Q^{\prime} serves as an enriched representation, ensuring it is aligned with the task requirements. Text Encoder. Following the query refinement in the Query Rewrite module, the enhanced query Q^{\prime} is passed to the text encoder \mathcal{E}_{\text{text}} . This encoder operates as a learned function that maps the refined query Q^{\prime} to a high-dimensional semantic embedding E_{\text{text}}^{\prime}\in\mathbb{R}^{d_{\text{text}}} , where d_{\text{text}} is the dimensionality of the embedding space. This process effectively captures the deeper semantic structure and contextual information of the query, facilitating its integration into the multimodal reasoning process. The text encoder can be formulated as a transformation function \mathcal{E}_{\text{text}}:\mathbb{R}^{n^{\prime}_{\text{words}}}\rightarrow\mathbb{R}^{d_{\text{text}}} , which takes the rewritten query Q^{\prime}\in\mathbb{R}^{n^{\prime}_{\text{words}}} as input and produces the embedding E_{\text{text}}^{\prime}\in\mathbb{R}^{d_{\text{text}}} : E_{\text{text}}^{\prime}=\mathcal{E}_{\text{text}}(Q^{\prime}),\quad E_{\text{text}}^{\prime}\in\mathbb{R}^{d_{\text{text}}}. Here, n^{\prime}_{\text{words}} denotes the number of words in the rewritten query Q^{\prime} , and d_{\text{text}} represents the dimensionality of the learned embedding space. The encoder \mathcal{E}_{\text{text}} is a pre-trained transformer model optimized to capture both syntactic and semantic features of the query. The resulting text embedding E_{\text{text}}^{\prime} serves as a rich, high-dimensional representation of the query, capturing its meaning in a manner that is agnostic to the specific wording of the query. This embedding is essential for understanding the query in the context of a multimodal task, where it will be fused with corresponding visual information to perform retrieval or reasoning tasks in the MIRA framework. Figure 2. Overview of the MIRA (Multimodal Intelligent Retrieval and Augmentation) pipeline. The system integrates image and text-based retrieval to enhance the generation process. Vision Encoder. Parallel to the textual processing pipeline, the input image I\in\mathbb{R}^{H\times W\times 3} is passed through the vision encoder \mathcal{E}_{\text{vision}} , which is designed to extract rich visual features from the raw image. The encoder \mathcal{E}_{\text{vision}} maps the input image to a high-dimensional feature space E_{\text{image}}^{\prime}\in\mathbb{R}^{d_{\text{image}}^{\prime}} , where d_{\text{image}}^{\prime} is the dimensionality of the image embedding space. This transformation is mathematically defined as: E_{\text{image}}^{\prime}=\mathcal{E}_{\text{vision}}(I),\quad E_{\text{image}}^{\prime}\in\mathbb{R}^{d_{\text{image}}^{\prime}}. The vision encoder \mathcal{E}_{\text{vision}} utilizes the CLIP-ViT-Large model ( Radford et al. 2021 ) , which is based on the Vision Transformer (ViT) framework. The model is optimized to capture complex spatial patterns, textures, and object relationships within the image, which are essential for tasks that require vision and language understanding structured as follows: E_{\text{image}}^{\prime}=f_{\text{ViT}}(I),\quad f_{\text{ViT}}:\mathbb{R}^{H\times W\times 3}\rightarrow\mathbb{R}^{d_{\text{image}}^{\prime}}, where f_{\text{ViT}} denotes the Vision Transformer-based function applied to the input image I . The resulting image embedding E_{\text{image}}^{\prime} serves as a high-dimensional representation of the image content, encoding important visual information such as object features, textures, and spatial relationships. MIRA further enhances the LLaVA framework by improving the precision of the vision encoder. The CLIP-ViT-Large model processes high-resolution images I^{\prime}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times 3} , where H^{\prime}>H and W^{\prime}>W , allowing the model to capture finer visual details. The enhanced vision encoder now generates a refined image embedding E^{\prime}_{\text{image}}\in\mathbb{R}^{d_{\text{image}}^{\prime}} , which improves the model’s ability to handle complex visual reasoning tasks. Mathematically, the operation is represented as: E^{\prime}_{\text{image}}=\mathcal{E}_{\text{vision}}(I^{\prime}),\quad E^{\prime}_{\text{image}}\in\mathbb{R}^{d_{\text{image}}^{\prime}}. Thus, the upgraded vision encoder with higher resolution images results in a more expressive visual feature space, better suited for challenging multimodal tasks that require detailed image understanding. 3.2. Multimodal Database Data Retrieval. Once the textual and visual embeddings E_{\text{text}}^{\prime} and E_{\text{image}}^{\prime} are computed, MIRA employs a data retrieval mechanism to search for the most relevant multimodal entries from both offline and online databases. The two primary sources of data retrieval are: 1. Offline Database: A collection of pre-indexed text and image embeddings, denoted as \mathcal{D}_{\text{offline}}=\{(T_{\text{offline},i},I_{\text{offline},i})\} , where each pair (T_{\text{offline},i},I_{\text{offline},i}) represents a text and its associated image in the offline database. 2. Online API Database: A set of real-time data entries retrieved via an API, denoted as \mathcal{D}_{\text{API}}=\{(T_{\text{API},i},I_{\text{API},i})\} , where each pair (T_{\text{API},i},I_{\text{API},i}) is a dynamically fetched text-image pair. The retrieval function \text{Retrieve}(E_{\text{text}}^{\prime},E_{\text{image}}^{\prime}) computes the similarity between the query embeddings E_{\text{text}}^{\prime} and E_{\text{image}}^{\prime} , and the embeddings stored in both the offline and online databases. This is achieved using vector matching algorithms, such as cosine similarity, to measure the closeness between the query embeddings and the stored entries in the respective databases: e_{\text{retrieved},j}=\text{Retrieve}(E_{\text{text}}^{\prime},E_{\text{image}}^{\prime},\mathcal{D}_{\text{offline}},\mathcal{D}_{\text{API}}), where e_{\text{retrieved},j}\in\mathcal{D}_{\text{offline}}\cup\mathcal{D}_{\text{API}} represents the j -th retrieved entry from either database. Next, the top k relevant entries, consisting of both text and image pairs, are retrieved. The number k represents the number of top entries to retrieve, where typically the retrieval involves: - k_{\text{text}}=3 retrieved text chunks, denoted as T_{\text{retr},1},T_{\text{retr},2},\dots,T_{\text{retr},k_{\text{text}}} , - k_{\text{image}}=2 retrieved images, denoted as I_{\text{retr},1},I_{\text{retr},2} . In addition to these, the API also retrieves a dynamically fetched text paragraph T_{\text{API\_retrieved}} , which provides an additional source of textual data for the fusion stage. Once the relevant entries are retrieved, the fusion process integrates the textual and visual embeddings from both the query and the retrieved database entries. The fusion of the query and retrieved embeddings is performed using a dynamic attention mechanism, which adaptively assigns different weights to the contributions of the text and image embeddings based on their relevance to the query. Fusion of Retrieved Embeddings. After retrieving the top-k relevant entries from the multimodal database, MIRA employs a dynamic attention mechanism to fuse the image and text embeddings. The attention mechanism adapts to the nature of the query, adjusting the relative contributions of each modality to ensure the most pertinent features are prioritized. The final fused representation E_{\text{final}} is computed as: E_{\text{final}}=\text{Att}(E_{\text{image}},E_{\text{text}}),\quad E_{\text{final}}\in\mathbb{R}^{d_{\text{final}}}, where Att() represents the attention mechanism, which computes the weighted combination of the visual ( E_{\text{image}} ) and textual ( E_{\text{text}} ) embeddings. The resulting E_{\text{final}} is a multimodal representation, where the contributions of text and image are adaptively balanced based on the query context. This dynamic fusion enables MIRA to prioritize either modality based on the specific nature of the query. For example, if the query is more text-centric, the model gives higher weight to the textual representation, while for image-centric queries, the visual features are emphasized. Alignment layer and Modality Fusion. Once the image and text embeddings are computed, MIRA employs an adaptive attention mechanism to align these modalities into a shared multimodal space. Unlike other frameworks, such as LLaVA, which utilize simple linear transformations for alignment, MIRA’s attention-based fusion mechanism allows for dynamic adjustment of the weight given to each modality. To align the image and text representations in a common multimodal space, MIRA computes the final multimodal embedding E_{\text{final}} as a weighted combination of the image and text embeddings. This is formalized as: E_{\text{final}}=\text{Att}(E_{\text{image}},E_{\text{text}}),\quad E_{\text{final}}\in\mathbb{R}^{d_{\text{final}}}, where \text{Att}(\cdot) is the attention function that computes the final fused representation by attending to both the image and text features. The fusion ensures that the multimodal representation captures the relevant features from both modalities, which are contextually weighted based on the query. In image-centric queries (e.g., "Describe this chest X-ray"), the attention mechanism increases the weight on the image representation E_{\text{image}} , while in text-centric queries (e.g., "What is the symptom of this disease?"), the model prioritizes the text representation E_{\text{text}} . Modality-Specific Query Processing. Given a query Q and an associated image I , the model determines the most relevant modality to focus on based on the nature of the query. This is achieved through an attention function that dynamically selects the contribution from each modality. The fusion of the image and text embeddings is adjusted using a learnable parameter \alpha , which defines the balance between the two modalities. The final multimodal representation is computed as: E_{\text{final}}=\alpha\cdot\text{Att}(E_{\text{image}})+(1-\alpha)\cdot\text{Att}(E_{\text{text}}), where \alpha\in[0,1] is a learnable parameter that determines the weighting of the image and text features. When \alpha is close to 1, the model focuses more on the image, while if \alpha is close to 0, the model prioritizes the text features. Finally, the processed input now consists of the following multimodal entries: Type Description I_{\text{original}} Original Image T_{\text{original}} Original Text \{T_{\text{retrieved},1},T_{\text{retrieved},2},T_{\text{retrieved},3}\} Retrieved Text Chunks \{I_{\text{retrieved},1},I_{\text{retrieved},2}\} Retrieved Images T_{\text{API\_retrieved}} API Retrieved Text Paragraph Table 1. Structured Input Components for Multimodal RAG This combined set forms the new user input for the subsequent reasoning steps in the MIRA framework, as shown in Table 1 . 3.3. Multimodal Medical RAG RAG Pipeline. The Medical MRAG Module is designed to process diverse medical data, integrating multimodal information to enhance diagnostic accuracy and provide contextually relevant insights. The core of the retrieval mechanism is powered by a high-performance vector matching algorithm utilizing the Faiss-CUDA framework. The knowledge base \mathcal{K} consists of vectorized representations of both text and images, enabling efficient similarity search. Given a query q , the system retrieves a relevant set of documents \mathcal{D} from \mathcal{K} , represented as: (1) \mathcal{D}=\{d_{1},d_{2},\dots,d_{n}\},\quad d_{i}\in\mathcal{K}. Retrieval-Augmented Generation (RAG) plays a central role in refining the generative process. Once the relevant documents are retrieved, they are conditioned into the generation model to ensure domain-specific accuracy. The probability distribution of the generated medical response y is computed as: (2) p(y|q,\mathcal{D})=\prod_{t=1}^{T}p(y_{t}|y_{}\}. Here, Rearrange truncates extraneous data while ensuring at least one textual and visual reference remains relevant. The extracted \mathcal{D} forms the basis for initial CoT generation. Leveraging Qwen2.5-VL models, vision-text embeddings f_{v}(q_{i}) and f_{t}(q_{t}) are jointly processed, refining knowledge selection through multimodal alignment: (4) \hat{\mathcal{D}}=\operatorname{argmax}_{d\in\mathcal{D}}\langle f_{v}(q_{i})+f_{t}(q_{t}),f(d)\rangle. Following knowledge retrieval, the initial CoT response y_{0} is generated: (5) y_{0}=p(y|q,\hat{\mathcal{D}}). To ensure coherence and factual correctness, an iterative refinement stage Rethink is applied, where an expert model reassesses y_{0} against the retrieved knowledge \hat{\mathcal{D}} , producing an optimized response y^{*} : (6) y^{*}=\operatorname{Refine}(y_{0},\hat{\mathcal{D}},\mathcal{K}). The entire process is reinforced using a reward function R(z,y) , which evaluates logical consistency and medical accuracy: (7) R(z,y)=\lambda_{1}\cdot\text{FactualScore}(z,\mathcal{K})+\lambda_{2}\cdot\text{CoherenceScore}(y,z), where \lambda_{1},\lambda_{2} control the balance between factual precision and coherence. Model parameters \theta are updated via policy gradient optimization: (8) \nabla_{\theta}J(\theta)=\mathbb{E}_{z\sim p_{\theta}}\left[R(z,y)\nabla_{\theta}\log p_{\theta}(z|q)\right]. This reinforcement learning (RL)-guided CoT refinement ensures structured, high-quality medical reasoning. The training dataset is derived from LLaVA-Med’s 500k alignment set, with an additional 50k high-quality CoT fine-tuning instances curated from PubMedVision and VQA-Med. 3.4. Optimization 4. Optimization with Reinforcement Learning and Chain-of-Thought Reasoning We optimize our multimodal retrieval-augmented generation (RAG) pipeline through a two-step training strategy combining supervised learning and reinforcement learning (RL), enhanced with Chain-of-Thought (CoT) reasoning for improved medical question answering. Two-Step Training Strategy • Step 1: Supervised Pretraining. The model is initially trained using annotated medical data in multiple field with cross-entropy loss. During this stage, only the multilayer perceptron projector connecting Vision Tower and LLM is tuned. • Step 2: Reinforcement Fine-Tuning (RFT). The model is then refined using pre-built RTRA-format data to stimulate performance, as we are building roll-out data like RL. This is similar to building thinking-based models. During this stage, MLP and LLM parameters are unfrozen for training. All training stages are using cross-entropy loss for model performance optimization. (9) \mathcal{L}_{\text{CE}}=-\sum_{t=1}^{T}\log p_{\theta}(y_{t}|y_{0.5 ) LLaVA-1.5 Zeroshot ( Liu et al. 2023a ) 0.45(578) 0.02(20) BiomedGPT ( Zhang et al. 2023b ) 0.58(752) 0.21(201) LLaVA-Med-7B ( Li et al. 2023 ) 0.54(700) 0.19(183) Huatuo-Vision-7B ( Chen et al. 2024 ) 0.63(823) 0.24(231) XrayGPT ( Thawkar et al. 2023 ) 0.59(769) 0.23(222) Qwen2.5-VL-72B 0.69(892) 0.47(457) OpenAI o1 0.69(893) 0.49(480) Kimi-K1.5 ( Team et al. 2025 ) 0.59(767) 0.35(340) MIRA (ours) 0.66(861) 0.39(385) Table 3. Performance comparison of PMC-VQA question set correctness analysis MIRA balances efficiency and effectiveness, performing well in medical VQA without the massive scale of models like Qwen2.5-VL-72B. Its strong results—despite no pretraining on PMC-VQA—indicate genuine generalization. Fine-tuning on edge cases or integrating medical knowledge bases could further narrow the performance gap. 5.6. Ablation Study Preference Analysis. To further assess the contribution of specific components within MIRA-8B, an ablation study was conducted based on preference analysis against OpenAI o1 and Kimi-K1.5 frameworks, as shown in Table 4 . The results reveal that MIRA is preferred over OpenAI o1 in 46% of cases (92 out of 200), suggesting that its design offers a balanced performance for PubMedVision QA pairs. However, when compared to Kimi-K1.5, MIRA-8B is favored more frequently, with a preference ratio of 59.5% (119 out of 200), indicating that the framework’s efficiency and multimodal reasoning capabilities provide a substantial advantage in this context. This ablation highlights the critical components of MIRA-8B that contribute to its generalization ability and its effectiveness across diverse PubMedVision tasks. Comparison OpenAI o1 Kimi-K1.5 ( Team et al. 2025 ) MIRA (ours) 0.460(92) 0.595(119) Table 4. Preference analysis on 200 sampled PubMedVision QA pairs shows that MIRA is favored by the judge model (Qwen2.5-VL-72B) in most cases. Component Conv (1297 in total) Details (971 in total, c_{s}>0.5 ) No Online (Local only) 0.49(639) 0.11(108) No Offline (Search only) 0.55(714) 0.23(222) Text Only 0.46(601) 0.19(181) Vision Only 0.50(654) 0.20(190) Table 5. Ablation study showing performance under various RAG settings. “Conv” and “Details” represent the number of correct/favorited responses, which is the same setting as previous experiment in PMC-VQA. Attention Visualization. To better understand how the framework utilizes retrieved information during response generation, we visualize the attention distribution over the input sources while decoding specific output tokens (highlighted in yellow) in Fig 3 . The attention map shows when inferencing the tokens highlighted in yellow, the model’s attention distribution in the user input’s components, respectively, input iamges, query, and each slice of RAG result. The map only calculates attention in user input, while normalizing all these attention components to a sum of 1. Our analysis shows that the framework’s attention is initially focused on the visual input, particularly around regions describing "chest X-ray showing changes consistent with rickets" and the anatomical location ("clavicular level"). As the framework proceeds to generate its response, attention shifts toward textual cues in the retrieved documents—especially phrases such as "atrial septal defect’s mechanisms", "if untreated", and "shunt". These elements are semantically aligned with the generated conclusion involving a broken shunt and its clinical implications. Note that the framework predominantly focuses its attention on the response-side context during decoding, leading to an uneven distribution across inputs. For this visualization, we normalize the attention weights across source tokens to highlight their relative importance in guiding generation, based on training-time behavior. Brief spikes on unrelated regions suggest room for improving alignment for better coherence and factual accuracy in complex medical cases. Figure 3. Visualization of attention distribution across all slices of input sequences, emphasizing the model’s focus on critical tokens. This highlights how the model has learned selectively attends to important parts of the input, guiding the generation process for more accurate and contextually relevant responses. Component Analysis To understand the contribution of different RAG components, we perform an ablation study under various settings: disabling online retrieval (local only), disabling offline DDGS memory, and isolating modalities (text-only and vision-only). As shown in Table 5 , removing the online component leads to a noticeable drop in performance, particularly in detail-level understanding, indicating the importance of up-to-date retrieval. Similarly, the absence of offline DDGS memory reduces overall grounding and specificity. Interestingly, vision-only settings outperform text-only in both conversational and detailed responses, highlighting the framework’s strength in visual reasoning. However, the best performance is achieved when both retrieval pathways and modalities are integrated. 6. Conclusion We present MIRA, a novel approach to improving the factuality and reliability of Multimodal Large Language Models (Multimodal-LLMs) in medical Retrieval-Augmented Generation (RAG). MIRA addresses two key challenges: mitigating factuality risks by rearranging retrieved contexts for relevance and reliability, and reducing excessive dependence on external data to prevent overfitting. By balancing intrinsic medical knowledge with external insights, MIRA generates more accurate and contextually grounded medical responses using our RtRa framework. Our experiments and ablation studies demonstrate significant improvements in factuality, clarity, and robustness, paving the way for more reliable AI systems in healthcare settings. Future work may focus on model fine-tuning and integration with specialized medical knowledge bases to increase performance in even more complex medical scenarios. Appendix Table 6. Hyperparameter Settings for LLaVA Training Hyperparameter Value version(llava_prompt_version) RTRA-Format-v1 mm_tunable_parts mm_mlp_adapter in Pretraining, mm_mlp_adapter, mm_language_model in RFT deepspeed_mode zero_2_offload in both pretraining and finetuning mm_vision_tower_lr 2e-6 vision_tower google/siglip-so400m-patch14-384 mm_projector_type mlp2x_gelu mm_vision_select_layer -2 rag_enabled True rag_tokenizer all-MiniLM-L6-v2 rag_topk 3 query_rewrite_enabled True query_rewrite_model Qwen-2-5-8B image_aspect_ratio anyres image_grid_pinpoints [(384, 768), (768, 384), (768, 768), (1152, 384), (384, 1152)] mm_patch_merge_type spatial_unpad precision bfloat16 Pretrain_num_train_epochs 1 Pretrain_per_device_train_batch_size 4 Pretrain_per_device_eval_batch_size 1 Pretrain_gradient_accumulation_steps 16 RFT_num_train_epochs 2 RFT_per_device_train_batch_size 1 RFT_per_device_eval_batch_size 1 RFT_gradient_accumulation_steps 32 learning_rate 1e-5 in pretraining, 2e-6 in finetuning weight_decay 0.0 warmup_ratio 0.03 lr_scheduler_type cosine tf32 True model_max_length 32768 torch_compile_backend inductor dataloader_drop_last True attn_implementation sdpa Notes • The experiments were conducted using 4 \times A100 80GB GPUs in RFT stage, and 4 \times A100 40GB GPUs in Pretraining stage. • For query rewrite, zero-shot models could be used for summarizing user inputs. Qwen-2-5-8B is recommended, and Mistral-small-22B is also tried during development. The external model is hosted using Ollama . Additional Dataset Visualizations RTRA DataSlice Example 1 Human Query: Is it a malignant lesion? MIRA Response: Rearrange: Selected: (1) Cryptococcus neoformans and C. gattii often cause invasive lung infections in adults with HIV or other weakened immune systems. It can look like lung cancer or tuberculosis on scans. (2) CT scan showing cryptogenic organizing pneumonia (biopsy-proven). Initial: The lesion identified in the CT scan, initially diagnosed as cryptogenic organizing pneumonia, is not definitively a malignant lesion. Cryptogenic organizing pneumonia is a type of lung inflammation that can mimic the appearance of cancer on imaging studies, but it is not a form of cancer. However, to rule out malignancy, additional tests such as a biopsy or further imaging may be necessary. Rethink: (1) I’ll confirm the initial diagnosis and the nature of the lesion, emphasizing that cryptogenic organizing pneumonia (COP) is a condition that can mimic the appearance of a malignant lesion on imaging but is not cancerous. (2) I’ll clarify that while COP is the initial diagnosis, the possibility of a malignant lesion cannot be entirely ruled out without further diagnostic tests, such as a biopsy or additional imaging studies. (3) I’ll mention that the differential diagnosis should also consider other conditions that can present similarly, such as pulmonary cryptococcosis, especially in immunocompromised individuals, and highlight the importance of a comprehensive evaluation. Final: The lesion was initially diagnosed as cryptogenic organizing pneumonia, a non-malignant lung condition. However, a definitive diagnosis would require further evaluation and tests, such as a biopsy, to confirm the nature of the lesion and rule out any malignancy. Additionally, conditions like pulmonary cryptococcosis, which can also mimic cancer on imaging, should be considered, especially in immunocompromised individuals. It’s always best to consult with a healthcare professional for a thorough assessment and accurate diagnosis. RAG Sequence For the Query: Nonspecific interstitial pneumonia’s Diagnosis: Diagnosis is made by reviewing patient history, imaging, lung function tests, and sometimes a surgical lung biopsy. Lung biopsies show two patterns: cellular (with inflammation but minimal fibrosis) and fibrosing (with interstitial fibrosis and varying levels of inflammation). Both patterns are uniform and lack the prominent fibroblastic foci found in other types of idiopathic interstitial pneumonia. Cryptococcosis’s Pulmonary cryptococcosis: Cryptococcus neoformans and C. gattii often cause invasive lung infections in adults with HIV or other weakened immune systems. While healthy adults may get it too, they usually have mild symptoms and don’t need treatment. However, these infections can spread to the brain, especially in immunocompromised individuals. Pulmonary cryptococcosis is common worldwide but often underdiagnosed due to limitations in testing. It can look like lung cancer or tuberculosis on scans. The CrAg test and blood cultures are usually negative unless the infection has spread. If not treated, pulmonary cryptococcosis can make cryptococcal meningitis worse. C: Cryptococcal meningitis’s Pulmonary cryptococcosis: Cryptococcus (both C. neoformans and C. gattii) commonly causes lung infections in adults with HIV or other weakened immune systems, but it can also affect healthy adults with milder symptoms. While these infections may not need treatment, careful monitoring is important. In some cases, the infection can spread to the brain and nervous system, especially in those with weakened immune systems. Pulmonary cryptococcosis has a worldwide presence but is often underdiagnosed due to diagnostic limitations. It typically presents as lung nodules that can mimic cancer, tuberculosis (TB), or other fungal infections. However, tests like blood cultures and the Cryptococcal antigen test are usually negative unless the infection has spread. Additionally, if not managed properly, this condition can worsen the prognosis of cryptococcal meningitis. Online RAG: hamburger line 1 hamburger line 2 hamburger line 3 Close Home (https://www.belmarrahealth.com/) » Brain Function (https://www.belmarrahealth.com/brain-function/) » Encephalomalacia: Definition, causes, types, symptoms, and treatment Encephalomalacia is a type of serious brain damage that results in the softening or loss of brain tissue. Causes of encephalomalacia are often linked to inflammation or hemorrhages that are a consequence of being afflicted by cerebral infarction, cerebral ischemia (https://www.belmarrahealth.com/types-cerebral-ischemia-symptoms-prevention-treatment-tips/) , infection, craniocerebral trauma, or another injury. The term, encephalomalacia, is often used upon gross pathological inspection, as it is much easier to better describe blurred cortical margins and decreased consistency of brain tissue, or brain softening. Advertisement [cky_video_placeholder_title] Softening may be seen in a localized area or be widespread, depending on the underlying cause. It can affect virtually any part of the brain, possibly damaging tissues of the frontal, occipital, parietal, and temporal lobe. This can present with several motor and sensory deficits. Considered the most common cause of encephalomalacia, strokes occur due to the interrupted blood supply to the brain either through a lack of blood supply to brain tissue or due to hemorrhage of a brain blood vessel. Oxygenated blood is vital for the function and maintenance of brain cells, which will become damaged or die if the blood supply is not promptly restored. Occurs when the flow of blood in the brain has become interrupted. This may be due to abnormal brain swelling or the removal of brain tumors resulting in brain damage. Damage to brain tissue due to conditions such as a stroke can result in the formation of scar tissue. This area of tissue undergoes contractions and eventually forms encephalomalacia within the brain Blunt force trauma to the skull will lead to contusions and possibly brain damage if the force is strong enough. Other forms of penetrating trauma, like from a knife or bullet to the skull, may also lead to the development of encephalomalacia Encephalomalacia may also be the result of: Also known as cerebrocortical necrosis (CCN), this neurological disease is caused by the disruption of thiamine production. Thiamine is a B vitamin (B1) and a key chemical in glucose metabolism. If a person were to become deficient in thiamine, significant side effects are likely to develop. The most threatening of which is the inhibition of neurological activity. Polioencephalomalacia will result in damage to the gray matter of the central nervous system, where the majority of information processing. • Image 2: CT of the chest demonstrating right-sided pneumonia (left side of the image). • Image 3: Cavitating pneumonia due to MRSA as seen on a CT scan. • Image 4: CT scan showing cryptogenic organizing pneumonia (biopsy-proven). • Image 5: MRI of Brain and MRA Brain showing encephalomalacia with extensive changes. • Image 6: Not retrieved. • Image 7: Not retrieved. RAG Image 2 RAG Image 3 RAG Image 4 Online RAGImg 5 Input Image (User-Provided) RTRA DataSlice Example 2 Human Query: What is the image showing? MIRA Response: Rearrange: Selected: (1) Brachydactyly type D: Brachydactyly type D, also known as short thumb or clubbed thumb, is a genetic trait where the thumb appears shorter and wider with a broader nail bed than usual. The distal phalanx of the affected thumb is approximately two-thirds the length of a normal thumb. It is quite common, affecting about 2–3% of the population, and is associated with the HOXD13 gene on chromosome 2q31.1. (2) Merkel cell carcinoma. Gross pathology specimen. Initial: The image is showing a gross pathology specimen of Merkel cell carcinoma. Rethink: (1) I’ll confirm the accuracy of the initial answer by verifying that the image indeed shows a gross pathology specimen of Merkel cell carcinoma. (2) I’ll enhance the initial answer by providing a brief description of what Merkel cell carcinoma is, including its clinical significance and common presentation. (3) I’ll ensure the answer is concise and directly addresses the question, while also offering enough context for the user to understand the significance of the image. Final: The image is showing a gross pathology specimen of a mass, specifically Merkel cell carcinoma, which has been resected. Merkel cell carcinoma is a rare and aggressive type of skin cancer that typically appears as a firm, flesh-colored or bluish-red nodule on sun-exposed areas. Early detection and treatment are crucial due to its rapid growth and potential to spread to other parts of the body. RAG Sequence For the Query: Carpal tunnel syndrome’s Anatomy: The carpal tunnel is a small space in the wrist, just above the base of the palm. It contains nine flexor tendons (which bend the fingers) and the median nerve. This nerve provides feeling to the thumb, index, middle, and half of the ring finger, as well as controls muscles that move the thumb. The carpal tunnel is bordered by bones and a strong ligament called the transverse carpal ligament or flexor retinaculum. It is located in the middle third of the palm, between the scaphoid bone (at the base of the thumb) and the hamate hook (along the ring finger). Brachydactyly type D: Brachydactyly type D, also known as short thumb or clubbed thumb, is a genetic trait where the thumb appears shorter and wider with a broader nail bed than usual. The distal phalanx of the affected thumb is approximately two-thirds the length of a normal thumb. It is quite common, affecting about 2–3 C: Brachydactyly type D’s Signs and symptoms: Brachydactyly type D is a skeletal condition where the tip of the thumb appears shorter due to partial fusion or early closure of the growth plate in the distal phalanx. This can affect one or both thumbs, making them approximately half to two-thirds the length of normal thumbs. Online RAG: The gross pathology specimen of Merkel cell carcinoma presents as a firm, nodular mass with a tan to pink cut surface. It is typically well-circumscribed but may show areas of ill-defined infiltration into surrounding tissue. On sectioning, the mass appears solid and homogeneous, sometimes with regions of hemorrhage or necrosis, especially in larger tumors. The surface may be smooth or lobulated, and involvement of the dermis and subcutaneous tissue is common. Despite its potentially benign appearance on gross examination, Merkel cell carcinoma is a highly aggressive neuroendocrine skin tumor often arising in sun-exposed areas of elderly or immunocompromised individuals. Image 2: Merkel cell carcinoma. Gross pathology specimen. Image 3: The cut surface of desmoid-type fibromatosis is firm, white, and whorled. The white tumor infiltrates the adjacent skeletal muscle (red tissue – lower left) and fat (yellow tissue – upper left). This tendency for invasion of adjacent normal tissues and structures is the reason that desmoid-type fibromatosis has a relatively high rate of local recurrence, even after surgical removal. Image 4: Image showing Gastrointestinal Stromal Tumor after surgical removal. Image 5: Squamous-Cell Carcinoma of the Skin | NEJM. Image 6: Radiology Workflow | Efficiencies Image 7: Patient enrollment workflow. MRI, magnetic resonance imaging; SLNB … RAG Image 2 RAG Image 3 RAG Image 4 Online RAGImg 5 Online RAGImg 6 Online RAGImg 7 Input Image (User-Provided)