RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
Haolin Li Tianjie Dai Zhe Chen Siyuan Du, Jiangchao Yao Ya Zhang Yanfeng Wang · 2025
Read RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis by Haolin Li Tianjie Dai Zhe Chen Siyuan Du, Jiangchao Yao Ya Zhang Yanfeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19980.txt.
Introduction
Clinical diagnosis demands guideline-grade rigor, and Haolin Li with six co-authors argue that knowledge should be injected openly, not baked into weights. Their framework, RAD, retrieves disease-centered knowledge from several medical sources and refines it for the case at hand. A guideline-enhanced contrastive loss pulls multimodal features toward that knowledge in latent space. A dual transformer decoder then reasons over image and text together.
Retrieval keeps the knowledge current and auditable, unlike parametric memory. Task-specific guidance matters because downstream diagnostic tasks differ widely. Trust is the stated goal: outputs anchored to named guidelines are easier for clinicians to check. The design targets multimodal diagnosis, where scans and notes must be read jointly. External knowledge, refined per case, is the core bet.
Catalog record
| CREATOR | Haolin Li Tianjie Dai Zhe Chen Siyuan Du, Jiangchao Yao Ya Zhang Yanfeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19980
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis Haolin Li Tianjie Dai Zhe Chen Siyuan Du, Jiangchao Yao Ya Zhang Yanfeng Wang Abstract Clinical diagnosis is a highly specialized discipline requiring both domain expertise and strict adherence to rigorous guidelines. While current AI-driven medical research predominantly focuses on knowledge graphs or natural text pretraining paradigms to incorporate medical knowledge, these approaches primarily rely on implicitly encoded knowledge within model parameters, neglecting task-specific knowledge required by diverse downstream tasks. To address this limitation, we propose R etrieval- A ugmented D iagnosis (RAD), a novel framework that explicitly injects external knowledge into multimodal models directly on downstream tasks. Specifically, RAD operates through three key mechanisms: retrieval and refinement of disease-centered knowledge from multiple medical sources, a guideline-enhanced contrastive loss that constrains the latent distance between multi-modal features and guideline knowledge, and the dual transformer decoder that employs guidelines as queries to steer cross-modal fusion, aligning the models with clinical diagnostic workflows from guideline acquisition to feature extraction and decision-making. Moreover, recognizing the lack of quantitative evaluation of interpretability for multimodal diagnostic models, we introduce a set of criteria to assess the interpretability from both image and text perspectives. Extensive evaluations across four datasets with different anatomies demonstrate RAD’s generalizability, achieving state-of-the-art performance. Furthermore, RAD enables the model to concentrate more precisely on abnormal regions and critical indicators, ensuring evidence-based, trustworthy diagnosis. Our code is available at this repository . 1 Introduction The rapid development of multimodal learning [ 45 , 36 ] has revolutionized numerous fields by enabling models to process and integrate diverse data types, including images, texts, audio, and structured records [ 4 , 76 , 11 ] . Biomedical applications particularly benefit from these advancements, given that diagnostic workflows inherently depend on multimodal evidence, ranging from radiographic imaging and reports to electronic health records (EHR) [ 51 , 74 , 16 ] . For instance, radiologists integrate X-ray or MRI scans with textual pathology reports, while clinicians combine electronic health records, vital signs, and even genomic data to form comprehensive patient profiles. Accordingly, recent research efforts have increasingly focused on developing multimodal architectures tailored to healthcare challenges, seeking to enhance diagnostic precision through cross-modal synergy [ 67 , 31 , 60 , 6 ] . While these approaches demonstrate significant progress in integrating data from different modalities, they often overlook the foundational principles governing clinical decision-making. Medical analysis fundamentally differs from natural scene understanding in its strict adherence to evidence-based principles, relying heavily on structured protocols [ 30 , 49 ] . Clinical decisions must be grounded in standardized diagnostic criteria derived from patient-specific symptoms, imaging findings, and laboratory results. This inherent rigor poses a critical challenge for black-box neural networks, whose vague decision-making mechanisms hinder trustworthy and practical deployment in clinical settings [ 52 , 48 ] . Consequently, there has been growing interest in integrating medical knowledge into AI models to simultaneously improve model performance and interpretability [ 8 , 72 , 9 ] . Existing approaches primarily focus on knowledge injection during pretraining phases. Researchers enhance the text encoders by pretraining them on large-scale medical corpora [ 29 , 46 ] or leveraging structured knowledge graphs to imbue models with semantic relationships between biomedical entities [ 47 , 35 ] . While effective in expanding the semantic coverage of text encoders, these approaches often struggle to explicitly integrate fine-grained knowledge tailored for downstream diagnostic tasks. To this end, we argue that effective knowledge integration requires task-centric, holistic alignment with disease-level knowledge throughout the entire diagnostic pipeline. As illustrated in Figure 1 (a), our framework systematically integrates refined knowledge to guide input augmentation, feature extraction, and modality fusion, contrasting with prior methods confined to a single perspective. Figure 1 (b) presents a case of the model’s attention distribution over the input text. The previous model fails to concentrate on critical indicators, but focuses on obvious disease terms in the reports. While the RAD model can not only attend to these terms but also consider other guideline-recommended key indicators. The explicit knowledge guidance enables RAD to prioritize critical indicators tailored for the current disease, making trustworthy diagnoses aligned with clinical standards. Figure 1 : The design motivation of RAD. Left : Previous methods mostly focus on enhancing a single aspect of the diagnostic process, whereas our approach is holistic. Right : Visualization of model attention to textual content. Color intensity reflects attention magnitude, with red highlighting disease-critical indicators mentioned in the guideline. Models without explicit knowledge guidance exhibit limited focus on key indicators, whereas our model can make evidence-based diagnoses. In this paper, we propose a holistic knowledge-injection framework RAD, which operates through three synergistic components spanning the entire diagnostic workflow. RAD begins by retrieving and refining disease-specific guidelines from diverse sources, flexibly adapting to downstream tasks in different scenarios. We then employ two modality encoders coupled with guideline-enhanced contrastive loss that explicitly aligns the modality-specific feature with the corresponding disease-guideline prototypes in the joint latent space. A dual decoder network is further developed to steer the cross-modal fusion process, which simultaneously incorporates disease labels and their corresponding guidelines to interact with fused multimodal features for final predictions. Through this systematic knowledge infusion paradigm, our framework achieves performance gains while establishing a traceable decision pathway grounded in clinical guidelines—a critical step toward clinically actionable AI. We further establish an evaluation system for model interpretability, which quantitatively assesses the model’s adherence to guidelines through both textual indicators and visual localization. Combined with qualitative visualization, this system provides measurable evidence that RAD’s decisions are driven by the injected knowledge. In summary, our contributions are three-fold: • We propose RAD to systematically inject external medical knowledge into multimodal diagnosis models. RAD incorporates a guideline-enhanced loss and a dual-decoder structure to explicitly steer multimodal feature extraction and cross-modal fusion with disease-guideline prototypes. • A dual-axis evaluation system for the interpretability of diagnosis models is developed, formulating both textual and visual metrics. This system enables quantitative analyses of the model’s adherence to clinical guidelines, demonstrating the transparency and explainability brought by RAD. • We aligned MIMIC-CXR [ 27 ] and MIMIC-IV [ 28 ] to construct the MIMIC-ICD53 dataset, covering three modalities with 53 types of disease. Extensive experiments on our dataset and three other public datasets demonstrate the superiority of RAD over SOTA baselines across various metrics. 2 Related Work 2.1 Multimodal Learning in Medicine Recent years have witnessed significant advancements in the field of multimodal learning, with models such as CLIP [ 45 ] , BLIP [ 32 ] , and LLaVA [ 36 ] exhibiting remarkable capabilities in natural domains. These developments have spurred increasing interest in extending multimodal frameworks to the medical field, where the integration of diverse data modalities demonstrates prominent potential in diagnostic tasks. Current research focus lies in multimodal pretraining methods, which focus on cross-modal alignment between imaging and textual data to improve the representation transferability [ 7 , 17 ] . ConVIRT [ 71 ] and GLoRIA [ 25 ] pioneered the application of CLIP-style architectures in the medical domain by constructing image-text pairs from radiology datasets. MedCLIP [ 55 ] and BiomedCLIP [ 69 ] addressed the scarcity of paired medical image-text data by leveraging