Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis
Jihyun Moon, Charmgil Hong · 2025
Read Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis by Jihyun Moon, Charmgil Hong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08338.txt.
Introduction
Jihyun Moon and Charmgil Hong improve skin cancer checks by letting a model consult like cases in a 2025 medical AI paper. Early, correct calls on melanoma save lives. Image networks read skin photos well, but they skip the patient's clinical facts. Broad vision-language models use both, yet lack medical sharpness.
Their fix is retrieval. For each new case, the system finds past cases that look and read alike. Those cases go into the prompt that guides the call. No retraining is needed. Scores rose sharply in their tests. Doctors build trust in a diagnosis the same way, by match to cases they have seen. A tool that shows its matches earns trust faster.
Catalog record
| CREATOR | Jihyun Moon, Charmgil Hong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08338
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis Jihyun Moon, Charmgil Hong Abstract Accurate and early diagnosis of malignant melanoma is critical for improving patient outcomes. While convolutional neural networks (CNNs) have shown promise in dermoscopic image analysis, they often neglect clinical metadata and require extensive preprocessing. Vision-language models (VLMs) offer a multimodal alternative but struggle to capture clinical specificity when trained on general-domain data. To address this, we propose a retrieval-augmented VLM framework that incorporates semantically similar patient cases into the diagnostic prompt. Our method enables informed predictions without fine-tuning and significantly improves classification accuracy and error correction over conventional baselines. These results demonstrate that retrieval-augmented prompting provides a robust strategy for clinical decision support. 1 Introduction Malignant melanoma is the most common and deadliest form of skin cancer, with 100,640 new cases and 8,290 deaths reported in the United States in 2024 [ 16 ] . Early detection significantly improves survival, which emphasizes the need for accurate and timely diagnosis. Automated diagnostic tools can assist clinicians in detecting malignant lesions at an earlier stage. This can lead to improved prognosis and make timely intervention more achievable. While convolutional neural network (CNN)-based methods have shown promise [ 10 , 11 ] , most rely solely on dermoscopic images and often require preprocessing steps such as region of interest (ROI) segmentation, which limits their utility in clinical practice. To address these limitations, recent efforts have explored multimodal frameworks that incorporate both images and clinical metadata have gained attention for improving diagnostic accuracy and personalization. Vision-language models (VLMs) [ 13 , 2 ] have emerged as strong candidates for such tasks, as they jointly process visual and textual data without the need for handcrafted preprocessing. However, off-the-shelf VLMs that are trained on general-purpose data often fail to capture domain-specific complexities [ 5 ] . Although fine-tuning with clinical data can mitigate this issue, it requires curated datasets and significant computational resources. As a result, this approach is often infeasible due to privacy constraints and institutional variability. As an alternative, retrieval-augmented generation (RAG) [ 12 ] provides external knowledge-based inference by retrieving similar patient cases and incorporating them into prompts. This approach is particularly appealing for clinical applications, since it enables reasoning without modifying model weights. Previous studies in content-based image retrieval (CBIR) have shown that case-based reasoning can support dermatological diagnosis by referencing visually similar lesions [ 19 , 3 ] . Building on this idea, our work extends case-based reasoning to a multimodal context by integrating retrieved image–text pairs into VLM prompts. This design supports clinical reasoning by reflecting the way how physicians interpret new cases through analogical comparison with prior patient examples. More specifically, this study proposes a multimodal diagnostic framework that integrates RAG into a VLM to support more accurate and clinically relevant melanoma classification. We investigate whether retrieved examples improve diagnostic decisions, particularly in correcting false positives and false negatives. Through comprehensive experiments, we show that our method consistently outperforms conventional classification models in both accuracy and error correction. Our main contributions are as follows: • We propose a retrieval-augmented VLM-based diagnostic framework for mela-noma classification by incorporating image–metadata–label examples into prompts to improve decision accuracy. • We evaluate the effects of different metadata serialization strategies and image encoders on retrieval effectiveness and diagnostic performance. • We show that the proposed method consistently outperforms conventional image-based, text-based, and early-fusion baselines across multiple metrics and architectures, without requiring fine-tuning. 2 Proposed Approach This section presents a multimodal diagnostic framework that combines a VLM with RAG to classify melanoma using both dermoscopic images and clinical metadata. In clinical practice, diagnosis often involves comparing a case with prior cases that share similar visual features or clinical attributes. This case-based reasoning improves diagnostic accuracy by using past experience. Our framework incorporates a retrieval module that searches a database of dermoscopic images and metadata to find semantically similar cases. These examples serve as clinical references and provide contextual support for the prediction of the model. By embedding the retrieved cases into the VLM input prompt, the system emulates the comparative reasoning process used in human diagnosis. This design addresses the limitations of general-purpose VLMs and better aligns the model with the specific demands of melanoma classification. An overview of the architecture is shown in Fig. 1 . Multimodal Embedding and Case Indexing Each training sample includes a dermoscopic image and associated metadata ( e.g. , age, sex, and lesion location). To process these modalities, we use modality-specific encoders: CNN-based backbones ( e.g. , ResNeXt-50 [ 22 ] , EfficientNet-V2-M [ 17 ] ) for images and BERT [ 8 ] for text. Metadata is serialized into natural language using template-based sentence transformations (see below) to improve compatibility with language models. The resulting image and text embeddings are concatenated into a single multimodal vector and stored in a FAISS [ 9 ] vector database for efficient approximate nearest neighbor search. This setup enables scalable indexing of large dermatological datasets and allows seamless updates as new data is added. Template-Based Sentence Transformations To make structured metadata compatible with VLMs, we convert patient records into natural language using predefined templates. We apply three serialization strategies: • Sentence format : Expresses each field as a simple sentence ( e.g. , Age is 45, Sex is female.), matching the VLM’s training style. • Attribute-value pair : Uses compact key-value pairs ( e.g. , Age: 45, Sex: Female) to reduce prompt length and improve parsing. • HTML format : Encodes tabular structure with tags like <table>, <tr>, <th>, and <td> to retain column semantics. Semantically-Guided Retrieval At inference time, the query sample (image and metadata) is encoded with the same modality-specific encoders used to index the database. We compute dot-product similarity between the query and database embeddings to retrieve the top- K most similar cases. These examples, selected based on both visual and clinical similarity, provide contextual support for VLM prompting. This retrieval introduces domain-specific knowledge without updating model weights and enables adaptation to the target domain. We found that K=2 offers the best balance between contextual relevance and noise. Figure 1: Overview of the proposed retrieval-augmented classification framework incorporating attribute-value pair-based prompting. Prompt Construction and VLM Inference To adapt general-purpose VLMs for binary melanoma classification, we design structured prompts consists of: (1) instruction specifying the task; (2) K retrieved examples as image–metadata–label triplets; and (3) the query sample with a classification request. The frozen VLM processes the prompt and generates a textual response indicating the predicted class. This few-shot design mirrors clinical reasoning by analogy and leads to better contextual understanding and more reliable predictions. Unlike early-fusion [ 18 ] or naive multimodal concatenation [ 14 ] , our method maintains modality alignment and exploits the ability of the VLM to perform implicit multimodal reasoning. Table 1: Comparison of image(I)-based, metadata(M)-based, early-fusion, zero-shot VLM, and our proposed framework (bold: the highest performance). I M Model Serialization Accuracy Balanced Accuracy Precision Sensitivity F1-score TN TP FN FP ✓ - ResNeXt-50 [ 22 ] - 0.7380 0.5054 0.2022 0.1316 0.1594 6402 223 1472 880 ✓ - EfficientNet-V2-M [ 17 ] - 0.6954 0.5061 0.1985 0.2018 0.2001 5901 342 1353 1381 Single - ✓ RF [ 4 ] - 0.8156 0.5209 0.6667