Vision Language Models in Medicine

Béria Chingnabé Kalpélbé, Angel Gabriel Adaambiik, Wei Peng · 2025

Read Vision Language Models in Medicine by Béria Chingnabé Kalpélbé, Angel Gabriel Adaambiik, Wei Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01863.txt.

Introduction

Beria Chingnabe Kalpelbe, Angel Gabriel Adaambiik, and Wei Peng surveyed vision-language models in medicine in 2025. Such models read images and text together, and their arrival has shifted medical AI. The survey traces how general models are adapted to medical tasks and where they are used.

Clinical care, teaching, and patient support all stand to gain. Standing in the way are scarce data, narrow skills, murky reasoning, and ethical worries over fairness and privacy. Uneven data sets and heavy computing deepen the gaps. The authors urge strict testing and sound rules before ward use. Federated learning, lighter models, and record linkage are floated as routes to wider, safer access.

Catalog record

CREATORBéria Chingnabé Kalpélbé, Angel Gabriel Adaambiik, Wei Peng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01863

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision Language Models in Medicine Béria Chingnabé Kalpélbé, Angel Gabriel Adaambiik, Wei Peng Abstract With the advent of Vision-Language Models (VLMs), medical artificial intelligence (AI) has experienced significant technological progress and paradigm shifts. This survey provides an extensive review of recent advancements in Medical Vision-Language Models (Med-VLMs), which integrate visual and textual data to enhance healthcare outcomes. We discuss the foundational technology behind Med-VLMs, illustrating how general models are adapted for complex medical tasks, and examine their applications in healthcare. The transformative impact of Med-VLMs on clinical practice, education, and patient care is highlighted, alongside challenges such as data scarcity, narrow task generalization, interpretability issues, and ethical concerns like fairness, accountability, and privacy. These limitations are exacerbated by uneven dataset distribution, computational demands, and regulatory hurdles. Rigorous evaluation methods and robust regulatory frameworks are essential for safe integration into healthcare workflows. Future directions include leveraging large-scale, diverse datasets, improving cross-modal generalization, and enhancing interpretability. Innovations like federated learning, lightweight architectures, and Electronic Health Record (EHR) integration are explored as pathways to democratize access and improve clinical relevance. This review aims to provide a comprehensive understanding of Med-VLMs’ strengths and limitations, fostering their ethical and balanced adoption in healthcare. I Introduction Leveraging advanced algorithms and neural network architectures like Transformers [ 1 ] , AI has been empowered with strong reasoning ability and made tremendous progress in recent years. Breakthroughs in model design and training methodologies have allowed machines to excel in complex tasks, including Natural Language Processing (NLP) applications such as language translation, sentiment analysis, and text generation, achieving high accuracy and fostering intuitive human-computer interactions. Similarly, advancements in Computer Vision (CV) have empowered AI to analyze and interpret images, videos, and audio sequences with remarkable precision. In healthcare, Artificial Intelligence (AI) is revolutionizing medicine by enabling data-driven insights, improving diagnostics, and personalizing treatments [ 2 , 3 , 4 ] . These innovations have enabled significant applications, such as medical imaging analysis, disease diagnosis, pathology, radiology workflow optimization, and surgical assistance, transforming patient care and clinical workflows [ 5 , 6 ] . The medical field faces unique challenges in data interpretation and decision-making for healthcare specialists; they must analyze diverse types of information including medical imaging (X-rays, MRIs, pathology slides), clinical notes, patient histories, and real-time observations. Medical images are critical for diagnostic checks and measurements, such as identifying anatomical abnormalities, quantifying disease progression, or assessing treatment efficacy. On the other hand, textual data, such as clinical notes, nurse evaluations, and patient histories, provide essential context for screening, understanding symptoms, and documenting disease progression. Textual outputs, such as radiology reports or discharge summaries, are equally vital, as they synthesize findings into actionable insights for clinicians. The complexity and volume of this multi-modal medical data often lead to cognitive overload, impacting the speed and accuracy of diagnoses. Traditional single-modality approaches, which treat images and text separately, fail to capture the intricate relationships between visual findings and clinical context. This limitation underscores the need for integrated vision-language models (VLMs) that can bridge the gap between these modalities [ 7 ] , enabling more comprehensive and accurate decision-making in healthcare. This integrated approach promises to enhance clinical decision-making by providing more contextually informed insights and reducing the cognitive burden on healthcare providers. Fig. 1: Comprehensive Framework for Medical Vision-Language Models (VLMs) . (a) Training involves processing diverse inputs such as images, texts, metadata, and historical data, followed by pre-training. (b) Benchmarking is conducted on a variety of medical datasets including GMAI-MMBench, OmniMedVQA, RadBench, and others. (c) Advanced training strategies are employed, such as vision-text alignment, knowledge distillation, masked language modeling, contrastive learning, and parameter-efficient tuning. (d) Evaluation strategies encompass automated metrics like BLEU, ROUGE, BERTScore, and clinical-specific tools like CheXpert Labeler and RadGraph, alongside human evaluation. (e) Integration of VLMs into the medical workflow leverages contextual data to provide actionable insights and improve clinical decision-making. However, visual and language provide totally different modalities that are not trivial to be integrated directly. As illustrated in Fig. 1 , existing works address this challenge through various strategies, including vision-text alignment (in MedViL [ 8 ] , MedCLIP [ 9 ] , BioMedCLIP [ 10 ] , VividMed [ 11 ] ), knowledge distillation with VividMed [ 11 ] , masked language modeling (in MedViL [ 8 ] and BioMedCLIP [ 10 ] ) and contrastive learning in MedCLIP [ 9 ] , BioViL [ 12 ] & ConVIRT [ 13 ] . More recent advancements have introduced additional approaches, such as frozen encoders and Q-Former (e.g., BLIP-2 [ 14 ] , InstructBLIP [ 15 ] ), image-text pair learning and fine-tuning (e.g., LLaVA [ 16 ] , LLaVA-Med [ 17 ] , BiomedGPT [ 18 ] , MedVInT [ 19 ] ), parameter-efficient tuning (e.g., LLaMA-Adapter-V2 [ 20 ] ), two-stage training (e.g., MiniGPT-4 [ 21 ] ), and modular multimodal pre-training (e.g., mPLUG-Owl [ 22 ] , Otter [ 23 ] ) and the Sigmoid Loss for Language-Image Pre-Training (SigLIP) [ 24 ] . SigLIP replaces traditional softmax-based contrastive learning with a simpler sigmoid loss approach, enabling more efficient and scalable training by treating image-text pair alignment as a binary classification task. These methods aim to establish coherent relationships between visual inputs and textual outputs, enabling models to effectively interpret and generate relevant information across modalities. As a comparison, the contrastive learning-based methods leverage the similarities and differences between paired visual and textual data to enhance model robustness and generalization. This paper provides a comprehensive review of VLMs and their applications in healthcare. We first discuss how VLMs are constructed by integrating advancements in NLP and computer vision. Next, we summarize key methodologies and advancements in the field, including state-of-the-art models like Qwen-VL [ 25 ] , RadFM [ 26 ] , and DeepSeek-VL [ 27 ] . We then explore how VLMs are applied in the medical domain, highlighting their potential to improve diagnostic accuracy, clinical decision-making, and other healthcare tasks. Finally, we conclude by outlining future directions and challenges in the integration of VLMs into healthcare practices. II Multi-modal Visual Text Models Vision-Language Models (VLMs) are a class of artificial intelligence models designed to process and integrate both medical imaging data (e.g., radiographs, histopathological slides) and clinical text (e.g., diagnostic reports, physician notes). The evolution of modern VLMs is rooted in advances in natural language processing, particularly the development of Bidirectional Encoder Representations of Transformers (BERT) [ 8 ] . BERT was originally developed for text classification and other language-specific tasks, including clinical text processing and medical literature analysis. Researchers soon recognized its potential for multimodal applications, enabling the integration of medical imaging with textual data within a unified framework. This advancement has paved the way for AI models capable of enhancing diagnostic interpretation, clinical decision support, and medical research. II- 1 Expanding BERT to Visual Data: Several models extended BERT’s text-processing capabilities to incorporate visual data, enabling applications in medical imaging and clinical decision support. A notable early example is VisualBERT [ 28 ] , which applies BERT’s transformer architecture to jointly model vision and language tasks. VisualBERT operates by stacking transformer layers, taking both a medical image (e.g., radiographs or histopathology slides) and its corresponding text (such as

Read the complete text