Prompt Mechanisms in Medical Imaging: A Comprehensive Survey
Hao Yang, Xinlong Liang, Zhang Li, Yue Sun, Zheyu Hu, Xinghe Xie, Behdad Dashtbozorg, Jincheng Huang, Shiwei Zhu, Luyi Han, Jiong Zhang, Shanshan Wang, Ritse Mann, Qifeng Yu, Tao Tan · 2025
Read Prompt Mechanisms in Medical Imaging: A Comprehensive Survey by Hao Yang, Xinlong Liang, Zhang Li, Yue Sun, Zheyu Hu, Xinghe Xie, Behdad Dashtbozorg, Jincheng Huang, Shiwei Zhu, Luyi Han, Jiong Zhang, Shanshan Wang, Ritse Mann, Qifeng Yu, Tao Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01055.txt.
Introduction
Hao Yang and co-authors survey prompt methods in medical imaging, where prompts guide deep learning models without full retraining, published in 2025. Deep learning holds great promise for medical images, but clinical use is slowed by scarce data, shifts in data distribution, and the need to generalize across tasks. Prompts offer a flexible way to adapt a model to a domain.
The review sorts prompts into text instructions, visual prompts, and learnable embeddings. It analyzes how each is built into models for tasks such as image generation, segmentation, and classification. Prompts are found to improve accuracy, robustness, and data efficiency, and to make a model's guidance more explicit. The authors flag open problems in prompt design, data variety, and scaling for real clinical deployment.
Catalog record
| CREATOR | Hao Yang, Xinlong Liang, Zhang Li, Yue Sun, Zheyu Hu, Xinghe Xie, Behdad Dashtbozorg, Jincheng Huang, Shiwei Zhu, Luyi Han, Jiong Zhang, Shanshan Wang, Ritse Mann, Qifeng Yu, Tao Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt Mechanisms in Medical Imaging: A Comprehensive Survey Hao Yang, Xinlong Liang, Zhang Li, Yue Sun, Zheyu Hu, Xinghe Xie, Behdad Dashtbozorg, Jincheng Huang, Shiwei Zhu, Luyi Han, Jiong Zhang, Shanshan Wang, Ritse Mann, Qifeng Yu, Tao Tan Abstract Deep learning offers transformative potential in medical imaging, yet its clinical adoption is frequently hampered by challenges such as data scarcity, distribution shifts, and the need for robust task generalization. Prompt-based methodologies have emerged as a pivotal strategy to guide deep learning models, providing flexible, domain-specific adaptations that significantly enhance model performance and adaptability without extensive retraining. This systematic review critically examines the burgeoning landscape of prompt engineering in medical imaging. We dissect diverse prompt modalities, including textual instructions, visual prompts, and learnable embeddings, and analyze their integration for core tasks such as image generation, segmentation, and classification. Our synthesis reveals how these mechanisms improve task-specific outcomes by enhancing accuracy, robustness, and data efficiency and reducing reliance on manual feature engineering while fostering greater model interpretability by making the model’s guidance explicit. Despite substantial advancements, we identify persistent challenges, particularly in prompt design optimization, data heterogeneity, and ensuring scalability for clinical deployment. Finally, this review outlines promising future trajectories, including advanced multimodal prompting and robust clinical integration, underscoring the critical role of prompt-driven AI in accelerating the revolution of diagnostics and personalized treatment planning in medicine. keywords Prompt mechanisms, medical imaging, deep learning, Interpretability Introduction The integration of Artificial Intelligence (AI), intense learning models such as convolutional neural networks (CNNs), has revolutionized medical image analysis, producing remarkable advancements in image classification, segmentation, and generation [ 1 , 2 , 3 , 4 ] . Despite these successes, clinical translation and widespread adoption of these models are often impeded by persistent challenges such as data scarcity and distribution shifts. Among these, distribution shifts between training datasets and various real-world clinical data present a significant obstacle, often leading to a degradation in model performance and reliability [ 5 ] . These shifts arise from inherent dataset biases, inter-scanner variability, and diverse image acquisition protocols. Consequently, improving model adaptability and robustness to such heterogeneous data distributions has become a significant research imperative in medical image processing [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ] . The recent advent of Foundation Models (FMs), large-scale models pre-trained on extensive and diverse datasets, underpinned by architectures such as the Transformer and encompassing architectures such as Large Language Models (LLMs), and crucially for this field, Vision-Language Models (VLMs), offers a transformative new paradigm for AI in medical imaging [ 14 ] . These FMs exhibit superior versatility and transfer learning capabilities, demonstrating immense potential in few-shot learning, cross-modal information fusion, and robust performance in complex data-scarce scenarios. However, effectively harnessing the power of these generalist FMs for specialized medical tasks requires sophisticated adaptation strategies beyond traditional fine-tuning. In this context, prompt-based mechanisms have rapidly emerged as a key and powerful approach to steering FMs. Concurrently, the underlying principles of prompting-guiding model behavior with external contextual information have also been increasingly explored and adapted for a broader range of deep learning architectures in medical imaging, extending beyond large-scale FMs. These mechanisms are proving particularly effective in addressing the aforementioned distribution shifts, managing complex specialized medical knowledge, and unlocking the full potential of FM and other customized models for applications in precision medicine and personalized diagnostics. Figure 1: An overview of the framework for prompt-based AI in medical imaging. The process begins with various medical imaging data sources, which are processed by an AI model capable of accepting multiple prompt types (visual, text, sequence, etc.). By performing classification, segmentation, or generation tasks, the model ultimately serves a range of clinical applications to improve diagnostic efficiency and accuracy. Figure 2: Rapid increase of the number of Prompt-driven Medical imaging papers. Generation, Classification, and Segmentation are the three main taxonomy categories introduced in this survey. The core tenet of prompting is to guide and optimize a model’s performance on specific downstream tasks by introducing targeted external contextual information. As illustrated in the general framework of figure 1 , this process begins with various sources of medical imaging data. An AI model, designed to be receptive to multiple prompt types (visual, text, etc.), is then guided by these targeted inputs. The prompt leads the model to perform a specific task, such as classification, segmentation, or generation, ultimately serving a range of clinical applications by enhancing diagnostic accuracy and efficiency. This information can be encoded as textual instructions [ 6 ] , visual prompts (e.g., points, boxes) [ 15 ] , or learnable embeddings. This strategy aims to significantly augment the model’s capacity to interpret and analyze medical images with high fidelity, especially when annotated data is scarce or expensive. A key technical advantage of prompting lies in its remarkable parameter efficiency: it often necessitates fine-tuning only a small subset of the model’s parameters-or none at all in zero-shot prompting scenarios-to adapt the pre-trained model to new tasks, thereby obviating the need for substantial architectural modifications or complete retraining from scratch [ 4 ] . Furthermore, prompting offers exceptional flexibility and adaptability, with diverse guidance strategies enabling models to tackle a broad spectrum of medical imaging tasks and their associated challenges [ 16 ] . By facilitating the integration of additional knowledge and context, prompting can significantly enhance the generalization and robustness of the model, particularly when faced with limited sample sizes [ 6 , 7 , 8 , 5 ] . Recent advances in this growing field showcase increasingly sophisticated techniques, including multimodal prompts that fuse information from various channels (e.g., image and text) to tackle complex problems [ 17 ] , adaptive prompting algorithms that dynamically tailor prompt content based on specific task requirements and data characteristics [ 16 ] , and personalized prompting schemes that customize inputs for individual images to mitigate issues arising from sample distribution discrepancies, thereby enhancing overall system performance [ 18 ] . Despite these promising developments and the rapid proliferation of research, the practical application and optimal design of prompt mechanisms in medical imaging are still confronted by several critical limitations deeply intertwined with the nature of prompting. While prompting is often positioned as a solution for data scarcity [ 11 ] , the design and validation of high-quality prompts introduce their data-dependent challenges. Crafting effective textual prompts that capture nuanced clinical knowledge or generating representative learnable embeddings requires a sufficient volume of diverse and meticulously annotated data [ 6 , 7 , 8 , 5 ] , the availability of which is curtailed by stringent privacy regulations. The complexities of medical annotation further undermine the quality and reliability of prompts; the process demands specialized expertise and is prone to inter-observer variability. This subjectivity translates directly into inconsistent or ambiguous prompts-whether textual, visual, or learned-which can mislead the model and degrade performance, complicating the development of standardized and reproducible prompting protocols. Crucially, the challenge of distribution shift [ 5 ] extends beyond the model’s core to the prompts themselves. A prompt that is effective for one data distribution (e.g., images from a specific scanner or institution) may fail dramatically on another, a phenomenon that can be described