Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations

Kangyu Zhu * 1,4 , Ziyuan Qin * 5 , Huahui Yi 1 , Zekun Jiang 1 , Qicheng Lao 3 , Shaoting Zhang 6 , Kang Li 1,2 1 West China Biomedical Big Data Center, West China Hospital, Sichuan University 2 Sichuan University Pittsburgh Institute 3 Beijing University of Posts and Telecommunications 4 Brown Uni · 2025

Read Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations by Kangyu Zhu * 1,4 , Ziyuan Qin * 5 , Huahui Yi 1 , Zekun Jiang 1 , Qicheng Lao 3 , Shaoting Zhang 6 , Kang Li 1,2 1 West China Biomedical Big Data Center, West China Hospital, Sichuan University 2 Sichuan University Pittsburgh Institute 3 Beijing University of Posts and Telecommunications 4 Brown Uni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02385.txt.

Introduction

Kangyu Zhu and co-authors teach medical vision-language models to look where a clinician points. General models grasp a whole scan but cannot focus on a region; they learn attention indirectly from image-text pairs. The team's framework, MedVP, extracts medical entities from text, draws visual prompts such as markers on the image, and adapts datasets so models can be fine-tuned to use prompts.

Across several medical question-answering datasets, the prompted models beat large systems. Experiments and human evaluation dissected which prompt forms help and why, separating the effect of the marker's shape from extra training data. The authors argue explicit visual guidance is a natural interface, since doctors point at scans. The framework turns that gesture into model input. For hospitals, a model that answers about the spot you marked is far more useful than one describing the whole image.

Catalog record

CREATORKangyu Zhu * 1,4 , Ziyuan Qin * 5 , Huahui Yi 1 , Zekun Jiang 1 , Qicheng Lao 3 , Shaoting Zhang 6 , Kang Li 1,2 1 West China Biomedical Big Data Center, West China Hospital, Sichuan University 2 Sichuan University Pittsburgh Institute 3 Beijing University of Posts and Telecommunications 4 Brown Uni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02385

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations Kangyu Zhu * 1,4 , Ziyuan Qin * 5 , Huahui Yi 1 , Zekun Jiang 1 , Qicheng Lao 3 , Shaoting Zhang 6 , Kang Li 1,2 1 West China Biomedical Big Data Center, West China Hospital, Sichuan University 2 Sichuan University Pittsburgh Institute 3 Beijing University of Posts and Telecommunications 4 Brown University 5 Case Western Reserve University 6 Shanghai Jiao Tong University Abstract While mainstream vision-language models (VLMs) have advanced rapidly in understanding image-level information, they still lack the ability to focus on specific areas designated by humans. Rather, they typically rely on large volumes of high-quality image-text paired data to learn and generate posterior attention maps. To address this critical issue, we propose leveraging visual prompts—simple visual markers in various forms—to guide and enhance the formation of region-specific attention. Thus, we introduce MedVP , a pioneering framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation for visual prompt-guided fine-tuning. We successfully outperform recent state-of-the-art large models across multiple medical VQA datasets. Extensive experiments and Human evaluation are conducted to analyze the impact of different visual prompt forms and how they contribute to performance improvement. The results demonstrate both the effectiveness and clinical significance of our approach. † † footnotetext: * Equal Contribution. 1 Introduction As human beings, we inherently rely on visual cues or prompts to understand complex visual content in greater detail. These visual prompts may take various forms, yet they serve a fundamentally similar purpose: to direct our attention toward critical areas that contain rich information. Figure 1: The figure illustrates the difference in the reasoning process of the Vision-Language Model (VLM) with and without visual prompts. By emulating how humans approach vision-language tasks, we use an auxiliary visual prompt, akin to pointing at a specific region, to help the model focus more easily on the relevant details and generate accurate responses. Current mainstream pre-trained vision-language models often lack mechanisms to direct their attention to specific areas of interest. Instead, their attention distribution is formed during training on specific datasets. However, these pre-trained attention patterns are not universally applicable, particularly in specialized domains such as biomedical images. Consequently, a substantial amount of high-quality image-text paired data is required to fine-tune pre-trained VLMs, ensuring their attention distribution aligns with medical image characteristics. This need persists despite the availability of prior knowledge that could guide attention to more meaningful areas. This intuition also aligned with active vision ( Karasev et al. 2011 ) and interactively segmentation ( Tanida et al. 2023 ) in medical domain. Therefore, inspired by the human recognition process, we propose leveraging visual prompts—free-form visual markers, such as arrows or circles, to indicate areas of interest—to guide VLMs in attending to specific regions during fine-tuning. This approach facilitates the adaptation of attention distribution and accelerates the formation of posterior attention maps in the medical domain. To be specific, Vision-Language Models (VLMs) are a class of models designed to process both image and text inputs for cross-modal tasks. Depending on the specific task, the architecture and pre-training strategies of these models can vary significantly. For the purposes of this discussion, we will focus specifically on Large Vision-Language Models (LVLMs), a subset of VLMs that are particularly designed for image-to-language tasks, such as Visual Question Answering (VQA) and Image Captioning. In this line of research, foundation models such as BLIP-2 ( Li et al. 2023 ) and LLaVA ( Liu et al. 2024b ) have significantly influenced the design of current mainstream LVLMs. Taking LLaVA as an example, these models are typically composed of an image encoder, a projection layer, and a Large Language Model (LLM)-driven decoder. In the majority of Vision-Language Models (VLMs) ( Li et al. 2023 ; Liu et al. 2024b ) , pre-training primarily involves generating descriptions of image content, which encourages the model to focus on global visual features rather than specific regions. During this phase, the attended regions in images remain unguided, allowing the attention mechanism to autonomously learn an implicit attention distribution. As previously discussed, conventional self-attention mechanisms in these models often struggle to generate region-specific content. However, certain vision-language tasks, such as Visual Question Answering (VQA) and Referring Expression Comprehension (REC), inherently require the model to attend to specific objects or localized regions. Learning an effective posterior attention distribution for a given textual query, however, typically demands a substantial amount of training data, making it a challenging problem in vision-language learning. To mitigate this problem, recent research ( Yang et al. 2023 ; Rasheed et al. 2024 ; Cai et al. 2024 ; Zhang et al. 2024 ; Kirillov et al. 2023 ) has increasingly recognized the value of visual prompts and their application in vision-language tasks. Visual prompts can be categorized as implicit or explicit concerning the visual inputs. In this study, we will focus on explicit visual prompts, which are basically visual markers in various form that can be directly added to the input images. Inspired by ViP-LLaVA ( Cai et al. 2024 ) , we decide to add visual markers into the input images to highlight the Region of Interest (ROI) for the visual-prompt guided VLMs to focus on important regions selected by prior knowledge. However, current works ( Zhang et al. 2024 ; Cai et al. 2024 ; Kirillov et al. 2023 ) usually assume these ROIs are provided interactively by human annotators. Given the high cost of acquiring human annotations in medical images, we propose a novel method, MedVP ( Med ical V isual P rompting) that will automatically generate visual prompts for the interested region given the context. Specifically, our framework first extracts entities or relevant keywords from the queries, as our focus is on medical VQA tasks. Next, we fine-tune an open-set grounding model that generates coordinates for the target area based on the identified medical entities. Finally, these coordinates are used to generate visual prompts in various formats—such as scribbles, bounding boxes, or circles—on the input images. Along with the visual-prompted images, we adapt the text Question-Answer pairs from SLAKE, VQA-RAD, and PMC-VQA datasets to ensure that the language decoder becomes aware of the presence of our visual prompt markers within the images. For example, we incorporate text such as "Carefully observe the area in the red box…" to guide the LLM-driven decoder to focus on the visually prompted area, ensuring that the model pays attention to the regions highlighted by the visual prompts. The visual-prompted images are fine-tuned with Vision-Language Models (VLMs) to develop our model, MedVP-LLaVA —the first medical VLM guided by visual prompts. We find that our MedVP-LLaVA performs effectively in medical VQA tasks, significantly enhancing performance across multiple datasets. In conclusion, our MedVP method has at least the following contribution: 1. We are the first to introduce explicit visual prompts into medical-specialized Vision-Language Models, and we validate the effectiveness of visual prompts in enhancing performance for medical VQA tasks. 2. We design a whole framework from extracting keywords from VQA queries to visual grounding given medical entities, generate the visual prompts to the images and adapt three VQA datasets to include the information of our

Read the complete text