Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis

Fengyi Zhang, Xujie Zeng, Mohan Liu, Zengyi Wang, Yalong Jiang · 2026

Read Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis by Fengyi Zhang, Xujie Zeng, Mohan Liu, Zengyi Wang, Yalong Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18130.txt.

Introduction

Fengyi Zhang and four colleagues present Rad-VLSM, a two-stage cross-modal method for masking and calls. A BLIP-2 sight-language module first finds lesion regions under meaning cues and turns them into box prompts. A SAM-based multi-job network then pools multi-candidate regions for stable prompts and masking. Predicted masks serve as spatial priors for calls, while a fusion head joins lesion-aware seen features with texture marks. Tests on a private breast ultrasound set and public benchmarks show strong masking and calls with good spread.

Catalog record

CREATORFengyi Zhang, Xujie Zeng, Mohan Liu, Zengyi Wang, Yalong Jiang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58145
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis Fengyi Zhang, Xujie Zeng, Mohan Liu, Zengyi Wang, Yalong Jiang Abstract Medical image segmentation is more clinically valuable when it supports diagnosis rather than merely producing lesion masks. However, diagnostically relevant lesion cues are often subtle and localized, while existing models may be distracted by background tissues, acoustic artifacts, and irrelevant visual correlations. To address this problem, we propose Rad-VLSM, a two-stage cross-modal framework for semantics-assisted lesion focusing, robust segmentation, and visually grounded diagnosis. In the first stage, a BLIP-2-based vision-language alignment module identifies lesion-related candidate regions under semantic guidance and converts them into box prompts. In the second stage, these prompts are fed into a SAM-based multitask network, where a multi-candidate region aggregation strategy improves prompt stability and guides lesion segmentation. The predicted masks are then used as spatial priors for diagnosis, and a visual-radiomics fusion head integrates lesion-aware visual features with selected radiomics descriptors. By using semantic information for localization rather than direct prediction, Rad-VLSM reduces text-to-diagnosis dependence and grounds diagnosis in lesion-level evidence. Experiments on a private clinical breast ultrasound dataset and public benchmarks show that Rad-VLSM achieves strong segmentation and diagnostic performance with favorable generalization. I Introduction Fig. 1: Comparison between existing methods and ours Rad-VLSM. (a) Existing segmentation–classification methods often rely on whole-image representations, making diagnosis vulnerable to Interfering information. (b) Rad-VLSM progressively focuses on lesion regions and performs lesion-grounded diagnosis using mask-guided visual and radiomics evidence. Medical imaging plays a pivotal role in disease screening, subtype diagnosis, and treatment planning, while accurate lesion segmentation remains one of its most fundamental technical components [ 1 ] , [ 2 ] , [ 3 ] . Recent deep learning models have achieved high-quality masks across a variety of medical segmentation tasks, and the emergence of foundation models has further improved their adaptability across imaging scenarios [ 4 ] , [ 5 ] , [ 6 ] . However, in real clinical workflows, a high-quality mask is not the ultimate goal. What clinicians truly need is a model that can not only delineate the lesion precisely, but also make a reliable diagnostic judgment and explicitly indicate the evidence supporting that judgment, thereby forming a diagnostic loop grounded in spatial evidence [ 7 ] , [ 8 ] , [ 9 ] . Therefore, beyond developing isolated segmentation or classification models, it is clinically meaningful to build a unified framework that connects lesion localization, segmentation, and diagnosis [ 10 ] , [ 11 ] , [ 12 ] . This challenge becomes even more pronounced in ultrasound-based lesion analysis. In breast ultrasound images, lesions often occupy only a small fraction of the image and are frequently accompanied by weak boundaries, low contrast, heterogeneous internal echoes, speckle noise, and acoustic shadows, making reliable lesion-level representation difficult [ 13 ] , [ 14 ] , [ 15 ] . Meanwhile, surrounding glandular tissues and extra-lesion contexts may introduce confounding visual patterns. As a result, models relying primarily on global image-level representations may be distracted by irrelevant regions, leading to the dilution of lesion-specific discriminative cues [ 16 , 17 ] ,as shown in Fig. 1 (a) . Promptable segmentation models such as SAM and MedSAM provide a promising way to extract lesion regions through spatial prompts [ 5 , 18 , 6 ] . Recent studies have further explored medical SAM adaptation from multiple perspectives, including uncertainty-guided multi-prompt adaptation, SAM-based semi-supervised decoding, lightweight modality-decoupled adaptation, and gaze-assisted prompting [ 19 , 20 , 21 , 22 ] . However, their effectiveness still depends heavily on whether the provided prompts can reliably indicate the lesion area. Manual prompts are costly, while automatically generated prompts may be unstable when lesion boundaries are weak or the background is complex [ 23 , 24 ] . Vision-language alignment provides a potential way to transform lesion-related semantic understanding into spatial localization cues [ 25 , 26 , 27 ] . By associating clinically meaningful lesion descriptions with corresponding local visual patterns, medical vision-language models can help identify lesion-relevant regions and suppress confounding background responses [ 28 , 29 , 7 ] , thereby providing a feasible basis for automatic box prompt generation. However, if semantic cues are directly injected into the diagnostic classifier, the model may exploit label-correlated textual semantics rather than grounding its prediction in lesion morphology, internal texture, and peri-lesional context [ 30 , 31 ] . Such semantic shortcut learning may weaken the lesion-centered nature of diagnostic evidence. Even after lesion regions are localized and segmented, unified diagnosis remains challenged by the ambiguity of fine-grained lesion evidence [ 14 ] . In many clinical scenarios, diagnostically important differences are not reflected in obvious global appearance, but in subtle and localized variations in lesion morphology, internal texture, and surrounding context [ 32 ] , [ 33 ] , [ 34 ] , [ 17 ] . These cues are often weak, heterogeneous, and difficult to characterize reliably, even when coarse lesion localization is available [ 33 ] , [ 34 ] , [ 35 ] . As a result, coarse visual cues or weak semantic priors alone are usually insufficient for reliable diagnosis [ 17 ] , [ 36 ] , [ 37 ] . To address these challenges, we propose a two-stage cross-modal framework that uses semantic priors for prompt generation and visually grounded lesion representations for downstream diagnosis, as shown in Fig. 1 (b). In the first stage, a BLIP-2-based vision-language module is adapted to associate visual patterns with clinically meaningful lesion descriptions, allowing the model to identify lesion-related candidate regions under semantic guidance, reducing the need for manual prompt annotation at inference time. In the second stage, these prompts guide a SAM-based multitask branch to perform lesion segmentation and diagnosis, while avoiding direct fusion of raw text embeddings into the final diagnosis head. To further improve robustness, we introduce a multi-candidate region aggregation (MCRA) strategy to stabilize lesion evidence under prompt variation, and a cross-modal evidence fusion head that combines deep visual features, selected radiomics descriptors, and region-level contextual information for diagnosis. In this way, our framework couples segmentation and diagnosis through lesion-grounded evidence, while reducing the need for manual prompt annotation at inference time, mitigating the impact of imperfect prompt generation, and improving fine-grained lesion characterization. The main contributions of this work are summarized as follows: • We propose Rad-VLSM, a semantics-assisted vision-language segmentation and diagnosis framework for medical image analysis. It leverages BLIP-2-based vision-language alignment to associate lesion-related visual patterns with clinically meaningful semantic descriptions, thereby enabling automatic lesion localization and box prompt generation without manual prompt annotation at inference time. • We introduce a MCRA strategy to improve the stability of lesion evidence under prompt variation and coarse localization errors. This strategy produces more reliable lesion-level representations and enhances the robustness of unified diagnosis. • We present a lesion-mask-guided diagnosis scheme with visual-radiomics fusion. By integrating implicit visual representations with explicit radiomic features, this scheme improves diagnostic reliability and interpretability. • We rigorously evaluate our framework on a private clinical breast ultrasound dataset provided by Peking Union Medical College Hospital, alongside several public benchmarks. Extensive experiments demonstrate stable and competitive performance in both segmentation and

Read the complete text