MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation
Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy, Min Hun Lee, [0.5em] Singapore Management University · 2026
Read MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation by Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy, Min Hun Lee, [0.5em] Singapore Management University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17325.txt.
Introduction
Tran and co-authors present MedSAD-CLIP, a supervised adaptation of CLIP for medical anomaly detection and segmentation. CLIP based detectors often work with global features and weak supervision, giving coarse localization. Their model uses token patch cross attention to exploit fine grained text and visual cues for sharper lesion maps. Light adapters and learnable prompt tokens adapt the encoder to medicine while keeping its semantic alignment. Across brain, retina, lung, and breast benchmarks, the model beat prior state of the art methods.
Catalog record
| CREATOR | Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy, Min Hun Lee, [0.5em] Singapore Management University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47595 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17325
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy, Min Hun Lee, [0.5em] Singapore Management University Abstract Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis by identifying abnormal regions in medical images and localizing pathological regions. Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings, and typically rely on global representations and weak supervision, often producing coarse localization and limited segmentation quality. In this work, we study supervised adaptation of CLIP for MAD under a realistic clinical setting where a limited yet meaningful amount of labeled abnormal data is available. Our model MedSAD-CLIP leverages fine-grained text-visual cues via the Token-Patch Cross-Attention (TPCA) to improve lesion localization while preserving the generalization capability of CLIP representations. Lightweight image adapters and learnable prompt tokens efficiently adapt the pretrained CLIP encoder to the medical domain while preserving its rich semantic alignment. Furthermore, a Margin-based image-text Contrastive Loss is designed to enhance global feature discrimination between normal and abnormal representations. Extensive experiments on four diverse benchmarks-Brain, Retina, Lung, and Breast datasets-demonstrate the effectiveness of our approach, achieving superior performance in both pixel-level segmentation and image-level classification over state-of-the-art methods. Our results highlight the potential of supervised CLIP adaptation as a unified and scalable paradigm for medical anomaly understanding. Code will be made available at https://github.com/thuy4tbn99/MedSAD-CLIP 1 Introduction Figure 1 : (a) Qualitative result comparisons of few-shot MVFA and our supervised MedSAD-CLIP on a Breast image. (b) Radar comparison of MedSAD-CLIP and zero-/few-shot CLIP-based SOTAs on Brain and Breast datasets. Anomaly detection (AD) plays an important role in medical image analysis, providing critical insights that assist clinicians in achieving accurate and timely diagnoses 22 . However, medical AD remains challenging due to the ambiguous definition of abnormality and the diverse manifestations of pathological patterns across modalities. In practice, AD must support two coupled outcomes (i.e., image-level anomaly detection and precise localization of lesions). Recently, zero-/few-shot methods 19 ; 49 ; 11 ; 27 ; 5 leverage pre-trained vision-language models (VLMs) with carefully designed textual prompts and adaptation schemes to enhance image-level anomaly detection. While these approaches appeal due to their scalability, they may yield coarse segmentation maps with low overlap with ground-truth lesions and exhibit poor calibration across domains, limiting reliability in practice (as shown in Fig. 1 with low Dice score). Importantly, this limitation is not solely attributed to the zero-/few-shot setting itself. Rather, it is closely related to the inherent design of CLIP, where the dominance of global representations tends to weaken the semantic correspondence between individual image patches and textual descriptions 35 . Despite the widespread adoption of CLIP in zero-/few-shot anomaly detection, limited attention has been given to supervised CLIP-based frameworks for medical anomaly detection and segmentation. In many realistic clinical scenarios, a limited yet meaningful amount of labeled abnormal data is available, providing an opportunity to further exploit CLIP representations for more reliable localization while preserving the generalization benefits of vision-language pretraining. To better understand the role of supervision in CLIP-based medical anomaly analysis, we study supervised CLIP adaptation and compare it with existing zero-/few-shot VLM approaches under realistic clinical settings where a limited yet meaningful amount of labeled abnormal data is available. Firstly, Token-Patch Cross-Attention is employed to establish fine-grained, direct interactions between individual text tokens and image patches, replacing the reliance on a global sentence embedding and enabling sharper lesion boundary extraction. Unlike existing work using dot-product heatmaps directly as masks, TPCA forms token-conditioned features and concatenates them with rich image features before decoding, injecting granular language cues while preserving visual detail and sharpening lesion boundaries. Secondly, we integrate multi-level image adapters and a learnable text prompt module into the pretrained CLIP 17 ; 46 . Image adapters transfer general-domain representations to medical imaging, while the learnable prompt module further facilitates adaptation to various clinical contexts, anatomical regions, and imaging modalities. Lastly, we propose Margin-based image-text Contrastive Loss (MC-Loss), which explicitly enforces a larger semantic distance between normal and abnormal image-text pairs, guiding the model to learn a discriminative boundary and thereby improving image-level anomaly detection. Moreover, this loss enhances both the image adaptation module and the learnable prompt tuning, enabling them to align visual-textual representations more effectively. In summary, our main contributions are: • We empirically quantify and reveal a significant performance gap within the CLIP-based paradigm itself, comparing zero-/few-shot models against fully supervised counterparts in medical anomaly detection and segmentation. We therefore explore a supervised CLIP-based framework designed to enhance abnormal region segmentation across diverse datasets. • To better exploit fine-grained visual-text information beyond the global representations typically emphasized in CLIP-based models, we introduce a Token-Patch Cross-Attention (TPCA) module that sharpens lesion boundaries and improves segmentation, together with a Margin-based Contrastive Loss (MC-Loss) to enhance discrimination in ambiguous cases. • Our MedSAD-CLIP achieves state-of-the-art performance across multiple medical anomaly detection benchmarks, showing robust and consistent generalization. 2 Related Work Figure 2 : Overall architecture of our proposed MedSAD-CLIP. It consists of two branches: the classification branch (Det) and the segmentation branch (Seg). The model extends CLIP 30 with adapter modules and learnable prompts to adapt to medical image analysis. 2.1 Medical Anomaly Detection Existing medical anomaly detection approaches can be broadly categorized into unsupervised and supervised methods. Unsupervised methods are widely adopted in the medical field, where models are trained exclusively on normal samples and identify anomalies as deviations from the learned distribution of healthy data. Traditional techniques 26 ; 48 ; 51 ; 7 ; 52 often use reconstruction to learn prior information from normal data. For example, WeakAnD 33 leverages Bayes U-Net 32 , which is trained on healthy data to model epistemic uncertainty as an indicator of anatomical deviations, while Han et al. proposed MADGAN 13 , based on GAN 12 with adjacent slices, which enables better exploitation of 3D data to classify anomalies on brain MRI data. More recently, SQUID 42 introduces a teacher–student framework with a memory queue for in-painting-based AD, while DDPM 40 and AnoDDPM 41 employ denoising diffusion strategies to reconstruct healthy-looking tissue while preserving normal anatomical details. However, in real-world medical scenarios, a non-trivial portion of anomalous data is typically available during dataset collection 3 . Effectively leveraging these abnormal cases can provide valuable supervisory signals, enabling the model to enhance its predictive capability beyond merely learning from normal patterns. DRA 9 learns disentangled representations of both seen and unseen abnormalities, while BGAD 44 introduces a supervised anomaly detection framework that employs an explicit boundary–guided semi–push–pull contrastive learning mechanism to enhance feature discriminability. In contrast, we cast supervised anomaly understanding in a vision–language model and show how token-level textual cues and margin-based calibration jointly improve localization and decision stability. 2.2 Medical Vision Language Model Recently, a growing line of research 47 ; 14 ; 22 ; 25 has adapted VLMs 20 ; 30 ; 45 to the medical domain, aiming to leverage their powerful vision–language alignment for cross-modal understanding of medical images. LViT 23 integrates short disease descriptions with CT or X-ray image information through a lightweight embedding-based text representation and a hybrid CNN–Transformer architecture with pixel-level attention for segmentation. When using pre-trained VLMs like CLIP, it is essential