XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Models Thanks: Corresponding author. Email: muzammil.behzad@kfupm.edu.sa Thuraya Alzubaidi, Sana Ammar, Maryam Alsharqi, Islem Rekik, Muzammil Behzad Abstract Medical image segmentation is a critical component of clinical workflows, enabling accurate diagnosis, treatment planning, and disease monitoring. However, despite the superior performance of transformer-based models over convolutional architectures, their limited interpretability remains a major obstacle to clinical trust and deployment. Existing explainable artificial intelligence (XAI) techniques, including gradient-based saliency methods and perturbation-based approaches, are often computationally expensive, require numerous forward passes, and frequently produce noisy or anatomically irrelevant explanations. To address these limitations, we propose XAI-CLIP, an ROI-guided perturbation framework that leverages multimodal vision-language model embeddings to localize clinically meaningful anatomical regions and guide the explanation process. By integrating language-informed region localization with medical image segmentation and applying targeted, region-aware perturbations, the proposed method generates clearer, boundary-aware saliency maps while substantially reducing computational overhead. Experiments conducted on the FLARE22 and CHAOS datasets demonstrate that XAI-CLIP achieves up to a 60% reduction in runtime, a 44.6% improvement in dice score, and a 96.7% increase in Intersection-over-Union for occlusion-based explanations compared to conventional perturbation methods. Qualitative results further confirm cleaner and more anatomically consistent attribution maps with fewer artifacts, highlighting that the incorporation of multimodal vision-language representations into perturbation-based XAI frameworks significantly enhances both interpretability and efficiency, thereby enabling transparent and clinically deployable medical image segmentation systems. I Introduction Medical image segmentation is a fundamental part of clinical workflows, allowing clinicians to assess anatomical boundaries and pathological structures for decision making in critical areas such as oncology, neurology, and cardiology [ 11 ] . Recent segmentation models that leverage transformer-based architectures far surpass traditional CNNs by effectively modeling long-range dependencies and capturing both global and local features, making them particularly effective for medical image segmentation tasks [ 32 ] . However, the transparency of their decision-making processes remains ambiguous. A recent study found that about 76% of medical imaging competitions failed to provide labeling instructions, underscoring the lack of clarity in how AI models are trained and validated, particularly for complex modalities like multiparametric MRI and CT scans [ 44 ] . This makes interpretability challenges a significant barrier for clinical adoption. Consequently, there is a need for explainable AI frameworks that can enhance transparency and unlock the black-box nature of these models [ 30 ] , allowing clinicians to understand and trust AI decisions and deploy such models responsibly. In this context, the current medical XAI methods face two key challenges. Gradient-based approaches, such as, Grad-CAM, generate saliency maps efficiently but occasionally highlight low-level features misaligned with anatomical structures [ 8 ] . On the other hand, perturbation-based methods iteratively modify inputs to quantify feature importance [ 1 ] , yet their exhaustive image-wide sampling incurs high computational costs as they apply perturbations even to regions outside of anatomically relevant structures, requiring hundreds of forward passes for a single explanation. To address these gaps, we propose XAI-CLIP, an ROI-guided perturbation framework leveraging contrastive vision-language modeling as shown in Fig 1 . By constraining perturbations to anatomically salient regions using the multimodal embeddings, our framework minimizes computational overhead while ensuring clinically meaningful explanations. Specifically, our framework utilizes a multimodal architecture that embeds visual and textual information in a shared representational space. The model enables zero-shot classification by measuring visual-semantic alignment between images and textual descriptions. These features can be used to extract clinically significant relationships, producing more meaningful saliency maps that align with anatomical structures and pathological regions of interest. Fig. 1: Overview of the XAI-CLIP framework. Text and image inputs are encoded using vision and text encoders. A joint embedding space guides region-restricted perturbation-based XAI, producing anatomically aligned explainability maps. The primary segmentation backbone employed in this study is MedSAM (Medical Segment Anything Model) [ 32 ] , a transformer-based framework derived from the Segment Anything Model (SAM) [ 27 ] and adapted for general-purpose medical image segmentation across multiple imaging modalities, including MRI, CT, and ultrasound. In this work, MedSAM serves as a representative high-performing segmentation model to evaluate the proposed explainability framework rather than as a central contribution of the study. The focus of the investigation remains on assessing the effectiveness and generality of the proposed XAI-CLIP approach in enhancing the interpretability of medical image segmentation outputs. To this end, both qualitative and quantitative analyses are conducted using samples drawn from established benchmark datasets, namely FLARE22 [ 33 ] and CHAOS [ 25 ] , which provide diverse anatomical structures and imaging characteristics suitable for evaluating explanation quality. The explanatory performance of XAI-CLIP is systematically compared against existing perturbation-based explainability methods to examine its ability to generate anatomically meaningful and visually coherent attribution maps. Rather than emphasizing experimental configurations, the discussion centers on the methodological advantages introduced by region-guided perturbations, which enable more focused and semantically aligned explanations. The proposed framework demonstrates a consistent ability to reduce unnecessary perturbations and suppress redundant activations, leading to clearer saliency maps with improved boundary localization and fewer visual artifacts. We present detailed quantitative evaluations and computational efficiency analyses, conducted under different hardware configurations, later in the paper, where the proposed method is shown to offer substantial runtime improvements over conventional perturbation strategies without compromising explanation fidelity. II Related Work With the emergence of the Segment Anything Model (SAM), a large-scale foundation model for image segmentation that enables prompt-driven and zero-shot generalization [ 17 , 22 ] , significant research attention has been directed toward its adaptation for medical imaging applications [ 35 ] . SAM’s architecture, composed of a high-capacity image encoder, a flexible prompt encoder, and a lightweight mask decoder, provides a generalizable framework that can be repurposed across diverse segmentation tasks. Its promptable design has motivated extensive exploration into leveraging foundation models for medical image segmentation, where data scarcity and modality heterogeneity remain persistent challenges. As an extension of SAM, MedSAM [ 32 , 35 ] represents a notable adaptation that is tailored to the medical domain through large-scale fine-tuning on approximately 1.57 million image-mask pairs spanning over ten imaging modalities and more than thirty cancer types. This broad training strategy enables MedSAM to generalize across modalities ranging from computed tomography to dermoscopy without relying on task-specific model variants. Empirical evaluations demonstrate strong segmentation performance across multiple benchmarks, with substantial improvements over conventional convolutional and transformer-based baselines. Nevertheless, the reliance on bounding-box prompts introduces limitations in anatomically complex scenarios, particularly in cases involving branching or overlapping vascular structures, where a single bounding box may inadequately capture fine-grained spatial distinctions. More recently, SemiSAM+ has been introduced as a semi-supervised medical image segmentation framework that integrates SAM with task-specific segmentation networks [ 55 ] . Rather than retraining the frozen SAM encoder, the method exploits SAM-generated priors to guide pseudo-label propagation, followed by refinement using downstream architectures. This strategy demonstrates strong performance in low-annotation regimes, achieving competitive Dice scores with only a small number of labeled samples on benchmark datasets [ 55 ] . SemiSAM+ reflects a broader trend toward coupling foundation models with lightweight adaptation mechanisms, aligning with recent approaches that employ adapter-based tuning for clinical deployment [ 51 , 47 ] . Despite its advantages, the framework remains sensitive to prompt quality, with segmentation accuracy degrading in regions with ambiguous or weak anatomical boundaries. Complementary efforts have also explored knowledge distillation strategies to further enhance semi-supervised segmentation performance [ 21 ] . Beyond foundation model adaptations, alternative architectural paradigms have been investigated to improve segmentation efficiency and scalability. Ruan et al. proposed VM-UNet, a segmentation network based entirely on state space models, replacing conventional convolutional and self-attention blocks with Vision State Space modules derived from Mamba [ 47 ] . The resulting architecture significantly reduces computational complexity while maintaining competitive accuracy, achieving strong Dice scores across multiple datasets and outperforming transformer-based counterparts such as Swin-UNet. This work highlights the potential of state space models as an efficient alternative to attention-based architectures in medical image segmentation. In a related direction, Wu et al. introduced Med-SAM, an adapter-based framework that augments a frozen SAM backbone with lightweight bottleneck modules to enable effective medical image segmentation [ 50 ] . The proposed Space-Depth Transpose mechanism facilitates the extension of a 2D vision transformer encoder to volumetric data, enabling efficient 3D adaptation with a limited number of trainable parameters. While Med-SAM achieves strong segmentation accuracy with substantially fewer tunable weights compared to transformer-based baselines, its performance remains sensitive to prompt design, with notable degradation observed under reduced prompt overlap. Complementing these efforts, UltraLight VM-UNet further demonstrates the feasibility of highly compact state space-based architectures, achieving competitive performance on skin lesion segmentation tasks with minimal parameter and computational budgets [ 52 ] . Together, these studies underscore ongoing efforts to balance segmentation accuracy, computational efficiency, and adaptability in modern medical image segmentation models. II-A Explainable AI in Medical Image Analysis Explainable artificial intelligence has received increasing attention in medical imaging, driven by the need for transparency and accountability in clinical decision-making. In this regard, Fontes et al. [ 14 ] presented a domain-specific taxonomy that systematically categorizes XAI techniques according to imaging modalities and associated clinical tasks. Their review analyzed nine representative studies published between 2021 and 2023, encompassing a range of explainability paradigms, including prototype-based, retrieval-based, counterfactual, and generative adversarial approaches. While the study provides valuable conceptual organization of existing methods, it does not include quantitative performance aggregation and limits its scope to English-language publications within a relatively narrow temporal window. Addressing limitations in earlier surveys, Hou et al. [ 20 ] presented a comprehensive review of self-explainable artificial intelligence (S-XAI) methods for medical image analysis, encompassing more than 200 studies published between 2018 and 2024. The authors structured the literature along three complementary dimensions, input-level, model-level, and output-level explainability, and reported a substantial increase in research activity, with medical imaging accounting for approximately half of all XAI-related publications. In addition, the survey compiled an extensive resource table covering over 70 publicly available datasets. However, while performance metrics were reported across studies, the absence of standardized experimental settings limits direct quantitative comparison. Building upon these conceptual frameworks, Iglesias et al. [ 24 ] proposed MOC-VAE, an explanation method that employs a SPEA2 optimizer to identify minimal perturbations in the latent space that induce misclassification. The proposed approach achieved a high degree of overlap between generated saliency maps and annotated pathological regions, surpassing several competing methods by a notable margin. Nevertheless, the underlying model exhibits relatively modest diagnostic accuracy compared to state-of-the-art chest X-ray classifiers, which constrains its clinical applicability despite improved explainability. From a theoretical standpoint, Abe and Asai [ 2 ] introduced a perturbation-based explanation framework that quantifies local input sensitivity via Jacobian matrix analysis. Their method enables instance-specific, data-free explanations that align with key clinical interpretability requirements. While the framework is mathematically well-founded, the study does not provide quantitative validation on medical imaging benchmarks, leaving its practical effectiveness largely unexplored. In a collaborative learning context, Mastoi et al. [ 34 ] developed an interpretable federated learning pipeline for brain tumor classification, training a GoogLeNet architecture on distributed MRI data while preserving patient privacy. The proposed ten-client framework achieved high classification accuracy across multiple tumor categories. Although Grad-CAM visualizations were incorporated into the inference process, the work did not include a quantitative evaluation of explanation quality or consistency, limiting insight into the reliability of the generated saliency maps. In domain-specific applications, Yoshida et al. [ 53 ] fine-tuned a foundation model for optical coherence tomography (OCT) image classification and employed relevance propagation to generate saliency maps. Their analysis led to the identification of previously unreported imaging biomarkers with high specificity. However, the attribution maps produced by the explanation method were not consistently aligned with pathological regions, suggesting susceptibility to over-attribution and reduced robustness in certain cases. Despite these advances across surveys, theoretical frameworks, collaborative learning, and specialty applications, many existing XAI approaches in medical imaging continue to operate primarily at the pixel level, without incorporating explicit semantic priors or anatomically meaningful region constraints. This lack of semantic grounding limits the interpretability and clinical relevance of generated explanations, particularly in complex medical imaging scenarios where contextual understanding is essential. II-B Vision-Language Models in Medical Images Vision-language models (VLMs) have demonstrated significant potential for medical image analysis through joint visual-textual representation learning [ 42 ] , exemplified by contrastive language-image pretraining (CLIP) [ 43 ] . In early foundational work, Poudel et al. [ 41 ] systematically evaluated vision-language segmentation models on 2D medical image segmentation tasks across eleven heterogeneous datasets. Their study showed that the model achieved strong segmentation performance, reaching a dice score of 92.40% on the BKAI dataset, while exhibiting substantially improved generalization compared to convolutional baselines, particularly in cross-dataset evaluations. Notably, the model maintained robust performance in out-of-distribution settings, significantly outperforming UNet when transferred between datasets. Despite these encouraging results, the authors observed limited utilization of textual semantics in CLIPSeg-based models, and unexpectedly weaker performance from BiomedCLIP variants, even when pretrained on large-scale medical image-text corpora [ 41 ] . To address the computational and parameter inefficiency of earlier VLM-based segmentation approaches, Dhakal et al. [ 12 ] introduced VLSM-Adapter, a parameter-efficient fine-tuning strategy based on lightweight adapters. Their approach reduced the number of trainable parameters to approximately three million while achieving performance comparable to full fine-tuning across multiple medical datasets. In particular, the CLIPSeg Dense Adapter variant demonstrated strong efficiency, outperforming SAN with significantly fewer parameters and remaining competitive with CRIS despite an order-of-magnitude reduction in trainable weights. However, these gains were accompanied by sensitivity to adapter placement, and performance gaps persisted on certain datasets when compared to fully fine-tuned models. Moving beyond single-model adaptations, Dietlmeier et al. [ 13 ] proposed an ensemble-based framework that combines CLIP-based VLMs with lightweight UNet architectures. Their approach integrates feature representations through concatenation and Efficient Channel Attention mechanisms, enabling complementary strengths between multimodal and convolutional components. This ensemble strategy yielded notable improvements in segmentation accuracy across several datasets, particularly for BiomedCLIPSeg-based variants. Nevertheless, the method exhibited modality-dependent variability, with certain datasets showing reduced performance relative to baseline VLM models, highlighting challenges in achieving consistent gains across diverse imaging domains. Rather than focusing solely on architectural refinements, Chen et al. [ 10 ] addressed reasoning deficiencies in VLM-based segmentation by introducing CausalCLIPSeg, which incorporates causal intervention mechanisms for referring medical image segmentation. Their framework improved both dice and mIoU scores over transformer-based baselines by enforcing causal consistency between textual queries and pixel-level predictions. While the cross-modal decoding strategy enhanced text-to-pixel alignment, its reliance on vision-language models pretrained exclusively on natural images constrained its ability to fully capture specialized medical terminology and domain-specific semantics. To strengthen cross-modal interaction, Sultan et al. [ 49 ] proposed BiPVL-Seg, a bidirectional fusion framework augmented with global-local contrastive alignment. This design enabled deeper information exchange between vision and language encoders while aligning representations at both concept and class levels. The resulting model achieved consistent performance improvements across multiple datasets, including higher dice scores and reduced Hausdorff distance. Complementarily, Chen et al. [ 9 ] reframed vision-language correspondence as a graph matching problem through Bi-VLGM, preserving intra-modal structural relationships and yielding substantial gains over strong convolutional and hybrid baselines on histopathology benchmarks. Recognizing the importance of domain knowledge, Nath et al. [ 38 ] introduced VILA-M3, which integrates expert medical knowledge via instruction-based fine-tuning. Their large-scale model achieved strong performance across medical question answering and classification benchmarks, outperforming existing medical VLMs by a significant margin. However, this specialization introduced trade-offs, including degraded performance on general-domain tasks and reliance on external expert systems, which may limit scalability and accessibility in real-world clinical settings. From a broader perspective, Hartsock and Rasool [ 16 ] provided a comprehensive survey of medical VLMs with a focus on report generation and visual question answering. Their analysis categorized a wide range of models and datasets while emphasizing the growing role of parameter-efficient fine-tuning techniques such as LoRA. Although several models achieved strong performance on radiology-centric benchmarks, the authors identified notable evaluation biases and highlighted underrepresentation of other clinically important domains, including pathology and surgical imaging. In parallel work related to segmentation, Liang et al. [ 31 ] examined vision foundation models for medical applications, with particular attention to efficient adaptations of SAM-based architectures. Their study demonstrated that lightweight variants, enabled through techniques such as low-rank adaptation, could deliver substantial inference speedups without compromising accuracy. Despite these engineering advances, the authors noted that many architectural modifications lack formal performance guarantees, while knowledge distillation methods often suffer from semantic mismatches between natural and medical image domains. Despite significant progress across architectural design, efficiency optimization, causal reasoning, and domain adaptation, medical VLMs continue to face persistent challenges that hinder clinical adoption. These include modality bias, limited coverage of diverse medical datasets, insufficient integration of spatial and semantic prompts, and high computational demands. Most critically, existing approaches often lack robust mechanisms for interpretable saliency or semantically grounded attribution, underscoring the need for explainability-aware frameworks that balance interpretability, efficiency, and clinical relevance in high-stakes medical decision-making environments. II-C CLIP in Medical Domain CLIP-based approaches have attracted growing interest in medical imaging due to their capacity for zero-shot transfer and multimodal representation learning. Zhao et al. [ 56 ] presented a comprehensive survey of CLIP adaptations for medical imaging, analyzing 224 studies and proposing a three-level taxonomy to categorize methodological trends. Their benchmark analysis demonstrated that domain-specialized CLIP variants, such as BiomedCLIP, achieved near-saturated accuracy across multiple imaging domains, substantially outperforming general-domain medical CLIP models. Despite its breadth, the survey exhibits a strong emphasis on two-dimensional imaging modalities and relies predominantly on a single evaluation metric, limiting insights into robustness and cross-modal generalization. In applied segmentation settings, Aleem et al. [ 4 ] proposed SaLIP, a zero-shot medical image segmentation framework that integrates SAM with CLIP-based region ranking. The method generates candidate masks using SAM, ranks them via vision–language similarity scores computed from automatically generated prompts, and subsequently refines segmentation through re-prompting. SaLIP demonstrated promising performance across several imaging modalities; however, its effectiveness depends critically on the ability of SAM to generate candidate masks that sufficiently cover the target anatomy. Furthermore, the limited spatial reasoning capacity of CLIP constrains its ability to reliably distinguish anatomically adjacent structures. To improve robustness and generalization across modalities, Khattak et al. [ 26 ] introduced UniMed-CLIP, a dual-encoder vision-language model trained on a large-scale, multimodal medical image-text corpus. By employing a multi-caption supervision strategy, the model achieved notable improvements in zero-shot performance across multiple datasets while requiring substantially less training data than comparable models. Although UniMed-CLIP demonstrates strong cross-dataset generalization, its reliance on caption-level supervision provides limited spatial grounding, which restricts its applicability for fine-grained localization and segmentation tasks. Focusing on few-shot anomaly detection, Zhang et al. [ 54 ] proposed MediCLIP, which adapts CLIP through learnable prompts, vision-side adapters, and synthetic anomaly generation. Their approach significantly improved both image-level and pixel-level anomaly detection performance under limited data regimes and demonstrated cross-dataset generalization without additional fine-tuning. Nonetheless, the use of fixed hyperparameters across heterogeneous modalities and the absence of standardized evaluation protocols limit the generalizability of the reported results. Integrating vision-language models directly with segmentation pipelines, Koleilat et al. [ 28 ] introduced MedCLIP-SAMv2, which combines fine-tuned vision-language representations with SAM-based mask refinement. Their framework leverages saliency-driven prompting to guide segmentation and achieves substantial gains in both retrieval and segmentation performance across multiple modalities. However, the method remains highly sensitive to prompt formulation, with segmentation accuracy varying considerably under minor changes in textual input, underscoring ongoing challenges in achieving stable and reliable vision-language-guided segmentation in medical imaging. A detailed overview of these methods is provided in Table I . TABLE I: Literature review summary of recent studies in medical imaging. Reference Authors Model / Framework Task Dataset(s) Key findings Accuracy of SAM in Medical Segmentation [ 18 ] He et al. (2023) SAM (semantic, point, box prompts) Medical image segmentation 12 public sets (e.g. BraTS, Kvasir, LiTS) SAM underperforms in zero-shot medical segmentation; lacks generalization without domain adaptation.7 SAM for Medical Images [ 23 ] Huang et al. (2023) SAM + medical‑tuned prompts Medical image segmentation BraTS, ACDC, Prostate, CHAOS Prompt tuning improves SAM, but still trails task-specific models on medical segmentation. Experimental Study on SAM for Medical Images [ 36 ] Mazurowski et al. (2023) SAM (point & box prompts) Medical image segmentation 19 public sets (e.g. spine MRI, hip X‑ray) IoU: 0.11–0.87 Segment Anything in Medical Images [ 32 ] Ma et al. (2024) MedSAM (fine‑tuned SAM) Medical image segmentation 1.57 M image–mask pairs across 10 modalities Outperforms SOTA in 146 tasks SemiSAM+: Semi‑Supervised Segmentation via Foundation Models [ 55 ] Zhang et al. (2025) SemiSAM+ (SAM + specialist) Semi‑supervised segmentation Two public sets, one clinical set Significant improvement with limited annotations Medical SAM Adapter for Segmentation [ 52 ] Wu et al. (2023) Medical SAM Adapter (Med‑SA) Medical image segmentation 17 tasks across various modalities Outperforms SOTA while updating only 2 % of parameters SAM‑Induced Prompt Distillation for Semi‑Supervised Segmentation [ 21 ] Huang et al. (2025) KnowSAM (SAM + LPS + SKD) Semi‑supervised segmentation Multiple public datasets Outperforms SOTA in semi‑supervised settings VILA‑M3: Expert‑Guided VLMs [ 39 ] Nath et al. (2025) VILA‑M3 Multi‑task med. VLM RadQA, MIMIC‑CXR, SLAKE, PathVQA, CheXpert, ChestX‑ray14 Avg. +9 % over Med‑Gemini VLMs for Report Generation & VQA (Review) [ 16 ] Hartsock et al. (2024) Various VLMs Report generation, VQA MIMIC-CXR, IU-Xray, MedICaT, VQA-RAD, PathVQA, VQA-Med 2020 Performance varies across tasks; challenges in clinical validity and interpretability Vision Foundation Models in Medical Imaging [ 31 ] Liang et al. (2025) Various (e.g. SAM, ViT) Medical image tasks Multiple public sets Near‑SOTA accuracy with few labels; robustness unresolved CLIP in Medical Imaging - Survey [ 57 ] Zhao et al. (2025) CLIP Various medical tasks Many public datasets Contrastive pre‑training enables strong zero‑shot transfer Test‑Time Adaptation with SaLIP [ 5 ] Aleem et al. (2024) SaLIP (SAM + CLIP) Zero‑shot segmentation Brain MRI, Chest X‑ray, Fetal US Dice: 63.46 % (brain), 50.11 % (lung), 30.82 % (fetal head) UniMed‑CLIP for Diverse Modalities [ 26 ] Khattak (2025) et al. UniMed‑CLIP Multi‑modal imaging [ 26 ] UniMed (5.3 M pairs) +12.61 % over BiomedCLIP MediCLIP: Few‑shot Anomaly Detection [ 54 ] Zhang et al. (2024) MediCLIP (learnable prompts + adapters) Few‑shot anomaly detection CheXpert, BrainMRI, BUSI AUROC: +10 % over SOTA MedCLIP‑SAMv2 for Text‑Driven Segmentation [ 28 ] Koleilat et al. (2025) MedCLIP‑SAMv2 Text‑driven segmentation Breast US, Brain MRI, Lung X‑ray/CT Dice: +13.07 % (zero‑shot), +11.21 % (weakly supervised) III The Propose XAI-CLIP Method III-A Preprocessing Our preprocessing pipeline transforms raw medical images into standardized formats suitable for our proposed pipeline. The process begins with standard input handling, including directory reading and format verification. Images are resized to 224×224 pixels using area interpolation to maintain aspect ratio, followed by grayscale conversion using the BT.601 luminance standard when necessary [ 46 ] . One of the core contribution of our pipeline is the adaptive contrast enhancement technique that adjusts processing parameters based on the input image characteristics. We first separate the foreground from the background using intensity thresholding: \text{Mask}(x,y)=\begin{cases}1,&\text{if }I(x,y)0.1 >0.1 131,921 420 283.7 XAI-CLIP LIME 0.412 0.297 94,431 311 223.7 \Delta_{\text{XAI-CLIP}} — — +28.4% +26.0% +21.1% Traditional RISE >0.1 >0.1 181,010 1,040 533.3 XAI-CLIP RISE >0.1 >0.1 169,932 753 803 \Delta_{\text{XAI-CLIP}} — — +6.1% +27.6% +33.6% Notably, XAI-CLIP-LIME successfully identified anatomical structures even in scenarios where the baseline method completely failed. In instances where both methods achieved reasonable localization, XAI-CLIP-LIME maintained significant heatmap quality, as shown in Fig 12 , while simultaneously reducing computational requirements by 28\% in GFLOPs and approximately 25\% in runtime. However, the consistently poor baseline performance (below 0.1 in many cases) precluded meaningful percentage improvement calculations for this method. Similarly, RISE-based analysis revealed a counterintuitive yet consistently observed behavior in which regions assigned the lowest importance, corresponding to the darkest areas in the saliency maps, aligned with the true organ locations, while importance progressively increased toward surrounding background tissue as demonstrated in Fig 13 . As a result, the generated saliency masks emphasized neighboring structures and background regions rather than the target organs, leading to dice and IoU scores below 0.1 for both the baseline and XAI-CLIP variants due to minimal overlap with ground-truth boundaries. While traditional RISE occasionally produced unstable or image-wide attributions, XAI-CLIP-RISE preserved the same inverted importance trend in a more stable and structured manner, while also achieving a runtime reduction of 287 seconds, corresponding to a 28% improvement. (a) (b) Fig. 11: (a) Distribution of Dice values for Traditional LIME vs. XAI-CLIP. (b) Distribution of IoU values for Traditional LIME vs. XAI-CLIP. In summary, we note that while the dice coefficient measures segmentation accuracy through spatial overlap with ground-truth masks, it is inherently sensitive to organ size, which accounts for the lower dice scores observed for smaller structures such as the spleen and pancreas. In contrast, AUC-ROC evaluates the model’s ability to assign higher confidence to true organ pixels relative to background without thresholding probabilistic outputs, making it particularly informative for assessing region-level confidence even when boundary delineation is imperfect. The computational efficiency gains achieved by the proposed framework arise directly from restricting perturbations to anatomically relevant regions, thereby avoiding unnecessary evaluations over background areas. Notably, the observed behavior in RISE-based explanations suggests that the segmentation model leverages contextual and boundary information surrounding organs more heavily than intra-organ texture cues; consequently, masking interior regions has limited impact on predictions, leading to low attributed importance for clinically relevant interiors despite their diagnostic significance. Fig. 12: LIME visualizations across different processing stages (a–f). VI Conclusions Explainable artificial intelligence remains a fundamental requirement for the safe and trustworthy deployment of medical image segmentation systems in clinical practice. In this work, we presented XAI-CLIP, a region-guided explainability framework that integrates vision-language representations, prompt-based learning, and segmentation models to enhance perturbation-based explanations. By leveraging semantically informed region localization, the proposed approach produces clearer, anatomically meaningful attribution maps while significantly reducing the computational cost associated with conventional perturbation strategies. Experimental results demonstrate that XAI-CLIP improves both interpretability and efficiency across multiple explainability methods and datasets. Future work will explore further optimization of the framework and its validation on large-scale, real-world clinical datasets to support robust and clinically deployable explainable medical imaging systems. Fig. 13: RISE visualizations across different processing stages (a–d).