Stable Vision Concept Transformers for Medical Diagnosis

Lijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, Di Wang · 2025

Read Stable Vision Concept Transformers for Medical Diagnosis by Lijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, Di Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05286.txt.

Introduction

Lijie Hu and co-workers stabilized concept-based medical image models in a 2025 paper. Concept narrow point models route a diagnosis through human-readable concepts, like lesion shape or density, so doctors can audit the thinking. Forcing all information through concepts usually costs accuracy and the explanations wobble when inputs are slightly perturbed. Their Vision Concept Model fuses concept traits with the image's own feature embeddings inside a vision model. Recovering the lost utility instead of choosing between them.

A stabilized successor adds tool set to keep explanations faithful under perturbation, the paper's second concern. The design targets the twin demands of medical AI: results a health worker can use and explanations a health worker can trust twice, on the same image, slightly changed.

Catalog record

CREATORLijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, Di Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stable Vision Concept Transformers for Medical Diagnosis Lijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, Di Wang Abstract Transparency is a paramount concern in the medical field, prompting researchers to delve into the realm of explainable AI (XAI). Among these XAI methods, Concept Bottleneck Models (CBMs) aim to restrict the model’s latent space to human-understandable high-level concepts by generating a conceptual layer for extracting conceptual features, which has drawn much attention recently. However, existing methods rely solely on concept features to determine the model’s predictions, which overlook the intrinsic feature embeddings within medical images. To address this utility gap between the original models and concept-based models, we propose V ision C oncept T ransformer ( VCT ). Furthermore, despite their benefits, CBMs have been found to negatively impact model performance and fail to provide stable explanations when faced with input perturbations, which limits their application in the medical field. To address this faithfulness issue, this paper further proposes the S table V ision C oncept T ransformer ( SVCT ) based on VCT, which leverages the vision transformer (ViT) as its backbone and incorporates a conceptual layer. SVCT employs conceptual features to enhance decision-making capabilities by fusing them with image features and ensures model faithfulness through the integration of Denoised Diffusion Smoothing. Comprehensive experiments on four medical datasets demonstrate that our VCT and SVCT maintain accuracy while remaining interpretability compared to baselines. Furthermore, even when subjected to perturbations, our SVCT model consistently provides faithful explanations, thus meeting the needs of the medical field. * * footnotetext: Equal Contribution. $\dagger$ $\dagger$ footnotetext: Corresponding Author. 1 Introduction As the field of medical image analysis continues to evolve, deep learning models and methods have demonstrated excellent performance in tasks such as image recognition and disease diagnosis [ 29 ] . However, these advanced deep learning models are usually regarded as black boxes and lack credibility and transparency. Especially in the medical field, this opacity makes it difficult for physicians and clinical professionals to trust the predictions of the models. Thus, the requirement for interpretability of model decisions is more urgent in the medical field [ 17 , 24 , 25 ] . The healthcare field, characterized by stringent requirements for trustworthiness, necessitates models that not only exhibit high performance but are also comprehensible and can be trusted by practitioners. Therefore, Explainable Artificial Intelligence (XAI) has become one of the hotspots for research and development. By introducing interpretability, XAI tries to make the decision-making process of deep learning models more transparent and understandable. Some compelling interpretable methods, such as attention mechanisms [ 51 , 23 ] , saliency maps [ 60 ] , DeepLIFT and Shapley values [ 37 ] , and influence functions [ 31 , 22 ] , attempt to provide users with visual explanations about model decisions. However, while these post-hoc explanatory methods can provide useful information, there is still a certain disconnect between their explanations and model decisions, and these explanations are generated after model training and fail to participate in the model learning process. Some studies [ 46 , 33 , 16 ] have shown that post-hoc is sensitive to slight changes in the input, making the post-hoc methods misleading as they could provide explanations that do not accurately reflect the model’s decision-making process. Figure 1 : An example of VCT framework on OCT2017 dataset [ 29 ] . The leftmost figure displays the input image, while the adjacent one on the left shows the concept output without perturbations. In contrast, the figure on the right presents the concept output after applying input perturbations, resulting in noticeable changes. Therefore, researchers have shown interest in self-explained methods. Among them, concept-based methods have attracted a lot of attention. These approaches strive to incorporate interpretability into machine learning models by establishing connections between their predictions and concepts that are understandable to humans. As an illustration, the Concept Bottleneck Model (CBM) [ 32 ] initially forecasts an intermediate set of predefined concepts, subsequently utilizing these concepts to make predictions for the final output. [ 43 ] introduce Label-free CBM, a novel framework designed to convert any neural network into an interpretable CBM without the need for labeled concept data compared to the original CBM. These inherently interpretable methods provide concept-based explanations, which are generally more comprehensible than post-hoc approaches. However, many existing methods rely solely on concept features to determine the model’s predictions. These approaches overlook the intrinsic feature embeddings within medical images. For instance, [ 48 ] solely utilizes concept labels to supervise the concept prediction results of the entire image. This oversight can lead to a decrease in classification accuracy, which is suggested to stem from the inefficient utilization of valuable medical information. Therefore, a significant challenge in the field of medical imaging is how to maintain a high level of accuracy while incorporating interpretability. To address the aforementioned challenges, we propose Vision Concept Transformer (VCT), a novel medical image processing framework that is interpretable and maintains high performance. Vision Transformers (ViTs) [ 5 ] have achieved state-of-the-art performance for various vision tasks, showing good robustness in prediction. Thus, in the VCT framework, we utilize ViTs as the foundational network. To enhance interpretability, we employ a label-free methodology for generating the conceptual layer. Moreover, unlike previous CBMs, which only use conceptual features for prediction, in the VCT framework, we integrate conceptual features with image features, utilizing the conceptual layer as supplementary information to augment decision-making. This integration effectively addresses the issue of accuracy degradation associated with a singular label-free CBM, ensuring interpretability without compromising accuracy. While VCT keeps the interpretability of CBMs, it also inherits their interpretability instability when facing perturbations or noise in the input. Specifically, adding slight noise to the input image can significantly change the top- k important concepts given by CBMs (see Figure 1 for an example), i.e., the top k -indices of the concept vector. Instability is a common issue in deep learning interpretation methods, making it challenging to understand model reasoning [ 19 ] , especially with unlabeled data and self-supervised training [ 12 ] . As in real medical scenarios, there is always natural and inherent noise or some adversarial examples manipulated by attackers [ 2 , 11 , 52 ] . Thus, VCT cannot be a faithful explainable tool for these applications. To address the faithfulness issue, by using the Denoised Diffusion Smoothing method, we can smoothly and directly transform VCT into a Stable Vision Concept Transformer (SVCT) framework that is capable of providing stable interpretations despite perturbations to the inputs, the structure is shown in Figure 2 . Our contributions can be summarised as follows. • We proposed the VCT framework, transforming ViTs into an interpretable CBM. VCT integrates conceptual features with image features, utilizing conceptual features as auxiliary decision-making components. This effectively addresses the performance degradation issue in existing CBMs due to inefficient utilization of medical information. • To further enhance the interpretability stability of VCT, we propose a formal mathematical definition of an SVCT, which ensures that the top- k index of its conceptual vectors remains relatively stable under slight perturbations. We utilize a Denoised Diffusion Smoothing (DDS) method to obtain an SVCT. Moreover, we theoretically proved that

Read the complete text