CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification
Cristiano Patrício, Isabel Rio-Torto, Jaime S. Cardoso, Luís F. Teixeira, João C. Neves · 2025
Read CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification by Cristiano Patrício, Isabel Rio-Torto, Jaime S. Cardoso, Luís F. Teixeira, João C. Neves (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12266.txt.
Introduction
Cristiano Patrício and co-authors present a training free method for medical image classification in a 2025 paper. Deep learning tools for medicine face two barriers: labeled data are scarce, and their decisions are hard to explain. Concept based models improve explanations but need heavy labeling, and adding a new concept means retraining.
Their method uses a large vision language model in two steps. First it asks whether each named concept is present in the image. Then it asks the model to classify the image based on those concept answers, with a retrieval step picking helpful examples for each prompt. Tests across four medical data sets and twelve models beat concept based rivals and task specific trained methods. The method needed only a few labeled examples and no training.
Catalog record
| CREATOR | Cristiano Patrício, Isabel Rio-Torto, Jaime S. Cardoso, Luís F. Teixeira, João C. Neves |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12266
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification Cristiano Patrício, Isabel Rio-Torto, Jaime S. Cardoso, Luís F. Teixeira, João C. Neves Abstract The main challenges limiting the adoption of deep learning-based solutions in medical workflows are the availability of annotated data and the lack of interpretability of such systems. Concept Bottleneck Models (CBMs) tackle the latter by constraining the model output on a set of predefined and human-interpretable concepts. However, the increased interpretability achieved through these concept-based explanations implies a higher annotation burden. Moreover, if a new concept needs to be added, the whole system needs to be retrained. Inspired by the remarkable performance shown by Large Vision-Language Models (LVLMs) in few-shot settings, we propose a simple, yet effective, methodology, CBVLM , which tackles both of the aforementioned challenges. First, for each concept, we prompt the LVLM to answer if the concept is present in the input image. Then, we ask the LVLM to classify the image based on the previous concept predictions. Moreover, in both stages, we incorporate a retrieval module responsible for selecting the best examples for in-context learning. By grounding the final diagnosis on the predicted concepts, we ensure explainability, and by leveraging the few-shot capabilities of LVLMs, we drastically lower the annotation cost. We validate our approach with extensive experiments across four medical datasets and twelve LVLMs (both generic and medical) and show that CBVLM consistently outperforms CBMs and task-specific supervised methods without requiring any training and using just a few annotated examples. More information on our project page: https://cristianopatricio.github.io/CBVLM/ . Figure 1 : Comparison between the proposed methodology ( CBVLM ) with existing methods. Unlike traditional approaches, our methodology grounds the final diagnosis on a set of clinical concepts predicted by the LVLM itself ( cf. Concept Detector ), thus increasing classification performance, while being interpretable. 1 Introduction The field of medical imaging analysis has witnessed significant advancements through the use of supervised deep learning methods. However, their performance greatly depends on the availability and quality of training samples, which are particularly costly to obtain, as clinician expertise is required. The recent rise of Large Vision-Language Models (LVLMs) has also made its way into the medical field, of which Med-LVLMs such as [ 1 , 2 , 3 , 4 ] are prominent examples, where pre-trained LVLMs are fine-tuned using medical data. However, despite the improved performance, these Med-LVLMs require even more annotated data. Consequently, a different research direction has emerged, exploring the potential of applying LVLMs to specific downstream tasks with minimal or no supervision, by leveraging their few-shot capabilities [ 5 , 6 , 7 ] , particularly the use of in-context learning (ICL) for improving model performance through a short number of examples in the prompt. In addition to the need for annotated data, another major challenge with deep learning-based medical imaging is the lack of interpretability, which is of the utmost importance in high-stakes decision-making scenarios [ 8 ] . One way of introducing interpretability into such frameworks is via concept-based explanations. Concept Bottleneck Models (CBMs) provide this type of explanations by introducing a layer, the “bottleneck”, that predicts the presence or absence of a predefined set of human-interpretable concepts. The final classification prediction is then based on these concepts. Nevertheless, CBMs still require training and annotated data; in fact, the amount of annotated data actually increases since concept annotations are needed on top of the classification labels. We propose combining the best of both worlds: the interpretability of CBMs and the few-shot abilities of LVLMs to decrease the annotation cost. Our methodology, CBVLM , encompasses a two-stage approach, where, first, the LVLM is prompted to predict the existence/absence of a set of concepts in an image, and, in a second stage, the predicted concepts are incorporated into the prompt such that the LVLM provides a final diagnosis grounded on these predicted concepts. Figure 1 represents this methodology compared to the common approach where an LVLM is simply prompted to perform classification. Moreover, contrary to CBMs, which require retraining the entire model to incorporate a new concept, CBVLM can easily incorporate additional concepts, requiring only two inference steps. We validate CBVLM on four medical datasets encompassing different imaging modalities and using twelve LVLMs, and show that CBVLM achieves comparable performance or even surpasses task-specific supervised methods, doing so without training, with few annotated examples, and without sacrificing interpretability. Although, individually, the techniques used in CBVLM (e.g., ICL) are not novel, this is, to the best of our knowledge, the first work combining them in a novel methodology exploring the capability of LVLMs to provide concept-based explanations for medical image classification. Figure 2 : Overview of CBVLM . Our methodology is organized into two key stages: 1) The Concept Detection stage, where the LVLM predicts the individual presence of each predefined clinical concept in the query image. This is achieved using a custom prompt that supports both zero- and few-shot settings. In the latter, we include a set of demonstration examples (middle block of Prompt Construction ) chosen by the Retrieval Module , responsible for selecting the N most similar examples to the input image. To evaluate the LVLM answer, we employ an Evaluation Block which first tries to extract the desired LVLM response using a rule-based formulation. If this fails, we adopt an auxiliary LLM to extract the desired response. 2) In the Disease Diagnosis stage, the final diagnosis is generated by the LVLM based on the clinical concepts predicted in the first stage, which are directly incorporated in the query (highlighted in yellow). This approach ensures that the diagnosis is grounded on the identified clinical concepts, enhancing the interpretability and transparency of the LVLM’s response. In this second stage, the Retrieval Module is also used to select the N most similar demonstrations. 2 Related Work Large Vision-Language Models in the Medical Domain The emergent capabilities of Large Language Models (LLMs) have led to the development of LLMs able to process visual inputs, i.e. LVLMs. These models can reason about images in a zero- or few-shot manner (e.g., LLaVA [ 9 ] , GPT-4V [ 10 ] ), allowing them to tackle downstream tasks without fine-tuning. General-purpose LVLMs have recently been adapted for the medical domain through training on extensive medical datasets, coining the term Med-LVLMs. The generalization capability of these models has prompted researchers to assess their performance in multiple medical data. Royer et al. [ 11 ] recently released an open-source toolkit for fair and reproducible evaluation of Med-LVLMs in tasks like classification and report generation. Han et al. [ 5 ] evaluated the performance of a general LVLM on the NEJM Image Challenge, as well as the impact of fine-tuning the LVLM. In [ 12 ] , Med-LVLMs were evaluated in different specialities and compared with medical experts, with results comparable to the performance of junior doctors. In [ 7 ] , the authors benchmarked the performance of GPT-4V against fully supervised image classifiers and found that their performance is comparable. In a different line of research, Xia et al. [ 13 ] evaluated the trustworthiness and fairness of Med-LVLMs and found that models often display factual inaccuracies and can be unfair across different