Weakly Supervised Concept Learning with Class-Level Priors for Interpretable Medical Diagnosis

Md Nahiduzzaman, Steven Korevaar, Alireza Bab-Hadiashar, Ruwan Tennakoon · 2025

Read Weakly Supervised Concept Learning with Class-Level Priors for Interpretable Medical Diagnosis by Md Nahiduzzaman, Steven Korevaar, Alireza Bab-Hadiashar, Ruwan Tennakoon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01131.txt.

Introduction

Md Nahiduzzaman and colleagues make medical image diagnosis more interpretable without demanding costly concept labels. Concept based models explain a diagnosis through human ideas, such as lesion features, but training them normally needs experts to annotate those concepts on every image, which clinics rarely can do.

Their Prior-guided Concept Predictor uses class level concept priors, typical concept patterns for each disease class, as weak supervision instead. A refinement step with divergence and entropy penalties aligns the predicted concepts with clinical reasoning. On skin lesion and blood cell data sets, concept prediction quality rose by over a third against zero shot baselines. Diagnosis accuracy stayed competitive with fully supervised concept bottleneck models across four medical data sets.

Catalog record

CREATORMd Nahiduzzaman, Steven Korevaar, Alireza Bab-Hadiashar, Ruwan Tennakoon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01131

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Weakly Supervised Concept Learning with Class-Level Priors for Interpretable Medical Diagnosis Md Nahiduzzaman, Steven Korevaar, Alireza Bab-Hadiashar, Ruwan Tennakoon Abstract Human-interpretable predictions are essential for deploying AI in medical imaging, yet most interpretable-by-design (IBD) frameworks, require concept annotations for training data, which are costly and impractical to obtain in clinical contexts. Recent attempts to bypass annotation, like zero-shot vision–language models or concept-generation frameworks, struggle to capture domain-specific medical features, leading to poor reliability. In this paper, we propose a novel Prior-guided Concept Predictor (PCP), a weakly supervised framework that enables concept answer prediction without explicit supervision or reliance on language models. PCP leverages class-level concept priors as weak supervision and incorporates a refinement mechanism with KL divergence and entropy regularization to align predictions with clinical reasoning. Experiments on PH2 (dermoscopy) and WBCatt (hematology) show that PCP improves concept-level F1 by over 33% compared to zero-shot baselines, while delivering competitive classification performance on four medical datasets (PH2, WBCatt, HAM10000, and CXR4) relative to fully supervised CBMs and V-IP. K eywords Concept prediction \cdot Concept bottleneck models \cdot Interpretability \cdot Weak supervision 1 Introduction Deep learning (DL) has achieved remarkable success in medical imaging, yet most models operate as “black boxes”, limiting clinical trust and adoption. Interpretable-by-design (IBD) models, such as Concept Bottleneck Models (CBM) [ 1 ] and Variational-Information Pursuit (V-IP) [ 2 ] , address this by mapping raw image features to human-understandable concepts (e.g., blue-whitish veil , irregular streaks ) and using them for classification. CBMs require all concepts to be predicted before making a decision, while V-IP adaptively queries only the most informative ones. Both approaches provide meaningful explanations aligned with clinical reasoning. Building on these ideas, the Multimodal Interpretable Concept Alignment (MICA) [ 3 ] and Concept-Based Vision–Language Model (CBVLM) [ 4 ] require few-shot or partially annotated concepts. However, collecting per-concept annotations at scale remains difficult in medical imaging, as concepts are subtle (there is often disagreement even amongst experts) and experts have limited time and resources. Thus, the central challenge is achieving concept prediction without explicit concept supervision. Recent works attempt to relax this dependency. Post-hoc CBMs [ 5 ] use Concept Activation Vectors from external datasets but fail when no curated concept bank exists. Language-guided models such as LaBo-CBM [ 6 ] use large language models (LLMs) to define textual concepts, while V2C-CBM [ 7 ] employs VLMs to tokenize visual concepts. Similarly, ConceptCLIP [ 8 ] learns image–text–concept embeddings via medical ontologies, but its reliance on predefined vocabularies and alignment quality limits generalization to subtle clinical findings. Recent V-IP extensions, including Bootstrapped [ 9 ] and Learned-Query V-IP [ 10 ] , integrate VLMs like CLIP to answer or learn interpretable queries but still rely on textual supervision. In a different line of work, the Visual Perception Module (VPM) [ 11 ] introduces attention-based refinement for concept-guided zero-shot classification but assumes access to ground-truth concept annotations , differing from our goal of predicting concept answers without annotations or VLM reliance. In this work, we address the question: Can medical concept prediction be achieved without explicit concept supervision or reliance on VLM guidance? We propose a novel weakly supervised Prior-guided Concept Predictor (PCP) framework (Fig. 1 ) for medical concept prediction without concept-level annotations or vision–language supervision . PCP learns concepts from image features using class-level concept priors, aligning predicted concept distributions with known statistics while preserving image-specific variability. These priors can be derived from domain experts, dataset–level statistics, or automated knowledge sources, and are far easier to obtain (e.g., from a single expert consultation) than exhaustive concept-level annotations, thus bypassing the prohibitive cost of manual supervision. Two regularizers, a KL-divergence loss and an entropy loss, promote alignment and concept selectivity. In contrast to VPM, which focuses only on the most discriminative features, our proposed PCP framework predicts all clinically relevant concepts , preserving interpretability. We validate PCP on four medical datasets spanning different imaging modalities: PH2 (dermoscopy) and WBCatt (hematology), which include ground-truth concept annotations for evaluating concept-level prediction accuracy, and HAM10000 (dermoscopy) and CXR4 (chest X-ray), which lack concept labels and are evaluated using LLM-derived class-level priors. Across all datasets, PCP outperforms zero-shot VLM baselines (CLIP, SigLIP, BioMedCLIP, ConceptCLIP) in concept prediction and achieves classification performance comparable to fully supervised CBM and V-IP models, demonstrating that reliable and interpretable concept reasoning is possible even without explicit concept supervision. In summary, the major contribution of this work is the introduction of the PCP , a weakly supervised framework that enables medical concept prediction without concept-level annotations or VLM supervision by aligning image-predicted concepts with class-level priors through a composite loss, achieving interpretable and competitive performance across multiple medical imaging datasets. 2 Methodology 2.1 Problem Setup The goal of this research is to predict an outcome label from an input image through a set of human–interpretable concepts. Given a training dataset \mathcal{D}_{\textrm{train}}=\{(x_{i},y_{i})\}_{i=1}^{N}, where each image x_{i}\in\mathcal{X} is paired with an outcome label y_{i}\in\mathcal{Y} , and the label space has cardinality |\mathcal{Y}|=L , we assume the existence of M human–interpretable concepts, denoted by \mathcal{C}=\{c_{1},c_{2},\ldots,c_{M}\} . A concept predictor maps an input image x to a probability vector of concept activations such that: \hat{\mathbf{c}}(x)=[\hat{c}_{1}(x),\ldots,\hat{c}_{M}(x)]\in[0,1]^{M},\quad\hat{\mathbf{c}}:\mathcal{X}\rightarrow[0,1]^{M}. The predicted concept vector is then used for classification. To train our model without sample-wise concept labels, we rely on concept-wise priors, which represent the likelihood of each concept appearing in an image belonging to a specific class. For each class y\in\mathcal{Y} , the probability of a concept c_{m}\in\mathcal{C} appearing is given by: P(c_{m}\mid y)\in[0,1] . Figure 1: Overview of the proposed framework. Given an input image, a ResNet backbone extracts features, which are projected into a concept space and refined through a prior-guided attention mechanism. Attention is computed by combining projected features with probabilistic vectors sampled from concept-priors and subsequently refined (Eq. 3 ). KL regularization aligns predicted concepts with priors, while entropy minimization sharpens attention distributions. 2.2 Proposed Model Our concept prediction model combines a backbone encoder (a projection into concept space) and a prior-guided refinement mechanism. We employ a ResNet backbone pretrained on ImageNet to extract visual features from the input image x\in\mathbb{R}^{C\times H\times W} . The classification head of the backbone is replaced with two bias-free linear layers to reduce dimensionality and map the features onto concept space, the result being \mathbf{z} . We use Bernoulli sampling from class-specific priors to generate surrogate concept vectors. For an image with label y , the surrogate vector is defined as: \tilde{\mathbf{c}}(x)\sim\text{Bernoulli}(P(c_{m}\mid y)),\quad\tilde{\mathbf{c}}(x)\in\{0,1\}^{M}, (1) These sampled vectors act as probabilistic substitutes for the unobserved ground-truth concept annotations, acknowledging that not all concepts associated with a class appear in every instance. We use element-wise multiplication to combine the projected image features with the surrogate concept vectors, akin to VPM [ 11 ] : \gamma(x)=\text{softmax}(\mathbf{z}\odot\tilde{\mathbf{c}}(x)),\quad\gamma(x)\in\mathbb{R}^{M}, (2) The attention vector \gamma(x) highlights the concepts most relevant for predicting class y . Directly masking with \gamma(x) as in [ 11 ] risks discarding weak but potentially informative concepts. To mitigate this, we adopt a residual refinement mechanism: \mathbf{z}^{\prime}=\mathbf{z}\odot\big(1+\beta\cdot\gamma(x)\big),\quad\mathbf{z}^{\prime}\in\mathbb{R}^{M}, (3) where \beta is a tunable parameter controlling the strength of refinement. This ensures that important concepts are amplified while less relevant but possibly useful ones are not completely suppressed, improving both stability and robustness. Finally, the

Read the complete text