Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

Nicola Debole, Andrea Passerini, Stefano Teso, Andrea Pugnana, Emanuele Marconato · 2026

Read Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations by Nicola Debole, Andrea Passerini, Stefano Teso, Andrea Pugnana, Emanuele Marconato (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16405.txt.

Introduction

Debole and four others introduce VH-CBM, a hybrid concept-bottleneck model combining vision-language annotations with dense expert annotations. Concept-bottleneck models classify from high-level input concepts, making predictions understandable, but such annotations are seldom available. Recent architectures obtain vision-language annotations but yield lower quality concepts. VH-CBM uses a Gaussian Process in the vision-language embedding space to propagate expert supervision to target data points. Experiments show VH-CBM predicts more accurate concepts than vision-language-guided models annotating 1 percent of the data.

Catalog record

CREATORNicola Debole, Andrea Passerini, Stefano Teso, Andrea Pugnana, Emanuele Marconato
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61484
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations Nicola Debole, Andrea Passerini, Stefano Teso, Andrea Pugnana, Emanuele Marconato Abstract Concept-bottleneck models (CBMs) are neural classifiers that compute predictions from high-level concepts extracted from the input. CBMs ensure stakeholders can understand the concepts – and the predictions they entail – by learning these from concept-level annotations, which are however seldom available. Recent CBM architectures work around this issue by obtaining annotations from Vision-Language Models (VLMs). While greatly broadening applicability, doing so can yield lower quality concepts and therefore less interpretable models. We strike for a middle ground by introducing Vision-plus-Human-guided CBM ( VH-CBM ), a hybrid approach that exploits both VLMs and a small amount of dense annotations. VH-CBM employs a Gaussian Process in the VLM’s embedding space, which captures useful global information about the target domain, to propagate the expert’s supervision to any target data point. Our empirical evaluation shows how VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1 % 1\% of the data, while sporting better concept calibration and supporting active learning. 1 Introduction Concept-bottleneck models (CBMs) [ 26 ] are explainable-by-design neural classifiers that infer predictions from high-level concepts – e.g. , objects present in an input image – they extract from the input. CBMs achieve task performance comparable to black-box alternatives, all while enabling stakeholders to easily inspect the concepts responsible for any prediction. To support these operations, CBMs encourage the concepts to be interpretable by learning them from expert annotations . Yet, these are often unavailable and expensive to obtain, hindering applicability. State-of-the-art vision-language-guided CBMs (henceforth, VLM-CBMs) work around this limitation by replacing expert annotations with weak supervision obtained from large vision-language models (VLMs) [ 50 , 37 , 60 , 21 ] . While architectural details differ (cf. Section 2 ), these models enjoy broader applicability while sporting similar or better task performance than CBMs. The downside is that VLM annotations are not entirely reliable [ 17 , 43 , 6 ] , meaning that VLM-CBMs’ concepts tend to be less accurate and therefore less interpretable [ 7 ] . This is problematic, as it compromises the promise of interpretability that CBMs are meant to uphold. Figure 1: Top : VLM-CBMs rely solely on concepts provided by VLMs – either at inference time (as above) or for distilling a concept extractor g , cf. Section 2 – which can be inaccurate, compromising interpretability [ 7 ] . Bottom : VH-CBM exploits the VLM’s embedding space to propagate expert concept supervision to obtain concept statistics (i.e., {\bm{\mathrm{\mu}}} and {\bm{\mathrm{\sigma}}} from the GP) for any test point, improving concept accuracy with few annotations. We tackle this overlooked trade-off by introducing Vision-plus-Human-guided CBMs ( VH-CBM s for short), a simple hybrid model that integrates widely available, but unreliable, VLMs with a modicum of trustworthy expert supervision. VH-CBM s build on the intuition that the VLM embedding space, despite yielding potentially poor point-wise annotations, encodes useful global information about the application domain. VH-CBM s realize this insight by introducing efficient, stochastic Gaussian Processes (GPs) [ 57 ] in embedding space: given expert concept annotations for selected examples, the GPs exploit its geometry to infer more accurate and calibrated concept activations for any incoming input. These are fed to a transparent inference layer to obtain a prediction, cf. Fig. 1 . We experimentally evaluate VH-CBM s on four synthetic and real-world datasets. Our results indicate that, compared to existing VLM-CBMs, VH-CBM s predict concepts more accurately from minimal amounts of expert annotations: even when annotating a small portion of the dataset, VH-CBM s substantially outperform VLM-CBMs according to different concept-prediction quality metrics, while sporting better concept-level calibration. Moreover, increasing the amount of supervision monotonically improves concept accuracy, indicating how VH-CBM s can successfully make use of expert annotations when available, especially when allowed to actively query concept annotations. Contributions : Summarizing, we ( i ) identify an interpretability-applicability trade-off affecting current CBM and VLM-CBM architectures; ( ii ) introduce VH-CBM , a simple approach that addresses this trade-off by mediating between the VLMs and the CBMs with a suitably designed Gaussian Process; and ( iii ) demonstrate the promise of VH-CBM in terms of concept accuracy and calibration on several data sets and two backbones w.r.t. regular and VLM-based CBMs. 2 Background Notation . We abbreviate \{1,\ldots,n\} to [n] . We denote inputs as {\bm{\mathrm{x}}}\in\mathbb{R}^{d} and labels as y\in[\ell] . We consider k (binary or categorical) concepts, each with v_{i}\geq 2 possible values, capturing task-relevant information present in the input, e.g. , objects and their properties. We encode them as concept activations in multi-hot format, {\bm{\mathrm{c}}}\in\mathbb{R}^{v} with v:=\sum_{i=1}^{k}v_{i} . We indicate with \mathcal{X}\subseteq\mathbb{R}^{d} the input space; with \mathcal{C}\subseteq\mathbb{R}^{v} , v<d , the space of concept activations; and with \mathcal{Y}=[\ell] the set of labels. Concept-based Models . CBMs sport a two stage architecture comprising ( i ) a concept extractor g:\mathcal{X}\to\mathcal{C} that maps the inputs ( e.g. , images of fruits) to concept activations ( e.g. , visual properties such as “ red ” and “ round ”), and ( ii ) a task predictor f:\mathcal{C}\to\mathcal{Y} – typically a linear classifier – that maps concepts to classes ( e.g. , “ apple ” vs. “ pear ”). Given a training set \mathcal{D}=\{({\bm{\mathrm{x}}}^{(i)},{\bm{\mathrm{c}}}^{(i)},y^{(i)})\} with n examples, CBMs can be trained (a) independently , i.e. , g and f are trained separately and later combined; (b) sequentially , i.e. , g is trained first, and its output is used to train f ; or (c) jointly , i.e. , g and f are trained together by optimizing a joint objective such as \sum_{i=1}^{n}\mathsf{CE}(f(g({\bm{\mathrm{x}}}^{(i)})),y^{(i)})+\frac{\lambda}{k}\sum_{j=1}^{k}\mathsf{CE}(\hat{c}^{(i)}_{j},c^{(i)}_{j}), (1) where \mathsf{CE} is the (multi-class) cross-entropy loss and \hat{c}^{(i)}_{j} is the probability or logit returned by the concept extractor g for the j -th concept of {\bm{\mathrm{x}}}^{(i)} . Thanks to this two-stage architecture, CBMs can be interpretable , as one can assess which concepts affect the task prediction, and intervenable , as users can correct mispredicted concepts to improve the model’s predictions [ 47 , 39 ] . Learning aligned concepts. The interpretability of CBMs hinges on whether the concepts they learn are aligned with human semantics [ 45 , 26 , 33 ] . Unless this is the case, one can encounter or construct inputs {\bm{\mathrm{x}}} ( e.g. , images of red apples) on which the CBM and stakeholders infer different concepts ( e.g. , “orange” vs. “red”). Such semantic mismatches complicate interpretation: CBM explanations ( e.g. , “this is an apple because it is red”) rely on the learned concepts, yet these may not possess the meaning that humans associate to them. Ensuring that CBMs satisfy this basic property requires training them with an abundance of high-quality concept annotations [ 26 , 32 , 62 ] . This is essential, as CBMs trained without concept supervision can learn completely misaligned concepts [ 4 ] . However, this restricts usage of CBMs to the few tasks for which high-quality concept supervision is available. Vision-language-guided CBMs . Recent CBM architectures work around this issue by leveraging a Vision-Language Model (VLM), such as CLIP [ 40 ] or Grounding Dino [ 28 ] , to generate weak concept supervision in

Read the complete text