On the Faithfulness of Post-Hoc Concept Bottleneck Models
Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler · 2026
Read On the Faithfulness of Post-Hoc Concept Bottleneck Models by Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30498.txt.
Introduction
Laines Schmalwasser and four colleagues check whether interpretable AI models' concepts mean what they claim to mean. Concept bottleneck models route predictions through human concepts first. Post-hoc versions learn those concepts without task-specific labels. They borrow outside datasets or vision-language model labels. High accuracy can hide meaningless concepts. Random projections sometimes score well, a warning sign. The authors analyze the learned projections directly.
Two failure modes are identified and formalized. Concepts learned from outside data can shift off-target harmfully. An upper bound sizes the error that shift introduces. Labels generated by vision-language models carry steady noise instead. That noise also produces unfaithful projections. New metrics separate faithfulness from raw accuracy cleanly. Experiments span real-world and synthetic benchmarks. The metrics catch unfaithful behavior that accuracy checks miss. The work argues evaluation must test meaning, not just scores.
Catalog record
| CREATOR | Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30498
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Faithfulness of Post-Hoc Concept Bottleneck Models Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler Abstract Human decision-making interprets the world through high-level concepts, such as recognizing a bird by its belly color. To bridge the gap between opaque deep learning representations and human understanding, Post-Hoc Concept Bottleneck Models (post-hoc CBMs) project latent features onto interpretable concept spaces using auxiliary datasets or vision-language models. However, relying on target task accuracy as the primary measure of post-hoc CBM success obscures whether the learned concepts are semantically meaningful or merely predictive artifacts. For example, random concept projections can achieve competitive accuracy despite being semantically meaningless. In this work, we analyze the learned projections directly and identify two failure cases: First, for concept projections learned from auxiliary data, covariate shifts can lead to unfaithful concept representations for the target task. In particular, we provide an upper bound on the error introduced by this shift. Second, systematic label noise in surrogate concept labels generated by vision-language models leads to unfaithful projections. After formalizing these failure modes, we introduce novel metrics that decouple concept faithfulness from predictive accuracy. Our empirical results across real-world and synthetic benchmarks confirm that these metrics identify unfaithful behaviors that standard accuracy-based evaluation fails to detect 1 1 1 Project page: https://posthoc-cbm-faithfulness.github.io/ . 1 Introduction Human understanding is fundamentally built on concepts [ 6 , 18 , 47 ] . Rather than analyzing raw sensory data, we identify objects through high-level attributes, such as recognizing a specific bird species by its “yellow belly” or “black crown.” In contrast, deep neural networks learn opaque, high-dimensional representations that often lack semantic clarity [ 37 , 32 , 7 ] . To bridge this gap and enforce interpretability, Concept Bottleneck Models (CBMs) were introduced [ 32 , 2 , 11 , 28 , 46 , 67 , 68 , 1 , 17 ] . CBMs constrain the model to first predict a set of human-understandable concepts from the input, and then base the final prediction solely on them, thereby promising transparent decision-making [ 25 , 41 , 43 ] . However, standard CBMs require dense concept annotations for the specific downstream task, which are rarely available in practice [ 49 , 32 , 15 ] . Post-Hoc Concept Bottleneck Models (post-hoc CBMs) circumvent this by freezing a pretrained feature extractor and learning only a lightweight projection from its internal representations to a concept space and final classifier [ 71 , 49 , 64 ] (we use post-hoc CBM to denote this general model class and PCBM for the concrete implementation in [ 71 ] ). Crucially, post-hoc CBMs tackle the need for in-domain concept labels via two strategies: (1) Auxiliary Concept Datasets , where the projection is learned on a separate, richly annotated dataset (e.g., Broden [ 7 ] ) before application to the target task [ 71 ] ; or (2) Surrogate Labels from VLMs , where a Vision-Language Model (e.g., CLIP [ 55 ] ) generates surrogate concept labels [ 50 , 49 , 64 , 57 ] . Both drastically reduce the annotation burden while maintaining competitive downstream classification accuracy [ 71 , 49 ] . Task Dataset \mathbb{X}_{\text{task}} Auxiliary Concept Set \mathbb{X}_{\text{aux}} Covariate Shift P_{\text{task}}\neq P_{\text{aux}} f Frozen Backbone a_{1} a_{2} a_{3} Activation Space \mathcal{A} \begin{array}[]{r}\text{\scriptsize red}\\ \text{\scriptsize green}\\ \vdots\\ \text{\scriptsize triangle}\end{array}\begin{bmatrix}1.0\\ 0.1\\ \vdots\\ 0.9\end{bmatrix} Concept Space \mathcal{C} Problem: Missing Labels to Train \boldsymbol{\pi}_{\boldsymbol{\theta}} \begin{array}[]{r}\text{\scriptsize red-striped-triangle}\\ \vdots\\ \vdots\\ \text{\scriptsize blue-solid-plus}\end{array}\begin{bmatrix}0.9\\ \vdots\\ \vdots\\ 0.0\end{bmatrix} Class Space \mathcal{Y} h \pi_{\theta} ① Train \pi_{\theta} on Auxiliary Dataset \mathbb{X}_{\text{aux}} ② Generate Surrogate Labels With VLM to Train \pi_{\theta} Figure 1 : In the post-hoc CBM setup [ 71 , 49 , 64 ] , inputs are mapped to activations using some frozen backbone model f . Then, a learned concept projection \pi_{\theta} extracts concepts before a final classifier h predicts a class based on these concepts. Unfortunately, concept labels in the downstream task domain are often unavailable. Thus, ① auxiliary concept sets (e.g., Broden [ 7 ] ) or ② surrogate VLM labels are used [ 71 , 49 , 64 ] . We study these approaches and identify two reasons for potential unfaithfulness of \pi_{\theta} : ① a covariate shift of the concept set, and ② systematic surrogate label errors. Unfortunately, evaluating downstream accuracy provides limited insight into the quality of the learned concept projection. Prior work empirically shows that random concepts achieve high performance [ 45 ] . In [ 42 ] , this phenomenon is connected to the Johnson-Lindenstrauss (JL) lemma [ 26 ] , with the expected error approaching zero as random projections increase [ 64 ] . We expand these insights via the smooth manifold JL lemma [ 5 ] , based on previous findings that neural activations tend to lie on low-dimensional manifolds [ 3 , 54 ] . We show that under certain assumptions, the original activations can be reconstructed from a modest number of random projections, resulting in high downstream performance. Consequently, we must investigate the concept projection itself to evaluate whether it faithfully extracts the intended concepts on the target distribution. Here, we analyze the two primary post-hoc CBM training mechanisms ( Figure 1 ). First, for auxiliary concept datasets, we identify covariate shift as the primary source of unfaithfulness. We demonstrate that even if a concept’s semantic definition is identical across domains, a geometric shift in the feature space can invalidate the learned projection on the downstream task. We formalize this by providing an upper bound for the error introduced by this shift based on [ 8 ] and propose an approximation of it as a practical measure of faithfulness. Second, for methods using VLM-based surrogate labels, the primary problem is label noise . However, our analysis shows that unfaithfulness is not solely caused by noise magnitude, but by systematic errors. In other words, random mistakes can cancel each other out, but the surrogate labeling function leads to unfaithfulness when mistakes are consistent, e.g., always predicting “sky” and “clouds” together. To detect this, we measure the surrogate-label error and propose a metric that explicitly quantifies the corresponding systematicity. We empirically validate our theoretical findings on standard benchmarks (CUB-200 [ 70 ] and CIFAR-10/100 [ 33 ] ) and the synthetic Elements [ 48 ] dataset, which provides known ground-truth concepts. Our evaluation includes a range of post-hoc CBMs across standard backbones representing the two main training paradigms: those using auxiliary datasets [ 71 ] (e.g., Broden [ 7 ] ) and those using surrogate labels generated by VLMs (LFCBM [ 49 ] and VLG-CBM [ 64 ] based on CLIP [ 55 ] , DINOv3 [ 62 ] and Grounding DINO [ 39 ] ). We first show that random concept projections can achieve competitive accuracy, confirming that downstream accuracy is an insufficient metric for meaningful representations. Then, we directly evaluate the learned concept projections using available ground truth [ 48 ] , apply our proposed faithfulness metrics, and demonstrate that they successfully identify failures caused by covariate shifts and systematic errors, offering a practical tool for evaluating post-hoc CBMs beyond predictive performance. Our contributions can be summarized as follows: 1. We show that under certain assumptions, random concept projections can