Measuring the (Un)Faithfulness of Concept-Based Explanations
Shubham Kumar, Narendra Ahuja · 2025
Read Measuring the (Un)Faithfulness of Concept-Based Explanations by Shubham Kumar, Narendra Ahuja (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10833.txt.
Introduction
Shubham Kumar and Narendra Ahuja questioned how account methods for vision models are graded in 2025. Concept-based accounts restate part of a network's computation in human concepts, and checking their faithfulness. Whether the account truly mirrors the model, needs a stand-in that recombines the concepts into the output. Simpler accounts tend to score worse, setting up a trade-off.
The authors argue that recent unsupervised methods' apparent faithfulness gains are partly an artifact. Some use surrogates so complex that the account's clarity cost goes unmeasured, and others rely on deletion tests that the paper shows do not measure faithfulness properly. Their offered measure, Stand-in Faithfulness, swaps in an on purpose simple linear stand-in, keeping the account fixed while grading it, so faithfulness can be assessed without quietly spending clarity.
Catalog record
| CREATOR | Shubham Kumar, Narendra Ahuja |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring the (Un)Faithfulness of Concept-Based Explanations Shubham Kumar, Narendra Ahuja Abstract Deep vision models perform input-output computations that are hard to interpret. Concept-based explanation methods (CBEMs) increase interpretability by re-expressing parts of the model with human-understandable semantic units, or concepts . Checking if the derived explanations are faithful —that is, they represent the model’s internal computation—requires a surrogate that combines concepts to compute the output. Simplifications made for interpretability inevitably reduce faithfulness, resulting in a tradeoff between the two. State-of-the-art unsupervised CBEMs (U-CBEMs) are seemingly more interpretable, while also being more faithful to the model. However, we observe that the reported improvement in faithfulness artificially results from either (1) using overly complex surrogates, which introduces an unmeasured cost to the explanation’s interpretability, or (2) relying on deletion-based approaches that, as we demonstrate, do not properly measure faithfulness. We propose Surrogate Faithfulness (SURF) , which (1) replaces prior complex surrogates with a simple, linear surrogate that measures faithfulness without changing the explanation’s interpretability and (2) introduces well-motivated metrics that assess loss across all output classes, not just the predicted class. We validate SURF with a measure-over-measure study by proposing a simple sanity check—explanations with random concepts should be less faithful—which prior surrogates fail. SURF enables the first reliable faithfulness benchmark of U-CBEMs, revealing that many visually compelling U-CBEMs are not faithful. Code is released. 1 Introduction Deep learning models have delivered state-of-the-art (SOTA) results across diverse tasks, yet their internal computation remain difficult to interpret [ 17 ] , which is especially problematic in high-stakes domains such as healthcare and finance. This gap has given rise to explainable AI (XAI), whose goal is to produce explanations of model behavior. Two properties characterize explanation quality: interpretability —explanations that humans can understand—and faithfulness —explanations that reflect the model’s internal computation. XAI methods must balance the natural tension between interpretability and faithfulness. While interpretability is assessed with human studies, faithfulness must be evaluated by defining a surrogate that maps the explanation to the model’s outputs and measuring the loss between the surrogate and model. Since explanations are inevitably lossy, some discrepancy is expected. For certain XAI methods, defining a surrogate is challenging, so alternative measures, or proxies , attempt to evaluate faithfulness using different criteria. One broad family of XAI methods constructs inherently interpretable models , injecting an explanatory structure directly into the model [ 6 , 3 ] . Merging the explanation and the model eliminates the need for faithfulness checks; however, these methods often underperform black-box models by reducing model complexity (capacity) for explainability. In contrast, feature attribution methods express a trained model’s outputs in terms of each input feature’s contribution, commonly done by finding the output’s sensitivity to changes in input features, either through gradients or permutations [ 47 , 39 , 33 , 37 , 10 ] . For images, attributions are visualized as pixel-level heatmaps. Although such explanations are intuitive, developing faithfulness measures has been challenging, since many attribution methods do not specify a surrogate to reconstruct the model’s output from the explanation. Thus, proxies—such as checking if important pixels come from foreground regions or looking for significant model degradation after deleting important pixels—are used to study faithfulness. Despite strong performance on the proxies, clever sanity checks have revealed faithfulness problems [ 1 , 15 , 41 , 22 ] . The resulting heatmaps also lack interpretability—they only indicate where the model attends, not what semantics it recognizes [ 35 , 7 ] . Concept-based explanation methods (CBEMs) improve interpretability by explaining predictions in terms of human-understandable concepts (e.g., edges, colors, object parts) [ 14 , 4 , 24 , 55 ] . To recognize concepts, supervised CBEMs require a concept-annotated image dataset, which is costly and subjective; moreover, any fixed concept vocabulary may bias the explanation and miss important aspects of the model’s computation, raising faithfulness concerns [ 38 ] . To avoid these issues, unsupervised CBEMs (U-CBEMs) discover concepts automatically as concept activation vectors (CAVs)—directions in the model’s intermediate representation space—paired with a concept importance score that captures the concept’s relevance to the output. Since CBEMs operate on the model’s intermediate representations, defining a surrogate is more natural, paving the way for proper faithfulness evaluations. However, a closer look shows that U-CBEMs have adopted inadequate faithfulness measures. We discover that existing evaluations rely on complex surrogates, which allow U-CBEMs to simultaneously show users interpretable explanations and report high faithfulness, yet the explanations do not clearly lead to the model’s output. Furthermore, U-CBEMs adapt popular deletion-based proxies from the attribution literature, yet Sec. 4.1 outlines serious unresolved limitations that prevent them from properly measuring faithfulness. Finally, each work that proposes a new U-CBEM also evaluates it with a new faithfulness measure, with no measure-over-measure comparison, indicating a lack of consensus on faithfulness measures in the field. We argue that these issues have misled us into believing that current U-CBEMs are faithful. To shed light on this, we make the following contributions: 1. Organize prior U-CBEM faithfulness measures under a common framework, allowing us to discuss their limitations and identify appropriate desiderata. 2. Propose Surrogate Faithfulness (SURF) , a faithfulness measure satisfying the desiderata. Our measure-over-measure comparison checks if faithfulness decreases as the explanation is increasingly randomized; only SURF passes this check. 3. Conduct the first, comprehensive faithfulness benchmark of current U-CBEMs across a variety of tasks and model architectures. SURF shows that SOTA U-CBEMs previously evaluated to be faithful are not. 4. Leverage SURF to provide a selection criterion for the number of concepts U-CBEMs discover, improving on prior work that sets this hyperparameter arbitrarily. 2 Preliminaries We first introduce notation used throughout the paper, as well as preliminaries to familiarize the reader with the field. Background: Model \phi:\mathcal{X}\to\mathcal{Y} maps from an input space \mathcal{X}\subseteq\mathbb{R}^{P} to an output space \mathcal{Y}\subseteq\mathbb{R}^{C} . Assume \phi admits an intermediate space \mathcal{H}\subseteq\mathbb{R}^{D} . Let g:\mathcal{X}\to\mathcal{H} and f:\mathcal{H}\to\mathcal{Y} . Thus, \mathbf{y}=\phi(\mathbf{X})=f(g(\mathbf{X})) . Let g(\mathbf{x})=\mathbf{h}\in\mathcal{H} represent the embedding of \mathbf{x} (e.g., image patch), and g(\mathbf{X})=\mathbf{H}\in\mathbb{R}^{(HW)\times D} denote the vectorized application of g on \mathbf{X}=\begin{bmatrix}\mathbf{x}_{1}&\cdots&\mathbf{x}_{HW}\end{bmatrix}^{T} , where H,W are the spatial dimensions. Let E be an explanation function, and let E(\mathbf{X})\in\mathcal{E} represent the explanation for an \mathbf{X}\in\mathcal{X} (e.g., image). Let s:\mathcal{E}\to\mathcal{Y} be a surrogate. Motivated by Ribeiro et al. [ 39 ] , E is completely faithful to \phi if s reproduces \phi(\mathbf{X})\;\forall\;\mathbf{X}\in\mathcal{X} . Thus, a faithfulness measure is defined by a choice of surrogate s(.) and metric d(.) as: Faith(E;d,s)=\int_{\mathcal{X}}d(\phi(\mathbf{X}),s(E(\mathbf{X})))d\mathbf{X} (1) U-CBEMs: To generate explanations, U-CBEMs find, for all outputs i\in C , a set of K CAVs V_{i}=\{\mathbf{v}_{i,k}\}_{k=1}^{K} and concept importances A_{i}=\{\alpha_{i,k}\}_{k=1}^{K} . Note that while, for a given input, K can vary per output class, all prior works find the same K per output class. U-CBEMs also define a mechanism \mathcal{P}(\mathbf{h};V):\mathcal{H}\to\mathbb{R}^{K} to project embedding h to the concept space defined by CAVs in V . The explanation for output i is: \displaystyle E_{i}(\mathbf{X}) \displaystyle=E_{i}(\mathbf{X};g,V_{i},A_{i},\mathcal{P}) (2) \displaystyle=\{\mathcal{P}(g(\mathbf{X});V_{i}),A_{i}\} E_{i}(\mathbf{X}) is then conveyed to the user through some visual interface. Note that \mathcal{P}(g(\mathbf{X})) denotes the vectorized application of \mathcal{P} on g(\mathbf{X})=\mathbf{H} . Following prior works, we study faithfulness in the final layer, where the spatial dimensions H=W=1 , so \mathbf{H}=\mathbf{h} . Given that U-CBEMs operate on g(\mathbf{X}) , the faithfulness measure can be simplified to: