Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning
Hubert Baniecki, Przemyslaw Biecek University of Warsaw Warsaw University of Technology {h.baniecki,p.biecek}@uw.edu.pl · 2025
Read Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning by Hubert Baniecki, Przemyslaw Biecek University of Warsaw Warsaw University of Technology {h.baniecki,p.biecek}@uw.edu.pl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08636.txt.
Introduction
Hubert Baniecki and Przemyslaw Biecek wrote a 2025 paper on models that claim to explain their own choices. Some image models point to sample parts or key traits as proof of how they decide. Many users trust those claims, yet the claims had not been fully tested for tricks or attacks.
The authors set out two attack plans that fool such models through the sample parts they rely on. They also test how a second model type, built on named traits, holds up against the same tricks. The fooled models still looked clear and right, which can give false trust. The authors call for more work on safe, honest models.
Catalog record
| CREATOR | Hubert Baniecki, Przemyslaw Biecek University of Warsaw Warsaw University of Technology {h.baniecki,p.biecek}@uw.edu.pl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08636
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning Hubert Baniecki, Przemyslaw Biecek University of Warsaw Warsaw University of Technology {h.baniecki,p.biecek}@uw.edu.pl Abstract A common belief is that intrinsically interpretable deep learning models ensure a correct, intuitive understanding of their behavior and offer greater robustness against accidental errors or intentional manipulation. However, these beliefs have not been comprehensively verified, and growing evidence casts doubt on them. In this paper, we highlight the risks related to overreliance and susceptibility to adversarial manipulation of these so-called “intrinsically (aka inherently) interpretable” models by design. We introduce two strategies for adversarial analysis with prototype manipulation and backdoor attacks against prototype-based networks, and discuss how concept bottleneck models defend against these attacks. Fooling the model’s reasoning by exploiting its use of latent prototypes manifests the inherent uninterpretability of deep neural networks, leading to a false sense of security reinforced by a visual confirmation bias. The reported limitations of part-prototype networks put their trustworthiness and applicability into question, motivating further work on the robustness and alignment of (deep) interpretable models. Figure 1: Birds look like cars. Interpretation of image recognition by a ProtoViT classifying bird species based on car prototypes with 85% accuracy. We overemphasize that what looks alike to humans is not representative of what is similar according to the model. Below is a schema of the prototype-based model’s architecture with a highlighted point of failure. Manipulating the model’s reasoning by exploiting its use of latent prototypes ( \theta_{g} ) manifests the inherent uninterpretability of prototype-based networks, which may be masked by human overreliance due to visual confirmation bias. 1 Introduction Deep learning has seen remarkable development over the last decade, solving with superhuman efficiency a number of important tasks, such as object classification, segmentation, detection, etc. Unfortunately, the increasing complexity of (deep) machine learning models poses a challenge to their reliable adoption in high-stakes decisions like in medicine ( Rudin, 2019 ; Combi et al., 2022 ; Jia et al., 2022 ; Baniecki et al., 2025b ) or security ( Andresini et al., 2022 ; Al-Essa et al., 2024 ) . Even beyond automated decision-making, deep learning models allow acquiring insightful knowledge that leads to valuable discoveries, e.g. discovery of a structural class of antibiotics with explainable deep learning ( Wong et al., 2024 ) , better understanding of optimal strategies in board games ( McGrath et al., 2022 ) or formulation of the laws of physics ( Schmidt and Lipson, 2009 ) . A limitation to such future discoveries is the black-box nature of deep learning architectures. To this end, a wide spectrum of interpretable machine learning methods was proposed, ranging from advances in intrinsically (aka inherently ) interpretable models ( Chen et al., 2008 ; Caruana et al., 2015 ; Arik and Pfister, 2021 ; Zhong et al., 2023 ) to improvements in post-hoc explainability algorithms for understanding large and complex models ( Biecek and Burzykowski, 2021 ; McGrath et al., 2022 ; Covert et al., 2024 ; Taimeskhanov et al., 2024 ; Baniecki et al., 2025a ; Chrabaszcz et al., 2025 ) . But post-hoc explanations only approximate the behavior of the model and can be inaccurate ( Rudin, 2019 ; Covert et al., 2024 ; Hesse et al., 2024 ) . To ensure that the model and explanations are consistent, various interpretable deep learning architectures are proposed ( Li et al., 2022 ) like prototype-based networks ( Chen et al., 2019 ) , bag-of-local-features ( Brendel and Bethge, 2019 ) , concept bottleneck models ( Koh et al., 2020 ) , or most recently B-cos networks ( Böhle et al., 2024 ) , which claim to offer image classification accuracy on par with their uninterpretable counterparts. The growing popularity of these model families comes from the intuitive, though never comprehensively tested, assumptions that inherently interpretable models show explanations consistent with human intuition, and are robust to accidental errors or intentional manipulation. After all, could these assumptions be wrong? Hoffmann et al. (2021) was the first to question the intrinsic interpretability of prototype-based networks relying on similarities between embeddings of uninterpretable network backbones. Ramaswamy et al. (2023) highlight that the choice of the probe distribution for concept-based explainability has a profound impact on the model’s interpretation. Based on their objections, we designed a set of examples that put into question the interpretability of prototypes. For example, Figure 1 shows an interpretation of the prototype-based network ( Ma et al., 2024 , specifically ProtoViT,) , where an image of a bird is classified based on features extracted from car images. Our proposed prototype manipulation highlights that visual confirmation bias ( Klayman, 1995 ; Kim et al., 2022 ) is a threat, potentially masking these models’ inherent uninterpretability . Furthermore, the manipulation might be hard to discover when only small patches and prototypes are presented. Aside from the problem with interpretability, we also still know little about the adversarial robustness of prototype-based networks. While there is a growing body of evidence that deep models are fragile, from the first adversarial examples fooling neural networks ( Szegedy et al., 2014 ) , through extensive robustness benchmarks for image classification ( Hendrycks and Dietterich, 2019 ) , also specifically in medical applications ( Finlayson et al., 2019 ; Jia et al., 2022 ) , to manipulation of large foundation models ( Jankowski et al., 2024 ) . Relatively well-studied are adversarial threats to post-hoc explainability, like adversarial examples and backdoor attacks against saliency maps ( Noppel and Wressnegger, 2024 ) , or data poisoning attacks against feature importance and effects ( Baniecki and Biecek, 2024 ) . A natural question arises: Are intrinsically interpretable deep learning models robust and secure? Our adversarial analysis aims to motivate further improvements in the alignment and security of deep interpretable models. Although adversarial examples crafted with projected gradient descent were already used to evaluate prototype stability ( Huang et al., 2023a ) and spatial misalignment ( Sacha et al., 2024 ) , as well as the robustness of concept bottleneck models ( Sinha et al., 2023 ) , we consider a threat of backdoor attacks instead ( Noppel et al., 2023 ) . To illustrate this, Figure 2 shows an adversarially manipulated interpretation of the prototype-based network ( Nauta et al., 2023b , specifically PIP-Net,) , where an image of a skin lesion is classified as either benign or malignant. We found it surprising that it is possible to embed a backdoor in a deep interpretable model in a way that conceals the adversarial trigger, or even manipulates the interpretation to contradict the prediction. Figure 2: Backdoor attacks on a PIP-Net model classifying malignant skin lesions with 85% accuracy. The naive backdoor attack could be detected by a visual inspection of explanations, where one prototype highlights the trigger. Below is a schema of the prototype-based model’s architecture with a highlighted point of failure. We exploit the vulnerabilities of latent-based models ( \theta_{f} ), considering two adversarial scenarios: disguising the attack when the model provides an original explanation for a new prediction, and a red herring that further manipulates the explanation, naturally covering up the reason why the prediction changed from benign to malignant. The focus of this paper