Interpretable Image Classification via Non-parametric Part Prototype Learning
Zhijie Zhu 1, Lei Fan 1, Maurice Pagnucco 1, Yang Song 1 · 2025
Read Interpretable Image Classification via Non-parametric Part Prototype Learning by Zhijie Zhu 1, Lei Fan 1, Maurice Pagnucco 1, Yang Song 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10247.txt.
Introduction
Zhijie Zhu, Lei Fan, Maurice Pagnucco, and Yang Song wrote this paper in two thousand twenty-five. It classifies images by object parts the model can point to, so each call comes with a reason a person can check.
Part prototype networks mimic how people judge: this looks like that beak, that wing. Past builds lean on dull, repeated parts, which thin the story they tell. Here the parts are learned by clustering deep traits from foundation vision models, with no set shape imposed. The parts come out distinct and varied for each class. Two new scores grade how distinct and how full the account is. On bird, car, and dog sets the method matches past scores while reading far better.
Catalog record
| CREATOR | Zhijie Zhu 1, Lei Fan 1, Maurice Pagnucco 1, Yang Song 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interpretable Image Classification via Non-parametric Part Prototype Learning Zhijie Zhu 1, Lei Fan 1, Maurice Pagnucco 1, Yang Song 1 Abstract Classifying images with an interpretable decision-making process is a long-standing problem in computer vision. In recent years, Prototypical Part Networks has gained traction as an approach for self-explainable neural networks, due to their ability to mimic human visual reasoning by providing explanations based on prototypical object parts. However, the quality of the explanations generated by these methods leaves room for improvement, as the prototypes usually focus on repetitive and redundant concepts. Leveraging recent advances in prototype learning, we present a framework for part-based interpretable image classification that learns a set of semantically distinctive object parts for each class, and provides diverse and comprehensive explanations. The core of our method is to learn the part-prototypes in a non-parametric fashion, through clustering deep features extracted from foundation vision models that encode robust semantic information. To quantitatively evaluate the quality of explanations provided by ProtoPNets, we introduce Distinctiveness Score and Comprehensiveness Score. Through evaluation on CUB-200-2011, Stanford Cars and Stanford Dogs datasets, we show that our framework compares favourably against existing ProtoPNets while achieving better interpretability. Code is available at: https://github.com/zijizhu/proto-non-param . 1 Introduction Figure 1 : a. Existing methods [ 39 , 19 ] encode various assumptions as regularizations to guide prototype learning, but often fail to diversify the concepts learned by each prototype. Here multiple part-prototypes attend to the same image region, thereby limiting their interpretability. In contrast b. Our method partitions the feature space into semantically distinct clusters, and updates each prototype with the empirical mean of its respective cluster, enabling each prototype to learn semantically different concepts. In recent years, self-explainable networks have emerged as a promising approach for improving the transparency of deep neural networks by providing inherent interpretability. Unlike traditional explanation methods that operate post-hoc [ 32 , 48 ] , recent approaches aim to mimic human reasoning by justifying predictions through an interpretable decision-making process. Examples include Prototypical-Part Networks (ProtoPNets) [ 7 , 11 ] , part discovery methods [ 36 , 21 ] , and Concept Bottleneck Models [ 23 , 38 , 12 , 47 ] , which offer explanations derived from object concepts. Among these, ProtoPNets have garnered attention as they provide intuitive visual explanations rooted in past examples. Specifically, they define a number of prototypes for each object category, each representing a distinguishable object part, and make the prediction based on the similarity of an input image to prototypes, thereby emulating the process of human visual reasoning. However, existing ProtoPNets train prototypes as learnable parameters, and guide their learning process with the core assumption that a region within each input image should match one of the prototypes from the corresponding category. However, we observe that multiple prototypes from the same class often correspond to the same object part, which hinders the quality of explanations provided by the network in two ways: First, explanations become non-comprehensive, covering only a subset of object parts. Second, this results in redundant and repetitive explanations that fail to adequately illustrate why the object is classified in a particular way. An example is shown in Figure 1 .a, out of the four most activated prototypes in a ProtoPNet, three of them focus on overlapping regions near the bird’s wing, ignoring other parts such as the bird’s legs and tail. Ideally, within an interpretability framework, the model is expected to recognize and differentiate a comprehensive set of distinctive parts for each object category, thus enhancing the clarity and transparency of its decision-making process. To approach this problem, we take inspiration from recent advances in prototype-based classification [ 49 , 46 ] , and encode distinct object parts using non-parametric prototypes [ 50 , 41 ] , as shown in Figure 1 .b. Non-parametric prototypes were originally proposed to learn a more robust feature space by capturing the rich variance within each class through multiple prototypes [ 50 ] . We extend this idea to part-based interpretability to address both of the aforementioned limitations. Specifically, in comparison to previous methods that guide the learning process of learnable prototypes with regularizations [ 39 ] , non-parametric prototypes allow finer control over the representation of each prototype within the same class by applying constraints on feature clustering. As a consequence, each prototype is able to capture unique characteristics of an object that differentiate it from others, resulting in semantically distinct prototypes, each latent feature patch of the foreground object is able to contribute to one of the prototypes, allowing the overall representation for each class to embody a comprehensive set of associated concepts. These properties greatly improve the interpretability of the explanations. Additionally, in order to obtain robust feature representations that encode object part semantics, We explore advanced foundation models as backbones to extract features, specifically utilizing self-supervised Vision Transformers pre-trained with self-distillation (DINO-ViTs) [ 6 , 26 ] , given that these backbone networks capture rich semantic information about object parts and can be used out-of-the-box [ 27 , 2 ] . We also propose an efficient fine-tuning strategy for self-supervised ViT backbones to adapt the feature space to the target domain while grounding the embedding of object parts to its assigned prototypes. Hence, our framework consists of two training stages where the initial stage focuses on prototype discovery, and the subsequent stage is directed towards feature refinement. Only the image-level class labels are required in both stages. We conduct experiments to compare our approach with existing ProtoPNets on the CUB-200-2011 [ 37 ] , Stanford Cars [ 24 ] , and Stanford Dogs datasets [ 10 ] . Moreover, to evaluate the diversity of concepts captured by ProtoPNets, we introduce two new metrics: Distinctiveness Score , which quantifies the extent of overlap among object parts learned by prototypes within the same class, and Comprehensiveness Score , which measures whether the prototypes comprehensively attend to different parts of the foreground object. To summarize the key contributions of this work: • We identify a key limitation of existing ProtoPNets, specifically generating repetitive and redundant visual explanations. To this end, we leverage non-parametric prototypes to build a self-explainable network. • Our method comprehensively explores non-parametric prototypes and foundation vision backbones, discovering diverse and semantic distinctive concepts. • We introduce two evaluation metrics, namely Distinctiveness Score and Comprehensiveness Score to quantify the diversity and comprehensiveness of object part concepts. We demonstrate that our method yields comparable results against state-of-the-art ProtoPNets models on three benchmarking datasets while recognizing holistic and distinctive concepts from each category. 2 Related Work Part-based Interpretable Image Classification. Drawing inspiration from the human ability to distinguish objects by recognizing their semantic parts, the Prototypical Part Network (ProtoPNet) [ 7 ] emulates this reasoning process by comparing image patches with learned prototype vectors that represent object parts specific to each class. Several variants have been proposed to enhance both its interpretability and classification accuracy [ 11 , 35 , 5 , 45 ] . Notably, TesNet [ 39 ] reformulates prototype learning from the perspective of constructing embedding space on Grassmann Manifold, and also improves the overall performance by applying various priors on prototypes, such as orthonormality. Building upon the architecture