Visual Explanation via Similar Feature Activation for Metric Learning
Yi Liao, Ugochukwu Ejike Akpudo, Jue Zhang, Yongsheng Gao, Jun Zhou, Wenyi Zeng, Weichuan Zhang · 2025
Read Visual Explanation via Similar Feature Activation for Metric Learning by Yi Liao, Ugochukwu Ejike Akpudo, Jue Zhang, Yongsheng Gao, Jun Zhou, Wenyi Zeng, Weichuan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01636.txt.
Introduction
Yi Liao and six co-authors devise a way to explain, with pictures, why a metric learning model judges two images as similar. Their paper came out in 2025. Metric learning models compare images by distance between compact summaries, and they lack the built-in classifier layer that standard explanation maps rely on.
Their method scores each feature channel by how much it contributes to the similarity verdict between a pair of images. Weighting the model's internal feature maps by those scores produces a heat map showing the regions that drove the match. Tests, both measured and visual, show the maps give clear and trustworthy explanations. The method works with the two common distance choices, straight-line distance and angle-based similarity.
Catalog record
| CREATOR | Yi Liao, Ugochukwu Ejike Akpudo, Jue Zhang, Yongsheng Gao, Jun Zhou, Wenyi Zeng, Weichuan Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01636
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visual Explanation via Similar Feature Activation for Metric Learning Yi Liao, Ugochukwu Ejike Akpudo, Jue Zhang, Yongsheng Gao, Jun Zhou, Wenyi Zeng, Weichuan Zhang Abstract Visual explanation maps enhance the trustworthiness of decisions made by deep learning models and offer valuable guidance for developing new algorithms in image recognition tasks. Class activation maps (CAM) and their variants (e.g., Grad-CAM and Relevance-CAM) have been extensively employed to explore the interpretability of softmax-based convolutional neural networks, which require a fully connected layer as the classifier for decision-making. However, these methods cannot be directly applied to metric learning models, as such models lack a fully connected layer functioning as a classifier. To address this limitation, we propose a novel visual explanation method termed Similar Feature Activation Map (SFAM). This method introduces the channel-wise contribution importance score (CIS) to measure feature importance, derived from the similarity measurement between two image embeddings. The explanation map is constructed by linearly combining the proposed importance weights with the feature map from a CNN model. Quantitative and qualitative experiments show that SFAM provides highly promising interpretable visual explanations for CNN models using Euclidean distance or cosine similarity as the similarity metric. † † address: ⋆ Griffith University † Shaanxi University of Science and Technology 1 Introduction Deep neural networks, particularly convolutional neural networks (CNN), have led to unprecedented breakthroughs in image classification tasks. Most image classification models require fully connected (FC) layers as classifiers that are trained via supervised learning, which requires annotated training datasets for a specific task. They make the classification decision by the highest probability in all classes. The probability is calculated via the softmax function, which is heavily dependent on the FC layer (classifier). These models, commonly known as softmax-based CNNs, can effectively recognize a testing sample from the class domain of the training set but struggle to identify a testing image from the unseen class. Unfortunately, it is impossible to collect the training images that cover endless classes, necessitating the studies on metric learning classification models that attract attention in computer vision community. Metric learning models can accurately recognize novel classes that are disjoint from the training set by making classification decisions based on similarity measurements rather than a linear classifier. Metric learning has proven effective for tasks such as few-shot learning [ 1 ] and image retrieval [ 2 ] . Figure 1: (A). CAM-based methods(e.g.,CAM [ 3 ] , Grad-CAM [ 4 ] , and Relevance-CAM [ 5 ] ) can not be used for interpreting metric-learning CNNs because of no FC layer as the classifier. (B). The proposed SFAM can generate the explanation map for the metric learning model, which highlights the body of the two birds as the similar feature for decision-making. Visual explanation methods aim at interpreting image classification models by highlighting where the model looked, i.e. the contributing pixels on an input image that predict the class label. However, much less work has focused on exploring the visual explanation for metric learning models. Current widely used explanation methods, including, but not limited to, RISE [ 6 ] , Class Activation Map (CAM) [ 3 ] , GradCAM-based methods [ 4 , 7 , 8 , 9 ] , Relevance-CAM [ 5 ] , LIFT-CAM [ 10 ] , and Ablation-CAM [ 11 ] , are developed to interpret softmax-based CNNs. As shown in Fig. 1 (A), the absence of a FC layer in these CNNs’ classifiers limits these visual explanation methods for metric learning [ 12 ] . Against the conventional use of Euclidean distance as the similarity metric, state-of-the-art visual explanation methods for metric learning [ 13 , 14 , 15 ] predominantly employ the cosine similarity metric for CNN explanation. However, visualizing the decision-making regions on input images for a metric-agnostic CNN models remains a challenge. To solve this problem, we propose a novel visual explanation algorithm called Similar Feature Activation Map (SFAM). The proposed SFAM aims to highlight discriminative image pixels/regions that contribute to a metric learning model’s decision-making logic. To this end, we propose a channel-wise contribution importance score (CIS) that assigns a weight to each channel. A class prediction can be explained by assigning a weight to each feature, indicating its influence, with the key to the explanation being the contribution of individual features. Specifically, the comparison between two vectors representing two input images respectively, the closer the two elements from the same channel are, the more important the channel will be. A single convolution kernel is responsible for capturing an individual feature, and the contribution of an arbitrary channel can represent the contribution of that feature. The proposed CIS, serving as the importance coefficients, can be linearly combined with the feature map out of the final layer of a CNN model to create the SFAM. Hence, the proposed SFAM highlights similar features by awarding a higher value, as shown in Fig. 1 (B). In a word, the key contribution in this work can be summarized as follows, • We propose a novel interpretable algorithm SFAM for generating visual explanation maps that highlight similar features looked at by metric-learning CNN models in few-shot learning image classification and image retrieval tasks. • The proposed SFAM has a flexible architecture, allowing the use of cosine similarity metric or Euclidean distance metric, unlike existing methods, which typically rely on cosine similarity for metric learning. • The proposed SFAM provides visual explanations for any metric-learning CNN without any constraints to the CNN architecture. • The quantitative and qualitative experiments demonstrate the effectiveness of the proposed method in few-shot image classification and image retrieval tasks. 2 Related Work 2.1 Visual Explanation for Softmax-based CNN Models The previous works [ 16 , 17 ] attempt to interpret CNNs by generating the saliency maps, which highlights the contributing pixels relevant to the prediction of a specific class. As the first effort to interpret CNNs by highlighting the contributing regions on an input image, CAM [ 3 ] is constructed by linearly combining the output feature map and the weights from the FC layer (classifier). Grad-CAM [ 4 ] , Grad-CAM++ [ 7 ] , XGradCAM [ 8 ] , and LayerCAM [ 9 ] all use the gradient of the classification score w.r.t. the neurons to generate the explanation map. Ablation-CAM [ 11 ] compute the importance of the feature map by using the variation of classification score obtained via multiple iteration. Relevance-CAM [ 5 ] inevitably requires the number of classes to calculate the initial relevance score when interpreting a CNN. RISE [ 6 ] interprets CNNs by randomized perturbations on input image. However, the above explanation methods are all built on the classification score, the computation of which relies heavily on the FC layer as the classifier. Therefore, they cannot be adopted to interpret metric-learning CNN models. 2.2 Visual Explanation for Metric Learning The aforementioned explanation methods for softmax-based CNNs are ineffective for metric-learning CNNs, which do not utilize a fully connected (FC) layer as a classifier for decision-making. To address this limitation, several methods have been developed, including Decomposition [ 13 ] , RAM [ 14 ] , CG-RAM [ 15 ] , and Grad-FAM [ 12 ] , to generate visual explanations. Decomposition [ 13