Soft-CAM: Making black box models self-explainable for medical image analysis
Kerol Djoumessi 1, Philipp Berens 1,2 · 2025
Read Soft-CAM: Making black box models self-explainable for medical image analysis by Kerol Djoumessi 1, Philipp Berens 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17748.txt.
Introduction
Kerol Djoumessi and Philipp Berens make image networks explain themselves for medical scans in a 2025 paper. Such networks can outscore humans, yet their hidden logic is hard to trust in high-stakes care. Add-on explanations guess at that logic after the fact and often miss it.
Their tweak, Soft-CAM, drops a pooling step and swaps the final layer for one that builds class maps in space. The maps then are the basis of each call, not a sketch drawn later. On three medical data sets, accuracy held while explanations grew sharper in both looks and scores. Built-in clarity, they argue, suits risky calls.
Catalog record
| CREATOR | Kerol Djoumessi 1, Philipp Berens 1,2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Soft-CAM: Making black box models self-explainable for medical image analysis Kerol Djoumessi 1, Philipp Berens 1,2 Abstract Convolutional neural networks (CNNs) are widely used for high-stakes applications like medicine, often surpassing human performance. However, most explanation methods rely on post-hoc attribution, approximating the decision-making process of already trained black-box models. These methods are often sensitive, unreliable, and fail to reflect true model reasoning, limiting their trustworthiness in critical applications. In this work, we introduce SoftCAM, a straightforward yet effective approach that makes standard CNN architectures inherently interpretable. By removing the global average pooling layer and replacing the fully connected classification layer with a convolution-based class evidence layer, SoftCAM preserves spatial information and produces explicit class activation maps that form the basis of the model’s predictions. Evaluated on three medical datasets spanning three imaging modalities, SoftCAM maintains classification performance while significantly improving both the qualitative and quantitative explanation compared to existing post-hoc methods. The code is available at https://github.com/kdjoumessi/SoftCAM . \midlauthor 1 Hertie Institute for AI in Brain Health, University of Tübingen, Germany and 2 Tübingen AI Center, University of Tübingen, Germany † † volume: – Under Review † † year: 2026 † † workshop: Full Paper – MIDL 2026 submission † † editors: Under Review for MIDL 2026 keywords ElasticNet, Self-explainable models, Convolutional Neural Networks, Class Activation Maps, Attribution maps. 1 Introduction Convolutional Neural Networks (CNNs) have revolutionized computer vision by effectively capturing local spatial patterns, reducing the number of parameters, and enabling faster convergence, leading to state-of-the-art performance in tasks such as image recognition and object detection Oquab et al. (2015) ; Li et al. (2021) . However, their lack of interpretability limits adoption in high-stakes fields such as medical image analysis, where transparency and trust are essential. To mitigate this issue, numerous post-hoc saliency- or attribution-based methods have been proposed to explain CNN predictions by highlighting “where” important features appear in the input through pixel-wise importance maps computed after training. Prominent approaches include class activation maps (CAM) techniques Zhou et al. (2016) ; Selvaraju et al. (2017) ; Jiang et al. (2021) , which combine convolutional features with class-specific weights; backpropagation-based methods Springenberg et al. (2014) ; Sundararajan et al. (2017) , which propagate output gradients to the input to estimate pixel relevance; and perturbation-based methods Wang et al. (2020) ; Ivanovs et al. (2021) , which systematically alter input features and measure the corresponding change in the model’s output. While post-hoc attribution-based methods provide intuitive visualizations of discriminative regions, they frequently approximate rather than accurately reflect the model’s true reasoning Adebayo et al. (2018) ; Arun et al. (2021) ; Saporta et al. (2022) and are often valid only under strong regularity assumptions Günther et al. (2025) . Their post-hoc nature further limits their effectiveness—particularly in clinical applications Arun et al. (2021) —due to low faithfulness, reliability, and consistency. Consequently, their visual outputs may not accurately reflect the model’s internal decision-making Adebayo et al. (2018) ; Saporta et al. (2022) . Moreover, post-hoc techniques often struggle to precisely localize disease-relevant regions in medical images Arun et al. (2021) , where the scarcity of ground-truth annotations further complicates validation. To address these challenges, inherently or self-explainable models have been proposed Rudin (2019) , embedding interpretability directly into their architectures Brendel and Bethge (2019) ; Chen et al. (2019) ; Koh et al. (2020) . By coupling prediction with explanation, self-explainable models provide interpretable and transparent insights, in line with human reasoning. However, their reliance on specialized architectures limits generalization to widely used CNN models. Motivated by these limitations, we introduce SoftCAM, a simple yet effective framework that makes CNNs inherently interpretable without relying on post-hoc explanation methods. SoftCAM generalizes the concept of class activation maps to transform conventional CNNs into self-explainable models. By removing the final global average pooling layer and replacing the fully connected classifier with a convolution-based class-evidence layer, SoftCAM converts standard CNNs into fully convolutional architectures that generate explicit, class-specific evidence maps used both for prediction and visual explanation. Evaluated on two widely used CNN architectures, we showed that the resulting SoftCAM-based models maintain competitive accuracy with their black-box baselines while achieving superior interpretability across three clinically relevant medical imaging datasets spanning three modalities. Furthermore, applying ElasticNet regularization, which combines ridge and lasso penalties, on evidence maps improves explanations both qualitatively and quantitatively, while revealing a task-dependent trade-off between sparsity and density. Finally, a comprehensive evaluation against five widely used post-hoc attribution methods showed that SoftCAM consistently outperforms these approaches on various explainability metrics. 2 Method Preliminaries Given an input image \mathbf{X}\in\mathbb{R}^{H_{X}\times W_{X}\times C_{X}} with height H_{X} , width W_{X} , and the number of channels C_{X} , consider a CNN network f_{\theta} that maps \mathbf{X} to a probability distribution over C classes, \mathbf{\hat{y}}=f_{\theta}(\mathbf{X})\in\mathbb{R}^{C} , where y^{c}\in\mathbf{y} denotes the predicted probability for class c . The network consists of a feature extractor g_{\phi} , and a classifier layer h_{\psi} , with learnable parameters \phi and \psi , respectively. The feature extractor produces a feature map \mathbf{Z}=g_{\phi}(\mathbf{X})\in\mathbb{R}^{N\times M\times D} , where N\times M is the spatial resolution and D is the feature dimension (e.g., D=2048 for standard ResNet variants). The classifier then generates the final prediction based on \mathbf{Z} . Let \mathcal{A}=\{\mathbf{A}_{k}\}_{k=1}^{D} denote the set of activation maps from the feature extractor, with A_{k}\in\mathbb{R}^{N\times M} representing the k -th channel. The 2D low-resolution saliency map S^{c}_{\text{Map}}\in\mathbb{R}^{N\times M} provides a visual explanation of the model’s prediction for class c . This work focuses on training self-explainable CNN classifiers that simultaneously produce both the class prediction y^{c} and its corresponding explanation S^{c}_{\text{Map}} . In contrast, traditional CNN are black-boxes: a global average pooling layer reduces \mathbf{Z} to a vector of size 1\times D , which is then fed into one or more linear fully connected layers (FCL) to generate the final prediction. Post-hoc methods are required to explain decisions. 2.1 CAM-based methods Class Activation Maps (CAM) Zhou et al. (2016) are closely related to our approach, providing visual explanations of CNN predictions through class-specific saliency maps. CAM operates by linearly combining the final convolutional feature maps with their corresponding importance weights from the FCL classifier to produce class-wise attribution maps: S_{\text{CAM}}^{c}(x_{1},x_{2})=\sum_{k=1}^{D}w_{k}^{c}A_{k}(x_{1},x_{2}), (1) where A_{k}(x_{1},x_{2}) denotes the activation of the k feature map at location (x_{1},x_{2}) , and w_{k}^{c} is the class-specific importance weight for the k feature map from the fully connected layer. Originally, CAM was designed for CNNs that use a GAP followed by a single FCL. Subsequent extensions, such as GradCAM Selvaraju et al. (2017) and LayerCAM Jiang et al. (2021) , introduced gradient-based approaches that use gradient to compute importance weights, enabling class-specific explanations for various architectures—particularly those with multiple FCLs after the GAP layer, such as VGG Simonyan and Zisserman (2015) . In GradCAM, the importance weights are computed by globally averaging the gradients of the target class score as w_{k}^{c}=\frac{1}{N\times M}\sum_{i}^{N}\sum_{j}^{N}\frac{\partial y^{c}}{\partial A_{k}(i,j)} , where A_{k}(i,j) represents the activation at spatial location (i,j) in the k -th feature map. Following GradCAM, several gradient-based and gradient-free extensions have emerged He et al. (2022) . While gradient-based methods differ primarily in how they aggregate gradients to compute importance weights, gradient-free approaches estimate these weights without backpropagation, often relying on perturbation-based methods like ScoreCAM Wang et