Scaling vision models does not consistently improve localisation-based explanation quality Authors: Mateusz Cedro, Marcin Chlebus Published: 2026 ABSTRACT AbstractArtificial intelligence models are increasingly scaled to improve predictive accuracy, yet it remains unclear whether scale improves the quality of post-hoc explanations. We investigate this relationship by evaluating 11 computer vision models representing increasing levels of depth and complexity within the ResNet, DenseNet, and Vision Transformer families, trained from scratch or pretrained, across the medical and natural image datasets with ground-truth segmentation masks. For each model, we generate explanations using five post-hoc explainable AI methods and quantify mask alignment using two localisation metrics: Relevance Rank Accuracy and the proposed Dual-Polarity Precision, which measures positive attributions inside the class mask and negative attributions outside it. Across datasets and methods, increasing architectural depth and parameter count does not improve explanation quality in most statistical comparisons, and smaller models often match or exceed deeper variants. While pretraining typically improves predictive performance and increases the dependence of explanations on learned weights, it does not consistently increase localisation scores. We also observe scenarios in which models achieve strong predictive performance while localisation precision is near zero, suggesting that performance metrics alone may not indicate whether predictions are based on the annotated regions. These results indicate that larger models do not reliably provide higher-quality explanations, and that explainability should therefore be assessed explicitly during model selection for safety-sensitive deployments.Keywords: Explainable AI, Deep learning, Computer vision, Model complexity, Trustworthy AISubject terms: Computational biology and bioinformatics, Mathematics and computing Introduction Deep neural networks show strong performance across many areas of artificial intelligence. However, their internal decision logic is often difficult to interpret, raising concerns in settings such as healthcare and finance where transparency matters1–4. Explainable AI (XAI) addresses this issue by developing methods that provide insight into how models produce outputs across applications1,5–8. In computer vision, Convolutional Neural Networks (CNNs) and Vision Transformers (ViT) achieve high performance across different tasks, yet are usually treated as black boxes due to their architectural complexity and large number of parameters9–12. Although images are directly interpretable to humans, it is often unclear which visual evidence drives a given prediction. Visual explanation methods, including saliency maps, aim to address this gap by identifying image regions that most influence model output and providing a spatial account of model attention4. Interpretable and explainable AI Research on interpretable machine learning and explainable artificial intelligence (XAI) develops methods that make the behavior of deep learning models more understandable to human users10,14–16. Interpretability describes the extent to which a system makes its reasoning understandable to humans17. Model assessment therefore often considers criteria beyond predictive accuracy, including whether explanations are coherent and reflect the model’s behavior under controlled checks18. Interpretability is commonly described at two levels. Global interpretability characterizes general decision patterns across inputs, while local interpretability explains individual predictions7,19. Together, these perspectives support bias analysis and decision justification in high-stakes settings. A central topic in XAI is feature attribution, which assigns quantitative relevance to input features with respect to a model output6. Methods such as DeepLIFT20, Layerwise Relevance Propagation21, LIME10, and Integrated Gradients6 estimate how inputs contribute to predictions. Game-theoretic approaches, including SHAP, derive feature contributions from cooperative game formulations5,22. In computer vision, these attributions are typically presented as saliency or sensitivity maps, and approaches such as Grad-CAM and NoiseGrad aim to reduce noise and improve the spatial specificity of these visual explanations23–25. Nevertheless, establishing trust in these explanations requires careful evaluation of developed methods26. Validation of feature importance remains challenging because perturbation-based tests that remove or modify features can change the input distribution, which can confound the interpretation of the results6,21. Explainable AI in sensitive imaging applications AI is increasingly used in domains such as healthcare to support decisions including diagnosis and treatment planning, often through deep learning models trained on large labelled imaging datasets27,28. As image-based systems move from rule-based approaches to data-driven methods, model complexity makes it harder for clinicians to understand the basis of individual predictions. This creates a need for methods that provide evidence for why a model produces a given output and that support informed review by domain experts29,30. XAI methods address this need by providing explanations that can be examined alongside model outputs, which helps users judge whether automated recommendations are reliable for the case at hand31–33. Using deep learning systems in practice also involves high computational and economic costs. Increasing model size can improve performance, but gains often decrease relative to the additional training and inference resources required34,35. These constraints motivate approaches that balance performance with compute and operational requirements in settings such as clinical workflows36–39. During the COVID-19 pandemic, many studies develop models that predict disease status from chest X-rays40–43. In this context, saliency maps are widely used to provide spatial explanations, allowing clinicians to assess whether predictions rely on lung regions consistent with pathology rather than on spurious correlations or acquisition artifacts44,45. Influence of model scale on performance and XAI evaluations A common view in machine learning is that larger and more complex models achieve higher predictive accuracy46. In practice, the relationship depends on the task, data regime, and training setup. Increasing depth can improve results, but larger architectures do not always outperform smaller ones. For example, in some settings, compact models such as ResNet-18 reach accuracy comparable to deeper variants47,48. When datasets are limited, increasing capacity can also increase overfitting49. Empirical studies further report diminishing returns when complexity grows beyond a task-dependent range. Additional layers can only yield small gains relative to increased compute, and performance can stagnate or decrease under some conditions34,46,50. These results motivate model selection procedures that consider both computational cost and performance requirements. Scaling laws in deep learning indicate that increasing parameter count often improves predictive performance, but it is unclear whether the same trend holds for the fidelity of model explanations27,51,52. A common assumption is that deeper models learn features with stronger semantic structure, which could yield explanations that align better with human interpretations. This study tests that assumption by examining whether predictive accuracy and interpretability diverge in over-parameterised settings. To separate architectural effects from training variability, the experiments control model depth and weight initialisation to identify factors associated with explanation quality. We evaluate whether increased depth improves the semantic alignment of saliency maps or instead increases gradient noise that reduces their interpretability. Understanding how architectural choices affect both predictive performance and explanation quality is therefore important. This study examines the relationship between model depth and interpretability in computer vision to test whether deeper networks produce higher-quality XAI explanations. The main contributions are as follows: We examine the relationship between model complexity, characterised by the number of trainable parameters, and classification performance. We examine the relationship between model complexity, characterised by the number of trainable parameters, and classification performance. We quantify how model scale and pretraining affect the quality of XAI explanations. We train eleven deep learning models on the medical and natural image datasets, and generate explanations using five XAI methods. We quantify how model scale and pretraining affect the quality of XAI explanations. We train eleven deep learning models on the medical and natural image datasets, and generate explanations using five XAI methods. We evaluate the spatial precision of model predictions with respect to ground truth masks and report this alignment using quantitative measures. We evaluate the spatial precision of model predictions with respect to ground truth masks and report this alignment using quantitative measures. We introduce a new XAI metric, Dual-Polarity Precision (DPP), which addresses the limitations of magnitude-only evaluation by explicitly measuring the spatial precision of both positive supporting evidence and negative counter-evidence, ensuring that explanations are penalized for contradictory or noisy background attributions. We introduce a new XAI metric, Dual-Polarity Precision (DPP), which addresses the limitations of magnitude-only evaluation by explicitly measuring the spatial precision of both positive supporting evidence and negative counter-evidence, ensuring that explanations are penalized for contradictory or noisy background attributions. Methods We evaluate nine ResNet and DenseNet models, together with two Vision Transformers (ViT-B/16 and ViT-L/16)12,53. Seven models are trained from scratch, and four use pretrained weights, including ResNet-50 pretrained on ImageNet-1k, DenseNet-121 pretrained on CheXpert, and ViT-B/16 and ViT-L/16 pretrained on ImageNet-21k54,55. In total, the study includes 11 vision models that differ in architecture and parameter count. Five XAI methods are used to obtain complementary attribution views of model behaviour. The methods are Saliency Maps4, GradientSHAP5, Integrated Gradients6, Feature Permutation56,57, and Grad-CAM25. Explanations are evaluated with two quantitative metrics. Relevance Rank Accuracy1 measures agreement between attribution rankings and ground-truth masks, and the proposed Dual-Polarity Precision measures the fraction of positive pixel attributions that fall inside the ground truth mask and the fraction of negative attributions outside the class-mask. Figure 1 summarises the experimental workflow. Schematic overview of the experimental methodology. Deep learning models are trained or fine-tuned on the image datasets and then used to generate saliency maps for test samples. The resulting explanations are compared with ground truth masks and assessed using Relevance Rank Accuracy13 and the proposed Dual-Polarity Precision to quantify the precision of the XAI outputs. Datasets and preprocessing To assess model interpretability across different visual recognition settings, we use two datasets: the Oxford-IIIT Pet dataset58 (https://www.robots.ox.ac.uk/~vgg/data/pets/) and the Chest X-ray Pneumothorax dataset59 (https://nihcc.app.box.com/v/ChestXray-NIHCC). The datasets differ in size and task difficulty. Binary ground truth masks mark regions of interest in both settings, covering the pneumothorax region annotated by expert radiologists in the medical dataset and object locations in the animal dataset60. These masks enable quantitative comparison of generated explanations against annotated regions. Table 1 summarises the dataset characteristics, and Fig. 2 shows example images with their corresponding masks. Table 1Image and ground-truth masks statistics per dataset.DatasetNumber ofNumber ofAverage maskAverage mask/imageImagesGT masksArea (pixels)Ratio (%)Oxford-IIIT Pet7, 3907, 349Chest X Pneumothorax12, 0472, 669 Image and ground-truth masks statistics per dataset. Examples of images from the datasets58,59 with corresponding ground-truth binary masks60. Masks show the specific segments of interest (disease or an animal) used to evaluate the quality of the XAI explanations. The images in each dataset are randomly divided into training (70%), validation (20%) and test (10%) sets. All inputs are resized to 224 × 224 pixels, converted to single-channel greyscale, and normalised using a mean and standard deviation of 0.5. Data augmentation is applied only to the training set to support generalisation, including random horizontal flips with probability 0.5 and random rotations up to 15 degrees. Validation and test images undergo only resizing and normalisation to keep evaluation conditions consistent with the original data distribution. Models We examine nine convolutional models and two Vision Transformers to study how model complexity relates to the quality of XAI explanations. Seven CNNs are trained from scratch, including ResNet-18, ResNet-34, ResNet-50, and ResNet-10143, as well as DenseNet-121, DenseNet-169, and DenseNet-20161. Training these models under a shared setup supports comparisons across depths within each architecture family. Four additional models use pretrained weights to represent common transfer-learning settings. These include two CNN pretrained models, namely ResNet-50 pretrained on ImageNet-1K_V2 dataset62 containing over a million images, and a DenseNet-121 pretrained on CheXpert dataset63 containing 224,316 chest radiographs. We also include two Vision Transformers, ViT-B/16 (224, IN21k) and ViT-L/16 (224, IN21k), pretrained on more than 14 million images (ImageNet-21k)12. Overall, the experiments cover 11 vision models that differ in architecture and parameter count. Table 2 reports the number of trainable parameters for each model. Table 2Total number of parameters (in millions) for each architecture.ModelNumber of parameters (M)ResNet-1811.2ResNet-3421.3ResNet-5023.5ResNet-10142.5DenseNet-1216.9DenseNet-16912.5DenseNet-20118.1ViT-B/16 (224)85.5ViT-L/16 (224)302.8 Total number of parameters (in millions) for each architecture. Model training setup We treat each task as binary classification and train with a class-weighted binary cross-entropy loss to address class imbalance. For the seven models trained from scratch, we use AdamW with learning rate 0.001, , , and weight decay 0.01. We apply a ReduceLROnPlateau scheduler that reduces the learning rate by a factor of 0.5 when the validation loss does not improve for five epochs. Training uses a batch size of 32 for up to 100 epochs, with early stopping based on class-weighted validation loss and a patience of 12 epochs. The 100-epoch setting serves as a shared upper budget across architectures, while the effective training duration is determined adaptively by validation performance. Pretrained models are fine-tuned in two stages. First, we freeze the feature extractor and train only the classification head for up to 100 epochs, using the same scheduler and early-stopping settings. Second, we unfreeze all layers and continue end-to-end training with the same scheduler while reducing the learning rate to . For reproducibility, we run each experiment with three different random seeds. For each seed, we train the models and perform the XAI analysis. XAI methods We apply five explanation methods that cover gradient-based and perturbation-based approaches. Saliency maps compute pixel-level importance from the gradient of the class score with respect to the input image4. Integrated Gradients addresses gradient saturation by accumulating gradients along a path from a baseline input to the observed image6. GradientSHAP approximates Shapley-style attributions by combining stochastic baselines with gradient information for efficient estimation5. We also use Grad-CAM, which produces a coarse localisation map by weighting feature maps in the last convolutional layer according to their contribution to the predicted class25. Finally, Feature Permutation method estimates importance by shuffling selected input features and measuring the resulting change in the model output56,57. Figure 3 shows an example of all five methods applied to a chest radiograph. Qualitative saliency example on chest radiograph. The top row shows the input X-ray59, the corresponding expert-annotated binary ground-truth mask60, and the model prediction with confidence as shown in the panel header. The bottom row shows attribution heatmaps generated by Saliency, Integrated Gradients, GradientSHAP, Grad-CAM, and Feature Permutation, with the localisation scores (RRA and DPP) reported above each map. Heatmaps show the importance regions, where warmer colours indicate higher attribution and cooler colours indicate lower attribution values. Grad-CAM is upsampled from the final convolutional feature-map resolution, and Feature Permutation is computed over 16× 16 patches and therefore shows blockwise structure. Saliency maps Saliency Maps represent a foundational methodology in XAI. This technique visualises the significance of specific pixels regarding the model predictions. Deep neural networks, due to their complexity and nonlinearity, are not inherently interpretable, unlike simpler models such as linear and logistic regression. Saliency maps address this difficulty by highlighting image regions that exhibit strong correlations to specific classes. Simonyan et al. (2014)4 introduced a technique based on gradients to compute saliency maps. Their method ranks pixels according to their impact on the class score. It approximates the complex function linearly using a Taylor expansion. The procedure involves determining the derivative via backpropagation. The final map displays the magnitude of the gradients. Integrated gradients Integrated Gradients addresses the axiomatic deficiencies found in earlier attribution techniques. Sundararajan et al. (2017)6 introduced this method to satisfy two fundamental properties known as sensitivity and implementation invariance. Sensitivity ensures that any feature responsible for a prediction change receives a non-zero attribution. Implementation invariance guarantees that functionally equivalent networks produce identical explanations regardless of their internal structure. The method defines feature importance by calculating the path integral of the gradients. It traverses a linear path from a baseline input to the actual input. In image classification tasks, the baseline is typically a black image. The integral is often approximated using a Riemann sum. This approach yields a complete attribution where the sum of importance scores equals the difference between the model output for the input and the baseline. GradientSHAP GradientSHAP links gradient-based attribution to Shapley values. Shapley values come from cooperative game theory and assign each player an allocation based on its average marginal contribution across coalitions. Lundberg and Lee (2017)5 adapt this idea to machine learning in the SHAP framework, which assigns a contribution score to each feature while satisfying properties such as local accuracy and consistency. GradientSHAP provides a practical approximation for deep networks by estimating expected gradients along interpolations between a baseline and the input. The method samples points on this path and adds Gaussian noise to the input, then computes gradients of the model output with respect to the perturbed samples. Attributions are obtained by multiplying these gradients by the input-baseline difference and averaging across samples. The approximation relies on assumptions commonly used in SHAP-style methods, including feature independence and approximately linear behaviour between the baseline and the input. Grad-CAM Gradient-weighted Class Activation Mapping (Grad-CAM)25 uses the spatial information retained in the final convolutional layer of a neural network to produce visual explanations. This method computes the gradients of the target score with respect to the feature maps of the last convolutional layer. These gradients are globally averaged to obtain importance weights for each channel. The algorithm calculates a weighted combination of the forward activation maps using these importance weights. A Rectified Linear Unit function is applied to the result to isolate features that have a positive influence on the class of interest. This ensures that the heatmap highlights pixels that contribute to the prediction rather than those that suppress it. The resulting coarse localisation map is upsampled to the resolution of the input image. This process reveals the specific regions responsible for the decision of the model without requiring architectural changes or retraining. For Vision Transformers, Grad-CAM was adapted by targeting the layer normalization after the last transformer encoder block (encoder.layer[-1].layernorm_after) rather than a convolutional layer. This follows common practice when applying CAM-style methods to ViTs, where a normalization layer in the last block is used as the attribution layer and token activations are reshaped into a spatial map64. We implemented this using Captum’s LayerGradCam65, and the resulting heatmap was upsampled to the input resolution using bilinear interpolation. Feature permutation Feature Permutation, also called permutation feature importance, measures how much a trained model relies on an input feature by comparing model performance before and after that feature is permuted on an evaluation set56,57. The baseline metric is computed on the original inputs. For each feature group j, the method then creates a perturbed batch by shuffling the values of group j across samples while keeping all other input values fixed. The model is evaluated on this perturbed batch, and the importance of group j is defined as the change in the metric relative to the baseline. The permutation is repeated multiple times and the results are averaged to reduce sampling variance. For images, we define feature groups as non-overlapping 16 × 16 pixel patches on the resized 224 × 224 inputs, giving a 14 × 14 grid of 196 patches. Group membership is encoded by an integer-valued mask where all pixels in the same patch share the same index. During permutation, all pixels that share an index are shuffled together across the batch, so the same sample-level permutation is applied to every pixel in the patch. The resulting patch score is then assigned to all pixels in that patch to obtain a patch-level importance map. Quantifying the quality of XAI explanations In this study, we define explanation quality as localisation agreement between an attribution heatmap and an expert-annotated binary region-of-interest mask. The evaluation therefore measures how well the explanation concentrates relevance inside the annotated region, rather than broader notions of human usefulness1,13. We compare each heatmap with its ground truth mask using Relevance Rank Accuracy, which reports the fraction of the highest-ranked attribution locations that lie inside the mask, and Dual-Polarity Precision (DPP), which reports a balanced measure of spatial precision by evaluating the localization of both positive supporting evidence and negative counter-evidence. Relevance rank accuracy Relevance Rank Accuracy (RRA)1 evaluates the spatial alignment between the most salient relevance scores and the ground truth. Let GT denote the set of pixel coordinates within the ground truth mask, and let K = |GT| represent the total number of pixels in this mask. To compute the RRA, we extract the top K pixels with the highest relevance values from the input image. The accuracy is then defined as the intersection over K , measuring the proportion of these highest-ranking pixels that fall within the ground truth mask. Formally, we define the set of the top K pixel coordinates as: such that their corresponding relevance values R satisfy , and for any pixel , it holds that . Consequently, the Relevance Rank Accuracy is calculated as: where is the total number of pixels in the ground truth mask. Dual-polarity precision Arras et al. evaluate localisation under the assumption that the major part of relevance should lie within the ground truth mask, either in terms of relevance mass or relevance ranking, and their metrics do not require binarising the heatmap1. In practice, mass-based evaluation is applied to a pooled heatmap, and common pooling choices map signed attributions to non-negative values, for example by squaring or rectifying per-channel relevances before aggregation1. This pooling can remove sign information, which means that negative attributions contribute by magnitude rather than acting as counter-evidence66. In our setting, the expert mask specifies regions expected to contain evidence for the target label. We therefore report a sign-aware metric that evaluates the spatial precision of both (i) positive attributions that indicate supporting evidence and (ii) negative attributions that indicate counter-evidence. We define the positive and negative parts of the attribution map as and . We then compute where GT is the set of pixel locations in the ground truth mask and Ω denotes all pixel locations. The precision of the positive signal is which corresponds to a mass-precision variant of relevance mass accuracy as implemented in common XAI evaluation toolkits13. The precision of the negative signal is which measures whether negatively signed attribution mass is concentrated outside the annotated region. Finally, we define Dual-Polarity Precision as a macro-average of the two terms, DPP takes high values when supporting attribution mass concentrates within the expert mask and negative attribution mass lies primarily outside the mask. We use ε to avoid numerical instability when the total positive or negative mass is close to zero. Complementary use of RRA and DPP While RRA and DPP may correlate, they measure distinct aspects of localisation performance. RRA relies on the ranking of attribution values within the top-K pixels. This metric remains robust to sparse but high-magnitude noise outside the target region. DPP operates as a mass-based metric that accounts for the total magnitude and sign of the attribution signal. Unlike RRA, which focuses on the highest-ranking features, DPP penalizes accumulated attribution mass that contradicts the ground truth. This penalty applies to positive mass in the background and negative mass within the region of interest. A method may therefore achieve a high RRA score by correctly positioning peak pixels but receive a lower DPP score if it generates considerable background noise. The interpretation of the DPP score depends on the polarity of the attribution. A score of 0.5 represents a baseline of random attribution allocation for methods that produce signed attributions. For methods limited to positive attributions the baseline is 0 and the highest possible alignment score is 0.5. We use both metrics to distinguish between a method that achieves local pixel-wise accuracy at its peaks and another that provides an explanation across the entire image. Results Models’ performance Table 3 summarizes the classification performance results. All evaluated architectures achieve high to near-perfect performance on the Oxford-IIIT Pet dataset. The CheXpert model yields the lowest AUC (0.85) on the Oxford-IIIT Pet dataset. This outcome supports the expectation that medical pretraining generalizes poorly to natural images. Smaller models trained from scratch, such as ResNet-18 and DenseNet-121, match or exceed the performance of larger architectures that contain millions of more parameters. Table 3Performance on the test set (means ± standard deviations) averaged across three randomly drawn seeds. Loss values reflect the final validation loss at the best model checkpoint. Training and inference times are reported in seconds. Bolded values indicate the best result per metric, whereas italics values indicate the second-best. Note: the denotes models for which none of the three seeded runs satisfied the convergence criterion within the 100-epoch budget.Dataset (# images)ModelAUC ↑ Loss ↓ Train time (s) ↓ Inference time (s) ↓ Oxford-IIIT Pet(7390)ResNet-18ResNet-34ResNet-50ResNet-50 (IN-1k)ResNet-101DenseNet-121DenseNet-121 (CheXpert) DenseNet-169DenseNet-201ViT-B/16 (224, IN-21k)ViT-L/16 (224, IN-21k)Chest X Pneumothorax(12047)ResNet-18 ResNet-34ResNet-50ResNet-50 (IN-1k)ResNet-101DenseNet-121DenseNet-121 (CheXpert)DenseNet-169DenseNet-201ViT-B/16 (224, IN-21k)ViT-L/16 (224, IN-21k) Performance on the test set (means ± standard deviations) averaged across three randomly drawn seeds. Loss values reflect the final validation loss at the best model checkpoint. Training and inference times are reported in seconds. Bolded values indicate the best result per metric, whereas italics values indicate the second-best. Note: the denotes models for which none of the three seeded runs satisfied the convergence criterion within the 100-epoch budget. Oxford-IIIT Pet (7390) Chest X Pneumothorax (12047) The Chest X Pneumothorax dataset presents the most challenging classification task and results in lower metrics across all models. AUC scores range from 0.77 for the standard ResNet-50 to 0.87 for the pretrained ResNet-50 and the scratch-trained DenseNet-201. Consistent with results from the other dataset, smaller architectures such as ResNet-18 and DenseNet-121 achieve performance levels comparable to or better than their deeper counterparts. We verify the training dynamics for all 66 experimental runs to ensure the validity of the subsequent XAI evaluation. We monitor training and validation loss curves to confirm convergence and prevent overfitting. The application of early stopping ensures that all models are evaluated at their point of optimal generalization. Only the CheXpert-pretrained DenseNet-121 fails to converge across three different seed runs for the Oxford-IIIT Pet dataset. In contrast, this model converges successfully on the Chest X Pneumothorax dataset and achieves the second-best result. Quantitative evaluation of XAI methods We evaluate explanation quality using the Relevance Rank Accuracy and the proposed Dual-Polarity Precision. Table 4 reports results averaged over three random seeds. Table 4Evaluation of explanations obtained from five XAI methods applied on 200 randomly drawn images from the test sets per seed (mean ± standard deviation averaged across 600 images sampled over three seeds). Bolded values indicate the highest scores per category, whereas italics values are the second-best results. Note: indicates model that did not converge within the training budget of 100 epochs across all seeds.ModelRelevance rank accuracy ↑ FeatPermGrad-CAMGradSHAPIntGradSaliencyOxford-IIIT PetResNet-18ResNet-34ResNet-50ResNet-50 (IN-1k)ResNet-101DenseNet-121DenseNet-121 (CheXpert)DenseNet-169DenseNet-201ViT-B/16 (224, IN-21k)ViT-L/16 (224, IN-21k)Chest X PneumothoraxResNet-18ResNet-34ResNet-50ResNet-50 (IN-1k)ResNet-101DenseNet-121DenseNet-121 (CheXpert)DenseNet-169DenseNet-201ViT-B/16 (224, IN-21k)ViT-L/16 (224, IN-21k)ModelDual-polarity precision ↑ FeatPermGrad-CAMGradSHAPIntGradSaliencyOxford-IIIT PetResNet-18ResNet-34ResNet-50ResNet-50 (IN-1k)ResNet-101DenseNet-121DenseNet-121 (CheXpert)DenseNet-169DenseNet-201ViT-B/16 (224, IN-21k)ViT-L/16 (224, IN-21k)Chest X PneumothoraxResNet-18ResNet-34ResNet-50ResNet-50 (IN-1k)ResNet-101DenseNet-121DenseNet-121 (CheXpert)DenseNet-169DenseNet-201ViT-B/16 (224, IN-21k)ViT-L/16 (224, IN-21k) Evaluation of explanations obtained from five XAI methods applied on 200 randomly drawn images from the test sets per seed (mean ± standard deviation averaged across 600 images sampled over three seeds). Bolded values indicate the highest scores per category, whereas italics values are the second-best results. Note: indicates model that did not converge within the training budget of 100 epochs across all seeds. Across all datasets, RRA and DPP scores remain low for most attribution methods and model variants, despite strong classification performance. The Oxford-IIIT Pet dataset yields the highest interpretability scores among the evaluated datasets. ResNet-50 pretrained on ImageNet achieves the best result in seven of ten comparisons, with DenseNet-121 and ResNet-50 accounting for the remaining best scores. Although ImageNet pretraining improves performance in this setting, smaller architectures still match or exceed larger models in several comparisons, nevertheless, pretrained ViTs do not show considerable advantage of pretraining over models trained from scratch. For DPP, Grad-CAM produces clearly higher values than the one-sided attribution baseline of 0.0, whereas methods that produce signed attributions remain close to the 0.5 baseline, indicating weak polarity alignment with the ground-truth signal. On the Chest X Pneumothorax dataset, RRA values are near zero and DPP values are near the baseline scores of 0.5 across models and attribution methods. This pattern indicates poor localization of the relevant pathological regions, even when classification performance is high. One possible explanation is the small spatial extent of the pneumothorax masks. On average, the annotated region covers 1.4% of the image area, which is far smaller than in the other dataset, where the masks cover 29.7% for Oxford-IIIT Pet (Table 1). This limited coverage makes the evaluation more sensitive to small localization errors and can depress RRA and DPP even when models capture coarse diagnostic cues. Among the low scores, the DenseNet-121 and DenseNet-169 yield the highest scores among the tested models. The XAI results for the scratch-trained ResNet-50 on the Chest X Pneumothorax dataset should be interpreted cautiously, as this model achieved lower classification performance than the other CNN variants in that setting. Overall, these results suggest that the models may rely on cues outside the annotated regions rather than the intended anatomical evidence. Additional examples are presented in Supplementary Figures 1 to 4. Overall, increasing the number of model parameters does not consistently improve explanation quality under rank (RRA) or mass (DPP) metrics used in this study. Deeper variants such as ResNet-101, DenseNet-201, and ViT-L/16 do not show systematic gains over smaller architectures on the datasets considered in this study. Although three seeds provide only a minimal estimate of inter-seed variability, the standard deviations are consistently small, likely reflecting stable aggregate localisation behaviour across initialisations and averaging over 200 randomly sampled test images per single seed, resulting in 600 evaluated images in total across the three seeds. Statistical analysis of differences in the XAI results We assess whether the observed differences in explanation quality are statistically significant by analysing RRA and DPP scores across all XAI methods and datasets. Experiments are grouped into five comparison settings: ResNet scale (comparing depths 18, 34, 50, and 101), DenseNet scale (comparing depths 121, 169, and 201), Vision Transformer scale (comparing pretrained ViT-B/16 and ViT-L/16) ResNet pretraining (comparing ResNet-50 trained from scratch with an ImageNet-pretrained variant), and DenseNet pretraining (comparing DenseNet-121 trained from scratch with a CheXpert-pretrained variant). Since both Vision Transformers are included only as pretrained models, the ViT-B/16 versus ViT-L/16 comparison should be interpreted as a pretrained-model comparison across scale rather than as a direct analogue of the CNN scale comparisons. For multi-group comparisons in the ResNet and DenseNet scale settings, we use the Kruskal-Wallis H-test. For binary comparisons in the ViT scale and pretraining settings, we use the Mann-Whitney U-test. All tests are run on explanations computed for 200 randomly sampled test images per seed, resulting in 600 images in total per group. We report effect sizes alongside test results. Cliff’s δ is used for pairwise tests, and is used for multi-group tests, using standard interpretation thresholds. To control the false discovery rate under multiple testing, we adjust all p-values with the Benjamini-Hochberg procedure. We estimate statistical power post hoc with bootstrap resampling. For each comparison, we resample the observed group distributions with replacement 1,000 times, re-run the corresponding test, and define power as the fraction of resamples with p<0.05. This approach is used because closed-form power calculations are not available for the Mann-Whitney U and Kruskal-Wallis tests. Because power is evaluated at α =0.05 rather than the Benjamini-Hochberg adjusted threshold, the resulting estimates are slightly optimistic and interpreted as indicative only. Full results are reported in Supplementary Table 1 and Supplementary Table 2. Table 5 and Table 6 summarise these results by reporting only statistically significant comparisons and the corresponding model ordering. Table 5Statistical comparison of XAI methods across model scales, ranked 1–4th by Relevance Rank Accuracy (RRA) and Dual-Polarity Precision (DPP) localization scores. p-values are Benjamini-Hochberg adjusted. Effect sizes are (Kruskal-Wallis) or δ (pairwise Mann-Whitney U). Lighter cells denote smaller models. Shared superscripts denote tied ranks. Statistical comparison of XAI methods across model scales, ranked 1–4th by Relevance Rank Accuracy (RRA) and Dual-Polarity Precision (DPP) localization scores. p-values are Benjamini-Hochberg adjusted. Effect sizes are (Kruskal-Wallis) or δ (pairwise Mann-Whitney U). Lighter cells denote smaller models. Shared superscripts denote tied ranks. Table 6Statistical comparison of XAI methods between scratch-trained and pretrained variants, ranked 1st–2nd by relevance rank accuracy (RRA) and dual-polarity precision (DPP) localization scores. p-values are Benjamini-Hochberg adjusted. Effect sizes are δ (pairwise Mann-Whitney U). Lighter cells denote scratch-trained models, darker denote pretrained versions. Shared superscripts denote tied ranks. Note: indicates model that did not converge within the training budget of 100 epochs across all seeds. Statistical comparison of XAI methods between scratch-trained and pretrained variants, ranked 1st–2nd by relevance rank accuracy (RRA) and dual-polarity precision (DPP) localization scores. p-values are Benjamini-Hochberg adjusted. Effect sizes are δ (pairwise Mann-Whitney U). Lighter cells denote scratch-trained models, darker denote pretrained versions. Shared superscripts denote tied ranks. Note: indicates model that did not converge within the training budget of 100 epochs across all seeds. Analysis of model scale effect We next describe results that differ significantly between model scales, followed by results with no detected differences. Table 5 summarises the XAI methods for which scale-dependent differences are observed. Across 60 tests assessing the effect of model scale on explanation quality, 21 tests (35%) yield statistically significant differences after correction. In 3 of the 60 comparisons (5%), the largest model within a family achieves the highest score. In the remaining 18 significant comparisons (30%), smaller models match or exceed the performance of the largest model. For 39 of 60 tests (65%), we do not reject the null hypothesis of equal explanation quality across model scales. Nevertheless, the associated effect sizes are typically negligible to small, indicating that even when differences are statistically significant, they are often limited in practical importance. Analysis of pretraining effects Table 6 summarises the XAI methods for which differences are observed between models trained from scratch and pretrained models. Across 40 tests assessing the effect of pretraining on explanation quality, 16 tests (40%) yield statistically significant differences after correction. In 5 of the 40 comparisons (12.5%), pretrained models achieve higher scores than their scratch-trained counterparts. In the other 11 significant comparisons (27.5%), scratch-trained models match or exceed the pretrained models. For the remaining 24 tests (60%), we do not reject the null hypothesis of equal explanation quality between the training variants. The observed effect sizes are also mostly negligible to small. For Oxford-IIIT Pet, however, DenseNet pretraining results involving the CheXpert-pretrained DenseNet-121 are marked with and should be interpreted for completeness rather than as primary evidence for the effect of pretraining, since this model did not satisfy the convergence criterion on that dataset. Fidelity tests for explanations We evaluate explanation fidelity using two sanity-check procedures applied to trained models. The first test measures dependence on learned parameters through layer-wise randomisation. We reinitialise the weights in ten randomly selected layers and compare explanations from the perturbed model with those from the original model. We define degradation as one minus the absolute correlation between the two explanation maps. Larger values indicate that weight randomisation changes the explanation, which indicates dependence on learned parameters rather than network structure alone. The second test measures sensitivity to input structure through input randomisation. We shuffle pixels in the test images and compute explanations for both the original and shuffled inputs. We quantify change using the distance between the corresponding attribution maps and report Cliff’s δ as an effect size for differences between the distance distributions. Under this test, explanations are expected to change when the input content is disrupted by pixel shuffling. Table 7 reports results for selected ResNet and DenseNet models together with pretrained ViT-B/16 and ViT-L/16, and Table 8 reports results for ResNet-50 trained from scratch and a ImageNet-pretrained variant. All metrics are computed on a batch of 32 randomly drawn test images per single seed from the Oxford-IIIT Pet dataset. We use this dataset because it yields the highest interpretability scores in our experiments and therefore provides favourable conditions for explanation quality. If a method fails these tests under these conditions, it is likely to fail on datasets where localisation performance is lower. For Vision Transformers, the sanity-check analysis is limited to pretrained models and should therefore be interpreted as a pretrained-model comparison across scale. Table 7Sanity check analysis on the Oxford-IIIT Pet dataset across selected ResNet, DenseNet, and pretrained Vision Transformer models. We evaluate Layer-wise Relevance Randomization via Mean Degradation (where higher values (↑ ) indicate dependence on trained weights) and Input Randomization via Mean Sensitivity (where higher values (↑ ) indicate responsiveness to input changes). Cliff’s δ effect sizes are provided to interpret the magnitude of changes in explanation maps under input randomization.ModelMethodLayer-wise relevance randomisation (degradation) ↑ Input randomisation (sensitivity) ↑ Input randomisation Cliff’s δ (interp.)ResNet-18Saliency0.12 (Negligible)Integrated Gradients-0.41 (Medium)GradientSHAP-0.20 (Small)Grad-CAM-0.01 (Negligible)Feature Permutation-0.20 (Small)ResNet-101Saliency0.15 (Negligible)Integrated Gradients-0.28 (Small)GradientSHAP-0.47 (Medium)Grad-CAM-0.04 (Negligible)Feature Permutation-0.07 (Negligible)DenseNet-121Saliency0.09 (Negligible)Integrated Gradients-0.23 (Small)GradientSHAP0.20 (Small)Grad-CAM0.17 (Small)Feature Permutation-0.44 (Medium)DenseNet-201Saliency0.33 (Medium)Integrated Gradients0.71 (Large)GradientSHAP0.07 (Negligible)Grad-CAM0.15 (Negligible)Feature Permutation0.04 (Negligible)ViT-B/16 (224, IN-21k)Saliency-0.47 (Medium)Integrated Gradients0.15 (Negligible)GradientSHAP-0.15 (Negligible)Grad-CAM-0.09 (Negligible)Feature Permutation0.25 (Small)ViT-L/16 (224, IN-21k)Saliency-0.20 (Small)Integrated Gradients-0.01 (Negligible)GradientSHAP0.12 (Negligible)Grad-CAM-0.15 (Negligible)Feature Permutation-0.15 (Negligible) Sanity check analysis on the Oxford-IIIT Pet dataset across selected ResNet, DenseNet, and pretrained Vision Transformer models. We evaluate Layer-wise Relevance Randomization via Mean Degradation (where higher values (↑ ) indicate dependence on trained weights) and Input Randomization via Mean Sensitivity (where higher values (↑ ) indicate responsiveness to input changes). Cliff’s δ effect sizes are provided to interpret the magnitude of changes in explanation maps under input randomization. Table 8Pretraining sanity check on the Oxford-IIIT Pet dataset for ResNet-50 trained from scratch and an ImageNet-pretrained ResNet-50. Metrics are defined as in Table 7.ModelMethodLayer-wise relevance randomisation (degradation) ↑ Input randomisation (sensitivity) ↑ Input randomisation Cliff’s δ (interp.)ResNet-50Saliency-0.20 (Small)Integrated Gradients-0.15 (Negligible)GradientSHAP-0.20 (Small)Grad-CAM-0.17 (Small)Feature Permutation-0.31 (Small)ResNet-50 (IN-1k)Saliency-0.04 (Negligible)Integrated Gradients-0.25 (Small)GradientSHAP-0.01 (Negligible)Grad-CAM0.12 (Negligible)Feature Permutation-0.04 (Negligible) Pretraining sanity check on the Oxford-IIIT Pet dataset for ResNet-50 trained from scratch and an ImageNet-pretrained ResNet-50. Metrics are defined as in Table 7. Sanity checks for models differing in scale Layer-wise randomisation yields moderate to high degradation across methods, indicating dependence on learned parameters. For the selected ResNet models, degradation is higher for ResNet-101 than for ResNet-18 for all methods except Grad-CAM. A similar pattern is observed for the DenseNet models, where DenseNet-201 shows higher degradation than DenseNet-121 for all tested XAI methods. For the pretrained Vision Transformers, degradation is also generally high, with ViT-B/16 exceeding ViT-L/16 for Saliency, GradientSHAP, Grad-CAM, and Feature Permutation, whereas Integrated Gradients remains nearly unchanged across the two models. Under pixel permutation, most gradient-based methods show low sensitivity, with attribution maps changing only slightly when spatial structure is removed. For ResNets, Grad-CAM show larger shifts than the other methods. For the DenseNet models, the largest changes appear for Grad-CAM and Feature Permutation. For the pretrained Vision Transformers, input sensitivity remains low for most methods, with Grad-CAM near zero for both ViT-B/16 and ViT-L/16, while Saliency and GradientSHAP show slightly larger changes. Cliff’s δ is usually negligible to small across methods and architectures, with only six of the 30 comparisons reaching the medium or large range. For the ResNet models, medium effects occur for Integrated Gradients on ResNet-18 and for GradientSHAP on ResNet-101. For the DenseNet models, Feature Permutation on DenseNet-121 and Saliency on DenseNet-201 show medium effects, while Integrated Gradients on DenseNet-201 shows a large effect. For the pretrained Vision Transformers, most effects remain negligible to small, with a medium effect observed only for Saliency on ViT-B/16. Overall, differences between architectures follow similar patterns across methods. Sanity checks for trained vs pretrained models We compare ResNet-50 trained from scratch with an ImageNet-pretrained variant. The pretrained model shows higher degradation under layer-wise randomisation, indicating stronger dependence of the explanations on learned representations and a larger loss of structure after weight reinitialisation. Sensitivity to pixel permutation increases with pretraining. The largest increase occurs for Saliency, from 0.003 to 0.082. Cliff’s δ shifts from mostly small negative effects in the scratch-trained model to negligible negative effects for Saliency and Grad-CAM in the pretrained model, while remaining low overall. Figures 4 and 5 illustrate example outputs under layer and input perturbations. In Fig. 4, Saliency, Integrated Gradients, and GradientSHAP methods produce visually noisy maps, however, RRA decreases after perturbations, while DPP remains close to 0.5, indicating the assignment of positive and negative attributions similar to a random baseline. Feature Permutation and Grad-CAM show clearer drops after layer randomisation, which is reflected in decreases in both RRA and DPP. Sanity checks for an ImageNet-pretrained ResNet-50 across five XAI methods under layer-wise weight randomisation. Explanations are shown for an image using the original model (top) and after layer randomisation (bottom). Columns show attribution maps from Saliency, Integrated Gradients, GradientSHAP, Feature Permutation, and Grad-CAM. For each method, the scores above the maps report DPP and RRA, and the coefficient r shows the correlation between the attribution maps generated by the original and perturbed model. Sanity checks for an ImageNet-pretrained ResNet-50 across five XAI methods under input (pixel) randomisation. Explanations are shown for two image samples using the original input (top) and a pixel-permuted input (bottom). Columns show attribution maps from Saliency, Integrated Gradients, GradientSHAP, Feature Permutation, and Grad-CAM. For each method, the scores above the maps report DPP and RRA, the values give the distance between the attribution maps from the original and perturbed input image, and the colour bar indicates normalised attribution magnitude. Figure 5 shows the effect of input randomisation. After pixel shuffling removes spatial structure, explanations change and RRA decreases for most methods. The highest decrease is observed for Grad-CAM, where the attribution focused on the cat’s face, despite the misclassification, becomes diffuse hotspots after shuffling. These patterns match the sensitivity measures and indicate that explanations generally depend on image structure. For Grad-CAM we observe a considerable decrease in both RRA and DPP metrics. Overall, deeper and especially pretrained models yield explanations that depend more on learned parameters and tend to show greater sensitivity to spatial input structure. This is reflected in higher degradation under layer-wise randomisation and higher sensitivity under pixel permutation. Additional examples are provided in Supplementary Figures 5 and 6. Discussion Across datasets and XAI methods, increasing model scale does not yield consistent improvements in explanation quality. In the statistical analysis with Benjamini-Hochberg adjusted p-values, many comparisons fail to reject the null hypothesis of equal interpretability scores across scales. When significant differences are observed, the largest models achieve the highest scores in only 5% of the comparisons, while in the remaining significant cases, smaller models match or exceed their larger counterparts. Moreover, the associated effect sizes are usually negligible to small, which suggests that even statistically significant differences often correspond to only modest practical differences in explanation quality. This finding challenges the assumption that larger models produce more meaningful insights due to their higher representational capacity. It suggests that while deeper models may capture abstract features for classification, these representations do not necessarily translate into better pixel-level attributions. Simpler models often provide explanations that align as well or better with ground truth masks compared to bigger counterparts. Moreover, the statistical analysis does not show a consistent advantage of pretraining in RRA and DPP. After Benjamini-Hochberg correction, 60% of the pretraining comparisons are not significant, and among the significant results, fewer than one third (5/16) show a significant improvement for pretrained models over scratch-trained counterparts. Mean scores are often higher for the ImageNet-pretrained model, but these differences are not systematic enough to translate into a clear overall gain. The sanity checks provide additional evidence about how explanations depend on model parameters. In scratch-trained models, smaller ResNet and DenseNet variants show slightly lower degradation, which suggests weaker dependence on the trained parameter configuration. The input randomisation across different ResNet and DenseNet model scales show changes that are mostly negligible to small across the sanity checks, indicating limited scale-related variation within these families under this evaluation. For the pretrained Vision Transformers, the sanity checks likewise show mostly negligible to small differences between ViT-B/16 and ViT-L/16, with a medium effect observed only for Saliency on ViT-B/16, which indicates limited variation across scale within this pretrained-only comparison. Pretrained ResNet-50 model show larger degradation under layer-wise weight randomisation with respect to the scratch-trained model, indicating that its attribution maps change more when learned weights are disrupted. Taken together, these findings suggest that pretraining strengthens the link between explanations and learned parameters even when aggregate localisation metrics such as RRA and DPP do not consistently improve. Results on the Chest X Pneumothorax dataset illustrate a gap between predictive performance and explanation quality that is relevant for medical model evaluation. Models reach reasonable classification performance, yet the corresponding explanations are not aligned with the target regions, and interpretability scores remain near zero. This pattern is consistent with models relying on confounds or imaging artefacts rather than anatomically meaningful cues. These findings support the use of explanation-based metrics for auditing, since evaluation based only on accuracy can miss failures in how predictions are formed. Influence of explanation resolution on localization metrics We acknowledge that the varying resolutions of the evaluated XAI methods introduce a trade-off when applying pixel-level metrics. Unlike gradient-based methods, which produce fine-grained pixel-level attributions, Grad-CAM generates coarse localization maps by upsampling activation maps from the final convolutional layer. This inherent smoothness may confer an advantage in mass-based evaluations (e.g., DPP) by producing contiguous regions of high relevance that align well with larger ground-truth masks, effectively filtering out the high-frequency gradient noise that penalizes pixel-wise methods. However, our results indicate that this resolution bias does not solely dictate explanation quality. For instance, Feature Permutation also operates at a coarse granularity, assigning importance to 16 × 16 pixel patches, yet it yielded distinct performance trends compared to Grad-CAM. Notably, while Grad-CAM performance varied considerably between pretrained and scratch models, Feature Permutation remained relatively stable. This divergence suggests that while upsampling artifacts can influence absolute scores, the relative differences we observed across model scales and pretraining regimes reflect genuine variations in the semantic quality of learned representations rather than simple resolution artifacts. Implications for robustness and reliability Robustness to distribution shift and adversarial perturbations is relevant for clinical AI use. Although this study does not include direct robustness evaluations, the interpretability results indicate potential failure modes. Models trained from scratch sometimes fail to localise task-relevant regions despite high classification performance, most clearly in the Pneumothorax dataset, which is consistent with reliance on confounds or dataset artefacts and reduced stability under shifts in acquisition conditions or patient populations. ImageNet-pretrained models show better alignment with ground truth and higher sensitivity in input-randomisation tests, suggesting stronger dependence on image structure. These observations support using RRA and DPP alongside accuracy when assessing reliability under distribution shift. Future work should test this link directly under controlled out-of-distribution and adversarial settings. Limitations and future work This study evaluates explanation quality in binary classification settings. However, multi-class problems may show different interactions between model scale, training regime, and interpretability, particularly when classes share overlapping visual cues. The evaluation also depends on ground-truth segmentation masks that define regions of interest. While these masks support quantitative localisation metrics, they may not represent the full set of cues used in medical decision-making. The metrics treat overlap with the mask as a proxy for explanation quality, which can be limiting when clinically relevant context lies outside the annotated region. In addition, heatmap-based measures do not cover explanation approaches defined at the level of human concepts or objects. For example, TCAV67 estimates the influence of a concept through concept activation vectors and directional derivatives without producing pixel-level attributions. Extending the analysis to concept-based methods would require concept annotations or object-level intervention protocols in addition to segmentation masks. Our experiments focus on post-hoc explanations for standard convolutional and transformer architectures. Models designed for interpretability, such as ProtoPNet68, provide an alternative that links predictions to learned prototypes and can reduce sensitivity to gradient noise. We do not include such models to maintain a consistent architecture family across scale comparisons. Future work can test how explanation quality in prototype-based and related architectures changes with model size. Future work could consider architecture-specific training schedules and longer optimization budgets to further examine the robustness of the present findings. In addition, further validation on larger and more diverse datasets would strengthen these findings. The use of greyscale inputs may affect both predictive performance and explanations in non-medical domains where colour carries signal. A practical constraint is that large image datasets rarely provide segmentation masks needed for localisation-based evaluation, as many benchmarks include only class labels. One direction is to use multi-rater datasets such as LIDC-IDRI, where soft masks derived from inter-annotator agreement allow probabilistic assessment of feature importance69. Finally, quantitative measures such as RRA and DPP capture aspects of localisation but do not address whether an explanation supports expert decision-making. In sensitive domains, such as medical image analysis, studies with radiologists that assess usefulness, consistency, and failure modes across explanation methods would provide complementary evidence about clinical relevance. Conclusion This study examines how model scale relates to the quality of post-hoc visual explanations produced by saliency-style and CAM-style methods. We evaluate eleven architectures (ResNets, DenseNets, and Vision Transformers) across two datasets using five explanation methods. Explanation quality is assessed through spatial alignment with ground-truth masks using Relevance Rank Accuracy (RRA) and the proposed Dual-Polarity Precision (DPP). Across datasets and methods, larger models do not show a consistent advantage in localisation quality. Most scale comparisons fail to reject the null hypothesis after Benjamini-Hochberg correction, and when significant differences occur, the largest model achieves the highest score in only 5% of scale comparisons. Smaller architectures, including ResNet-18, DenseNet-121, and ViT-B/16, often match or exceed deeper variants such as ResNet-101, DenseNet-201, and ViT-L/16 as measured by RRA and DPP. Pretraining affects explanations more than scaling in many settings. While improvements in RRA and DPP are not uniform, pretrained models show larger degradation under weight randomisation and higher sensitivity under input randomisation. The results also show that predictive performance and explanation quality can diverge. On the Pneumothorax task, models reach reasonable classification performance while producing near-zero alignment scores, which indicates that correct predictions can be obtained without localising the annotated regions. This supports evaluating explainability alongside accuracy when selecting models for clinical use. Overall, model selection should not assume that the largest architecture provides the most reliable explanations. Smaller models can provide similar explanation quality with lower computational cost. Future work should extend the evaluation to multi-class settings and concept-based explanations, and should test whether RRA and DPP relate to reliability under controlled distribution shifts and adversarial perturbations. Acknowledgements This work uses the NIH Chest X-ray dataset provided by the NIH Clinical Center. Author contributions Mateusz Cedro conducted research, experiments and wrote the manuscript. Marcin Chlebus supervised the research. Funding Mateusz Cedro received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen”. Data availability The datasets used in this study are available at: https://www.robots.ox.ac.uk/ vgg/data/pets/ and https://nihcc.app.box.com/v/ChestXray-NIHCC. Code availability The code used in this study is accessible at: https://github.com/mateuszcedro/XAI-at-scale.git. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used in this study are available at: https://www.robots.ox.ac.uk/ vgg/data/pets/ and https://nihcc.app.box.com/v/ChestXray-NIHCC. The code used in this study is accessible at: https://github.com/mateuszcedro/XAI-at-scale.git.