Feature salience – not task-informativeness – drives machine learning model explanations

Benedict Clark, Marta Oliveira, Rick Wilming, Stefan Haufe · 2026

Read Feature salience – not task-informativeness – drives machine learning model explanations by Benedict Clark, Marta Oliveira, Rick Wilming, Stefan Haufe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09238.txt.

Introduction

Clear AI is meant to show why a model decides as it does, including shortcut flops, on the guess that traits flagged key must carry target facts. Benedict Clark and three co-authors test that guess with a set trial: deep models trained on image sorting with see-through marks that are absent, class-tied mixes, or class-free noise.

Across five popular credit methods, mark zones score far higher weight in all settings, no matter the training setup, while whether marks foretell the class barely matters. Credit acts much like model-blind edge spotting, and flips when bright strengths are coded by small rather than large values. The driver of weight, the paper concludes, is trait pop, not how much it tells.

Catalog record

CREATORBenedict Clark, Marta Oliveira, Rick Wilming, Stefan Haufe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09238

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Feature salience – not task-informativeness – drives machine learning model explanations Benedict Clark, Marta Oliveira, Rick Wilming, Stefan Haufe Abstract Explainable AI (XAI) promises to provide insight into machine learning models’ decision processes, where one goal is to identify failures such as shortcut learning. This promise relies on the field’s assumption that input features marked as important by an XAI must contain information about the target variable. However, it is unclear whether informativeness is indeed the main driver of importance attribution in practice, or if other data properties such as statistical suppression, novelty at test-time, or high feature salience substantially contribute. To clarify this, we trained deep learning models on three variants of a binary image classification task, in which translucent watermarks are either absent, act as class-dependent confounds, or represent class-independent noise. Results for five popular attribution methods show substantially elevated relative importance in watermarked areas (RIW) for all models regardless of the training setting ( R 2 ≥ .45 R^{2}\geq.45 ). By contrast, whether the presence of watermarks is class-dependent or not only has a marginal effect on RIW ( R 2 ≤ .03 R^{2}\leq.03 ), despite a clear impact impact on model performance and generalisation ability. XAI methods show similar behaviour to model-agnostic edge detection filters and attribute substantially less importance to watermarks when bright image intensities are encoded by smaller instead of larger feature values. These results indicate that importance attribution is most strongly driven by the salience of image structures at test time rather than statistical associations learned by machine learning models. Previous studies demonstrating successful XAI application should be reevaluated with respect to a possibly spurious concurrency of feature salience and informativeness, and workflows using feature attribution methods as building blocks should be scrutinised. 1 Introduction While Machine Learning (ML) increasingly solves complex problems, trained models often display undesirable behaviour upon deployment. Models may unduly rely on protected attributes ( 19 ; 13 ) . or rely on confounding relationships between in- and outputs, such as chest tubes and portable radiographic markers, rather than actual disease biomarkers ( 6 ; 5 ) , thereby bypassing the direct causal relationships they might be expected to learn. Such reliance on confounding, often termed shortcut learning ( 10 , e.g.,) , can lead to poor generalisation when these associations shift or vanish in test data ( 15 ; 10 ) . Diagnosing and correcting such failure modes is one of the primary goals of explainable artificial intelligence ( 41 , XAI,) . Feature attribution methods are XAI tools that assign importance scores to individual input dimensions. It is frequently assumed that these scores reveal ‘what models have learned’, which could then be used to inform subsequent scientific follow-up experiments or to diagnose and possibly improve models, training data, or test inputs. In a prominent example, Layer-wise Relevance Propagation (LRP) was demonstrated to highlight a photographer’s watermark in a horse vs. car classification task ( 32 ) . As the watermark was unevenly distributed across classes, it acted as a confounder; the authors attribute this discovery to the method’s ability to identify the model’s reliance on the artifact. Similar workflows are used in medical imaging to verify if models track pathology rather than confounding artifacts ( 2 ; 48 ) . Formally, confounding in machine learning occurs when an unobserved variable C causally influences both an input feature X_{i} and the target Y , inducing a statistical association that makes X_{i} informative with respect to Y ( 39 ) . Conversely, suppression occurs when an input feature X_{\text{coll}} (a collider ) is caused by both the target Y and a feature X_{\text{supp}} unrelated to Y (a suppressor ). While suppressors are statistically independent of the target (and thus non-informative ), they introduce non-target-related variance to the collider. As a result, optimal ML models must utilise suppressors to denoise informative features ( 14 ; 22 ; 50 ) . In image classification, a non-informative watermark overlapping with a target object could act as a suppressor, prompting an optimal model to use the watermark to remove it from the object. Critically, these causal structures have distinct implications: unstable confounding can hurt generalisation, whereas suppression often aids model performance. Consequently, 51 formulate the presence of a statistical association between feature and target as a necessary condition for feature importance. Theoretical and empirical evidence suggests, though, that popular attribution methods violate this condition by systematically assigning importance to non-informative suppressors ( 22 ; 29 ; 51 ; 53 ) , a finding recently extended to non-linear settings ( 12 ; 36 ; 52 ) . It is currently unclear whether task-informativeness can, nonetheless, be considered the main driver of feature importance in practice. We hypothesise that suppressor features, novel features not contained in the training data, and features with highly salient structure (e.g., watermarks, sharp edges), might also receive substantial importance attribution. To test this, we conduct two controlled studies using local watermarks insertions and global lightness manipulations. We evaluate attribution behaviour across three experimental settings: a confounded setting where the manipulation is class-specific; a balanced setting where the manipulation is non-informative (potentially inducing suppression effects); and a baseline setting where manipulations are absent during training but present at test time. By contrasting these settings, we isolate whether attributions really respond to what models have learned (i.e., confounding or suppression), to novel features, or simply to the low-level properties of the test image itself. In both studies, we train models on datasets generated in all three settings and evaluate the trained models on images with and without experimental manipulations applied. We measure the relative importance attributed to the watermarked area compared to the entire image (RIW) and the relative importance attributed to the lightness channel compared to all three channels of an HLS colour model (RIL). With these metrics, we investigate which properties of data and models are most strongly associated with high importance attribution: discriminative information, non-informative suppression, structures unobserved during training, salient low-level properties of the test image (such as sharp edges within the watermarked area or elevated lightness across the entire image), or none of the above. 2 Results We report on extensive experiments in which we studied the dependence of feature attributions on fundamental data and model properties. To this end, we created synthetic image data by altering real images via controlled manipulations in two different studies – watermarks and lightness . In each, we contrast three different experimental settings where the manipulated image features possess different statistical properties: a confounded settings, where manipulated features are informative with respect to the target, a balanced settings, in which the same manipulated features are non-informative (yet, potentially induce suppression), and baseline/no-watermark settings, in which again the same features constitute artifacts with respect to the training distribution – they only occur during model evaluation on test data. Features differing in such properties require different treatment when the goal is to identify actual failure modes and to take corrective measures. 2.1 Watermark study In the watermark study, N_{\text{cat}}=4,800 images of cats and the same number of dog images were obtained from Kaggle ( 46 ) , forming the baseline, or no-watermark , dataset \mathcal{D}_{\text{no-wm}} . Next, a static translucent watermark was superimposed

Read the complete text