From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation
Kristoffer Wickstrøm, Marina Höhne, Anna Hedström · 2024
Read From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation by Kristoffer Wickstrøm, Marina Höhne, Anna Hedström (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05592.txt.
Introduction
Software that explains its decisions sounds trustworthy, but judging the explanations is itself slippery: there are no ground-truth labels for what a good explanation is. Kristoffer Wickstrøm and co-authors show how that slipperiness can be gamed. Their 2024 paper frames the problem as an attack on evaluation itself.
Evaluation tools come with many knobs, and turning them in seemingly innocent ways can swing the verdict on which explanation method wins. The team demonstrates the trick across several datasets, models, and explanation methods, with large swings in outcome. Their proposed shield is to rank methods across many knob settings at once, so no single lucky setting decides. The authors close with a plea for open, whole-picture evaluation in explainable AI.
Catalog record
| CREATOR | Kristoffer Wickstrøm, Marina Höhne, Anna Hedström |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation Kristoffer Wickstrøm, Marina Höhne, Anna Hedström Abstract The lack of ground truth explanation labels is a fundamental challenge for quantitative evaluation in explainable artificial intelligence (XAI). This challenge becomes especially problematic when evaluation methods have numerous hyperparameters that must be specified by the user, as there is no ground truth to determine an optimal hyperparameter selection. It is typically not feasible to do an exhaustive search of hyperparameters so researchers typically make a normative choice based on similar studies in the literature, which provides great flexibility for the user. In this work, we illustrate how this flexibility can be exploited to manipulate the evaluation outcome. We frame this manipulation as an adversarial attack on the evaluation where seemingly innocent changes in hyperparameter setting significantly influence the evaluation outcome. We demonstrate the effectiveness of our manipulation across several datasets with large changes in evaluation outcomes across several explanation methods and models. Lastly, we propose a mitigation strategy based on ranking across hyperparameters that aims to provide robustness towards such manipulation. This work highlights the difficulty of conducting reliable XAI evaluation and emphasizes the importance of a holistic and transparent approach to evaluation in XAI. Code is available at https://github.com/Wickstrom/quantitative-xai-manipulation . 1 Introduction Explainable artificial intelligence (XAI) is a crucial research area to ensure trustworthiness in computer vision [ 44 ] , which contains a wide range of methods that provide explanations for the output of a predictive model [ 7 , 38 , 53 ] . To determine which XAI method is suitable for a given problem setting, quantitative evaluation analysis is necessary to provide an objective measurement for comparison. Such quantitative analysis of XAI methods has made great leaps forward over the last couple of years [ 26 , 2 ] , and generally consists of evaluating several metrics that measure desirable properties that an XAI method should have i.e., metric-based quality estimation [ 24 ] . However, the progress in XAI and its evaluation has led to an overwhelming variety of methods and metrics, making it challenging for researchers to navigate their choices [ 33 , 24 , 12 , 25 ] . A fundamental limitation in XAI evaluation is the lack of ground truth explanation labels [ 24 ] . Since such information is generally not available, we approximate explanation quality by measuring desirable properties like faithfulness [ 43 , 4 , 40 , 17 , 9 , 41 ] , complexity [ 39 , 9 , 16 ] , or robustness [ 37 , 3 , 55 , 17 ] and translate these properties into empirical tests . In this translation, a challenge appears in the parameterization of the empirical tests. For example, how do we mask out pixels and how large should the masks be? Preliminary works [ 24 , 33 ] have shown that the evaluation outcomes are sensitive to choices like these. This sensitivity underscores the need to investigate the impact of hyperparameter choice, making it an important research area to ensure the reliability of XAI evaluations. This challenge becomes particularly prominent for evaluation methods with many hyperparameters that must be set, since it is generally not possible to find an objective measure of the optimal set of hyperparameters. For instance, faithfulness evaluations view model behavior changes as signals of explanations quality, with substantial changes reflecting the explanations faithfulness [ 7 , 43 , 4 , 40 , 17 , 9 ] . This type of evaluation often requires replacing pixel values with some baseline value, which can be highly data-dependent and difficult to tune [ 46 ] . Furthermore, it can often be computationally impractical to evaluate all possible choices for hyperparameters. Therefore, hyperparameters are usually selected normatively with the researcher’s own subjective judgment, frequently drawing on prior studies. Since there is variation in what hyperparameters are being used in the community [ 33 , 24 ] , there is some flexibility in selecting hyperparameters from an acceptable set of possible choices. XAI method Faithfulness score ( \downarrow ) LRP 25.19 Saliency 20.23 Kernel SHAP 23.94 XAI method Faithfulness score ( \downarrow ) LRP 19.31 Saliency 22.96 Kernel SHAP 24.87 Table 1 : Faithfulness comparison of XAI methods on MNIST before (left table) and after manipulation (right). Here, the different between the left and right table is the perturbation methods used (uniform noise vs. blurring, respectively). Both perturbation methods are commonly used, but completely change the outcome of the evaluation. In this work, we demonstrate how this flexibility of XAI evaluation can be exploited to manipulate the evaluation outcome. By making seemingly small changes to hyperparameters that are widely used in the literature, the outcome of the faithfulness evaluation can change completely. Tab. 1 illustrates this, where standard XAI methods are compared with only slight changes in hyperparameters but with significant changes in evaluation outcome. We propose to frame the finding of these small changes as an optimization problem that manipulates the evaluation, where either the evaluation of a single XAI method is manipulated or the evaluation of multiple XAI methods is manipulated jointly. Our contributions are: C1 A method-specific manipulation method that can increase the evaluation score for a specific XAI method, which we entitle intra-manipulation . C2 A holistic manipulation method that can manipulate the quantitative comparison of several XAI methods, which we entitle inter-manipulation . C3 A comprehensive experimental analysis on manipulation of faithfulness evaluation that demonstrate how the evaluation outcome can completely change after manipulation. C4 Towards improving the robustness of quantitative evaluation of XAI, we propose Mean Resilience Rank, a ranking-based procedure that reduces the sensitivity to hyperparameter manipulation. Our findings have significant implications for the XAI community. Quantitative evaluation is crucial to provide objective measurements of explanation quality, which can be used to select an appropriate method for a particular task or for comparison in method development. If these measurements can be easily altered, it reduces the trustworthiness of both method selection and comparison. Therefore, the findings and solutions in this work are of critical importance for the community both by highlighting the issue of manipulation and by presenting strategies towards mitigating the issue. 2 Related Work Metric-based Quality Estimation Quantitative analysis of XAI explanation has improved considerably in recent years, and researchers now have a vast amount of evaluation metrics at their disposal [ 26 , 2 ] . Due to the lack of ground truth explanations, researchers try to quantify the quality of an explanation by measuring desirable properties, which can be categorized into 6 families of properties [ 26 ] ; faithfulness [ 10 ] , robustness [ 3 ] , localisation [ 51 ] , complexity [ 16 ] , randomisation [ 1 ] , and axiomatic [ 30 ] . Within each family, a variety of metrics exists. Prior Studies on Hyperparameter Sensitivity in XAI Increasing attention has been given to the influence and potential confounding effects of hyperparameters in XAI evaluations [ 24 ] . These studies vary in defining dependent versus independent variables and the hyperparameter space of intervention, be it model, explanation, or evaluation space. Studies have examined the sensitivity of attribution methods to explanation hyperparameters like random seed