Are machine learning interpretations reliable? A stability study on global interpretations
Luqin Gan, Tarek M. Zikry, Genevera I. Allen · 2025
Read Are machine learning interpretations reliable? A stability study on global interpretations by Luqin Gan, Tarek M. Zikry, Genevera I. Allen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15728.txt.
Introduction
Luqin Gan, Tarek M. Zikry, and Genevera I. Allen test whether tools that explain machine learning models give steady answers in a paper from 2025. Such tools aim to build trust in high-stakes uses. Yet an explanation that shifts with small noise is hard to trust.
The team ran the first large, systematic check of global explanation methods. They used both guided and unguided tasks on table data. Small random changes to data or code often changed the explanations a lot. In fact, explanations proved less steady than the predictions themselves. No single method won across all data sets, so the authors urge routine stability checks.
Catalog record
| CREATOR | Luqin Gan, Tarek M. Zikry, Genevera I. Allen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 127 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15728
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are machine learning interpretations reliable? A stability study on global interpretations Luqin Gan, Tarek M. Zikry, Genevera I. Allen Abstract As machine learning systems are increasingly used in high-stakes domains, there is a growing emphasis placed on making them interpretable to improve trust in these systems. In response, a range of interpretable machine learning (IML) methods have been developed to generate human-understandable insights into otherwise black box models. With these methods, a fundamental question arises: Are these interpretations reliable? Unlike with prediction accuracy or other evaluation metrics for supervised models, the proximity to the true interpretation is difficult to define. Instead, we ask a closely related question that we argue is a prerequisite for reliability: Are these interpretations stable? We define stability as findings that are consistent or reliable under small random perturbations to the data or algorithms. In this study, we conduct the first systematic, large-scale empirical stability study on popular machine learning global interpretations for both supervised and unsupervised tasks on tabular data. Our findings reveal that popular interpretation methods are frequently unstable, notably less stable than the predictions themselves, and that there is no association between the accuracy of machine learning predictions and the stability of their associated interpretations. Moreover, we show that no single method consistently provides the most stable interpretations across a range of benchmark datasets. Overall, these results suggest that interpretability alone does not warrant trust, and underscores the need for rigorous evaluation of interpretation stability in future work. To support these principles, we have developed and released an open source IML dashboard and Python package to enable researchers to assess the stability and reliability of their own data-driven interpretations and discoveries. 1 1 footnotetext: Department of Statistics, Rice University, Houston, TX 2 2 footnotetext: Departments of Statistics, Columbia University, New York, NY 3 3 footnotetext: Center for Theoretical Neuroscience, Zuckerman Mind Brain Behavior Institute, Columbia University, New York NY 4 4 footnotetext: [email protected] 1 Introduction In recent years the field of Interpretable Machine Learning (IML) has arisen, a class of ML methods focused "the use of machine learning techniques to generate human understandable insights into data, the learned model, or the model output" ( 10 ) . These methods are employed to generate data-driven discoveries in both unsupervised and supervised settings, with a focus on opening up the black box of machine learning to users and practitioners across domains, particularly critical in domains such as health, security, and even finance, where ML methods have been applied to approve or deny mortgages ( 126 ) . Existing review papers have defined concepts of IML methods, provided systematic taxonomy and principles ( 143 ) , and discussed its importance in various aspects including model validation [ 40 , 87 , 17 ] , model debugging [ 82 , 40 ] , model transparency & accountability [ 40 , 68 , 51 , 122 , 29 ] , trust [ 57 , 98 , 113 , 123 , 39 , 104 , 98 ] , ethics [ 82 , 37 , 68 , 38 , 132 , 29 ] , and scientific discoveries [ 102 , 82 , 40 , 68 , 38 , 132 , 148 ] . Though these terms have been defined before, we redefine them here for clarity: • Reproducibility: Do you observe the same results when the pipeline is rerun on the same data? • Replicability: Is your code and pipeline easily usable to run on unseen data? • Reliability: Do you observe similar results to the ground truth? • Generalizability/Predictability: Can you predict your results or observe similar results on unseen data? • Stability: How sensitive are your results to random data or algorithm perturbations? Here, we believe a broad prerequisite criterion to trust interpretations is reliability . For example, from the perspective of knowledge discovery, a new finding cannnot be accepted if the discovery is unreliable after many repeated experiments. Therefore, we consider the reliability of the interpretations to be significant in building trustworthy AI systems. As machine learning methods become ever present in a wide range of critical domains, including biomedicine and drug discovery ( 49 ) , physics ( 27 ) , national security ( 54 ) , finance ( 36 ) , climate science ( 84 ) , transportation ( 144 ) , and more [ 128 , 6 ] , significant attention has been drawn towards the quality of interpretations generated by ML models. In supervised learning tasks such as regression and classification, interpretations include feature importance ranking for a predictive model, whereas in an unsupervised task such as clustering and dimension reduction, interpretations are in the groupings generated, either in the high-dimensional space or a low-dimensional embedding. These interpretations are widely used across domains, either as standalone findings such as ranking features important in aircraft engine longevity ( 11 ) , or as part of step in a downstream analysis, such as identifying groups of drug-resistant cancer cells ( 149 ) . Though vast bodies of work have been devoted to ML methods, and recently, IML methods, the reliability of interpretations themselves have not been deeply explored. As compared to traditional statistical tasks such as prediction, in which we can easily measure its accuracy with the true response, the quality of interpretations from a machine learning model are more difficult to evaluate, as ground truth interpretations are rarely observed, making measures of reliability challenging. In lieu of being able to directly compare to ground truth interpretations, many have turned to studying stability as a practical proxy; if an interpretation changes dramatically under small perturbations to the data or model, it is unlikely to be reliable. Since the inception of the idea to study feature selection in regularized models ( 79 ) , further work has arisen for feature interactions [ 18 , 69 ] , graphical models ( 71 , 86 ) , and dimension reduction ( 139 , 26 ) . In clustering, it is commonly referred to as consensus clustering ( 85 ) , where it has gained popularity for identifying optimal hyperparameters in unsupervised class discovery ( 137 , 138 , 72 ) . Though widely advocated for validating reproducible discovery ( 10 , 143 ) , stability analyses can quickly become computationally burdensome, and the choice of perturbation (noise addition, different train/test splits, random reinitializations) and evaluation metrics are not always immediately clear, and can require specific methodological knowledge [ 141 , 10 ] . Despite the large volume of IML-related research on supervised learning methods [ 40 , 105 , 37 , 48 , 122 ] , there is no systematic framework to evaluate the reliability of the interpretations on unsupervised scientific discovery. [ 70 ] explored the impact of prediction accuracy on the quality interpretability in terms of accuracy and stability of feature importance ranking. However, this work focuses only on feature importance and intrinsic global IML methods. Many past efforts focus on the reliability of model-specific methods, such as [ 147 ] who investigated the uncertainties in the surrogate interpretability model LIME [ 98 ] due to sampling procedure, and sampling proximity and explained model credibility across different observations. Similarly, [ 8 ] provides a framework to