Constructing sensible baselines for Integrated Gradients
Jai Bardhan, Cyrin Neeraj, Mihir Rawat, Subhadip Mitra · 2024
Read Constructing sensible baselines for Integrated Gradients by Jai Bardhan, Cyrin Neeraj, Mihir Rawat, Subhadip Mitra (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13864.txt.
Introduction
Integrated gradients is a popular method for asking which inputs drove a model's answer, but its verdicts hinge on the baseline it compares against. Jai Bardhan and co-authors show in this 2024 paper that the common choice, an all-zeros baseline, gives poor attributions. Their case study comes from particle physics, where background events offer a natural comparison set.
Averaging baselines sampled from those background events produced consistently more sensible attributions. The lesson reaches beyond physics: an explanation method is only as good as the nothing it measures against. The authors argue that designing baselines from real background data should be standard practice when explaining black-box models in science.
Catalog record
| CREATOR | Jai Bardhan, Cyrin Neeraj, Mihir Rawat, Subhadip Mitra |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13864
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Constructing sensible baselines for Integrated Gradients Jai Bardhan, Cyrin Neeraj, Mihir Rawat, Subhadip Mitra Abstract Machine learning methods have seen a meteoric rise in their applications in the scientific community. However, little effort has been put into understanding these “black box” models. We show how one can apply integrated gradients (IGs) to understand these models by designing different baselines, by taking an example case study in particle physics. We find that the zero-vector baseline does not provide good feature attributions and that an averaged baseline sampled from the background events provides consistently more reasonable attributions. Introduction Machine learning (ML) models are now used in almost all branches of science, with various large-scale efforts like AlphaFold ( Jumper et al. 2021 ) showing good promise. This has been possible mainly because of the current availability of large amounts of data and compute to train these models. As these models (and their impacts) grow and become more autonomous, it becomes crucial to question their understanding of the world and gauge what influences their decisions. With this motivation, we explore the interpretability of the simplest deep learning architecture – the fully connected deep neural network (DNN) used in many domains of science and engineering. For example, in the case of bioinformatics, morphological features from cell imaging, such as size, shape, texture of nuclei, etc., may be used as input to train the model to predict whether the cell is cancerous or not ( Zhu et al. 2020 ) . Similarly, in the domain of astronomy, features derived from the light curves, such as rise time, peak luminosity, decay rate, etc., can be used as features for DNNs to classify supernovae from the background ( Smith and Geach 2023 ) . In the case of High Energy Physics, features from particle collisions, such as the collision energy, jet types, etc., are used for predicting whether the collision has emerged from an interesting signal event ( Aad et al. 2020 ) . Approaches to interpretability can be broadly divided into two types. The first of the two types work by designing interpretable model architectures and training paradigms. These paradigms trade off performance for interpretability and are harder to work with, which has prevented their widespread adoption. The second type of interpretability works in a post-hoc approach where we try to understand the inner workings of the model which we have already trained. Here, we focus on post-hoc interpretability, in particular on a tool called Integrated Gradients (IGs) ( Sundararajan, Taly, and Yan 2017 ) . IGs provide attributions for each input feature to indicate how relevant it was for the model’s prediction for a particular sample. IG-based feature attributions are calculated by linearly interpolating along the feature space from the baseline to the data sample and evaluating the model prediction at each interpolated step. The choice of baseline depends on the problem at hand. For example, for image classification tasks, a completely dark image (zero pixel intensity) can be used as a baseline as each pixel is then devoid of any information; any image data sample can be constructed from a dark image to see which pixels are more important. An in-depth analysis of IGs in the context of image classification is found in Ref. ( Sturmfels, Lundberg, and Lee 2020 ) . This intuition of a baseline with zero information can be extended to classification tasks in particle physics. If jets are represented as images centred around the jet axis, such a baseline could be used to derive feature attributions. Ref. ( Apolinário et al. 2021 ) used randomly initialised images as baselines, which are then interpolated to correctly classify samples of quenched jets to derive pixel attributions. However, if we move away from the domain of image data to other data representations, there are no clear-cut intuitive prescriptions. For example, in jet classification using point clouds or event classification tasks where the data is represented as graphs or n -dimensional vectors of kinematic features. To use powerful tools like IGs, it is important to investigate baselines for each representation used in particle physics to understand the predictions of deep learning models. In this article, we investigate baseline constructions for a case study on event classification in particle physics, where events are represented by kinematic features. We train a model to distinguish a new physics signal (e.g., from heavy (vectorlike) quarks production) from the Standard Model backgrounds. The baselines will focus on the question that is of most interest to the particle physics searches: according to the DNN model, what makes the signal events differ from the background ones? As typical new physics signal events at collider experiments occur at a tiny rate compared to those from backgrounds, this will help us to identify features that are important to tag signal events, compare the features’ attributions to our intuitions to a reasonable degree and build better, economical models that will speed up the searches at the collider experiments. Integrated Gradients IGs explain an ML model’s predictions by assigning an attribution score to each input feature, which reflects the contribution of that feature to the predictions. These attributions serve two main purposes: (1) they allow us to assess the trustworthiness of the model’s prediction, and (2) they help us identify relevant and distinguishing features between the classes. IGs work on the principle that the contribution of each input feature to a model’s prediction can be estimated by integrating the gradient along a path from a baseline input to the actual input. Formally, the importance of the i^{\text{th}} input is expressed as \phi_{i}^{IG}(f,\mathbf{x},\mathbf{x^{\prime}})=(x_{i}-x_{i}^{\prime})\int_{\alpha=0}^{1}\frac{\delta f(x^{\prime}+\alpha(x-x^{\prime}))}{\delta x_{i}}d\alpha, (1) where x and x^{\prime} represent the features of the input and the baseline input, respectively. The attribution score is blind to the common information between the baseline and the input x . Depending on the construction of x^{\prime} , the attributions obtained from Eq. 1 will be different and, therefore, the inferences from the model. However, we have a few additional domain-specific constraints: (1) the baseline must be from the data distribution, and (2) the baseline must be physically intuitive. Then, one would be able to infer from the feature attributions that, for example, compared to the baseline, the features f_{1},f_{2},\dots,f_{k} (in the order of importance) make this sample more < input class > -like as per the DNN model. If the baseline does not contain any information, one could claim, for example, the information in features f_{1},f_{2},\dots,f_{k} (in the order of importance) when presented to the DNN makes the sample more < input class > -like. A baseline analogous to the dark image baseline in image classification tasks (i.e., a vector of all zeros) is unnatural for model inferences in particle physics. Firstly, the zero vector may not belong to the data distribution, i.e., there could be features that cannot even take the value 0 . Another reason is that the zero vector may not actually be a baseline void of any information; for instance, rotation angles in physics can take values -\pi to \pi where a value of 0 would indicate something specific of the setup. Thirdly, the zero baseline generally does not correspond to a physical object, which would make interpretations even more difficult. Averaged Baselines Averaging