SHAP-based Explanations are Sensitive to Feature Representation
Hyunseung Hwang, Andrew Bell, Joao Fonseca, Venetia Pliatsika, Julia Stoyanovich, Steven Euijong Whang · 2025
Read SHAP-based Explanations are Sensitive to Feature Representation by Hyunseung Hwang, Andrew Bell, Joao Fonseca, Venetia Pliatsika, Julia Stoyanovich, Steven Euijong Whang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08345.txt.
Introduction
Hyunseung Hwang and co-authors show that plain data preparation choices can bend model explanations. Their 2025 paper studies SHAP, a popular tool that scores how much each feature drives a prediction. Scores are computed against a feature representation that users often treat as neutral.
Simple moves like binning age into a histogram or recoding race shift the importance scores. The shifts are large enough to hide discrimination in a model output. Earlier attacks needed poisoned data or rigged models; this one uses standard, innocent-looking engineering. The authors call the study the first systematic proof of the effect. Their warning lands on auditors: an explanation is only as honest as the representation fed into it.
Catalog record
| CREATOR | Hyunseung Hwang, Andrew Bell, Joao Fonseca, Venetia Pliatsika, Julia Stoyanovich, Steven Euijong Whang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08345
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SHAP-based Explanations are Sensitive to Feature Representation Hyunseung Hwang, Andrew Bell, Joao Fonseca, Venetia Pliatsika, Julia Stoyanovich, Steven Euijong Whang Abstract . Local feature-based explanations are a key component of the XAI toolkit. These explanations compute feature importance values relative to an “interpretable” feature representation. In tabular data, feature values themselves are often considered interpretable. This paper examines the impact of data engineering choices on local feature-based explanations. We demonstrate that simple, common data engineering techniques, such as representing age with a histogram or encoding race in a specific way, can manipulate feature importance as determined by popular methods like SHAP. Notably, the sensitivity of explanations to feature representation can be exploited by adversaries to obscure issues like discrimination. While the intuition behind these results is straightforward, their systematic exploration has been lacking. Previous work has focused on adversarial attacks on feature-based explainers by biasing data or manipulating models. To the best of our knowledge, this is the first study demonstrating that explainers can be misled by standard, seemingly innocuous data engineering techniques. Conference: The 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece The 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25), June 23–26, 2025, Athens, Greece DOI: 10.1145/3715275.3732105 ISBN: 979-8-4007-1482-5/2025/06 CCS: Human-centered computing CCS: Computing methodologies Machine learning CCS: Information systems Data management systems CCS: Social and professional topics Socio-technical systems 1. Introduction Explainable AI (XAI) is becoming increasingly critical for justifying the behavior of AI systems implemented in high stakes domains like education, lending, public employment, and healthcare ( Zejnilovic et al., 2021 ; Alwarthan et al., 2022 ; Chaddad et al., 2023 ) . One of the key components of XAI is the use of local feature-based explanations , which quantify the importance of an observation’s features to an outcome (or some other quantity of interest). For example, these explanations are foundational for algorithmic recourse , where understanding why an individual is rejected for a loan by an AI-assisted system allows them to contest or reverse that unfavorable decision. Local feature-based explanations can also be used to surface unfairness in decision-making, for example, if a model is revealed to be making individual-level decisions on the basis of features like age, gender, or race, which may be illegal to use under the disparate treatment doctrine. 1 1 1 https://en.wikipedia.org/wiki/Disparate_treatment Further, these explanations are becoming an essential tool to fulfill legal and regulatory requirements, such as the European Union’s AI Act, and General Data Protection Regulation’s “right to explanation” ( Fresz et al., 2024 ) . The Shapley value framework ( Shapley et al., 1953 ) , originally developed for dividing revenue in cooperative games, is widely used to quantify local feature importance in predictive classification. It underpins prominent explanation methods like SHAP (Shapley Additive Explanations) ( Lundberg and Lee, 2017 ) and QII (Quantitative Input Influence) ( Datta et al., 2016 ) . The framework explains the classification outcome for an observation by assessing how changes to a feature’s value, individually or in combination with others, impact that outcome. This process simulates interventions, aligning with causal inference principles by isolating each feature’s influence while controlling for others. A high Shapley value for a protected feature like age suggests its significant influence on the classifier’s decision. However, Shapley-value-based explanations have limitations: they can mislead users (intentionally or unintentionally) ( Huang and Marques-Silva, 2024 ) and are vulnerable to adversarial attacks and manipulations ( Laberge et al., 2023 ) . In this paper, we focus on the key observation that local feature-based explanations, derived from trained models and post-processed data, are susceptible to manipulations through feature engineering, which occurs upstream from classification in the machine learning pipeline. We use SHAP ( Lundberg and Lee, 2017 ) , the most widely adopted implementation of the Shapley value framework, to show that local feature-based explanations are influenced by simple data engineering operations, such as transforming continuous values or encoding categorical values, which modify feature representations. For instance, bucketization—a common method of grouping values into ranges—can make a feature value appear less (or more) important in terms of SHAP. We now provide an intuition through the example below: Figure 1 . A hypothetical lending example: For a given classifier model, if we bucketize the age feature to generate a local explanation of the outcome with SHAP, then the importance of age for Ann decreases compared to using the raw value of the feature. Intuitively, this happens because age=30 is infrequent—and low—in this hypothetical dataset, while age<50 appears to be typical, both on its own and in combination with education. Example 1.1 (Motivating example). Consider a vendor — a financial institution that uses a binary classifier to approve loans (see Figure 1 ). Suppose that Ann applies for the loan and is incorrectly rejected (a false negative). The vendor would like to see if its model made this rejection decision based on Ann’s age, and decides to compute feature importance using SHAP as part of its analysis. Indeed, when SHAP is run over the raw feature values, age appears to have high importance, likely because an age of 30 is comparatively low in the vendor’s data. Worried about a potential lawsuit, the vendor attempts to generate a different explanation for the same classification outcome: they keep the classifier model fixed, but change the representation of age, “bucketizing” it into the ranges “below 50” and “50 and above”. The vendor is relieved to see that this simple manipulation substantially diminishes the importance of age when explaining Ann’s outcome. Figure 2 demonstrates this very scenario for an individual in the ACS Income dataset, with the SHAP plot on the top showing an explanation on raw feature values, and the plot on the bottom showing an explanation after age is bucketized. Observe that the importance of age drops from rank 1 (most important) in Figure 2(a) to rank 5 (somewhat important) in Figure 2(b) , a decrease of 5 positions in terms of importance. Note also that, because of the efficiency property of Shapley values ( Shapley et al., 1953 ) , a SHAP explanation can be used to reconstruct the outcome (by summing feature weights and returning the positive label if the sum is positive). In the example in Figure 2 , both explanations are consistent with the classifier’s prediction: they both predict that the individual would be rejected for the loan. If the vendor is worried about being challenged for using a protected feature like age to incorrectly reject applicants, it can look for an explanation that agrees with the prediction, but diminishes the importance of age. In this paper, we refer to this kind of a manipulation as a data engineering attack. (a) SHAP values before bucketization of age (b) SHAP values after age is bucketized Figure 2 . SHAP values of features before (a) and after (b) bucketization for a fixed individual from the ACS Income dataset. Note that the classifier model and SHAP explainer remain fixed; the only modification to the individual’s features from (a) \rightarrow (b) was the bucketization of age. In (a), age is encoded as a continuous feature and is deemed most important by SHAP, with a rank of 1 and a