Locally Explaining Prediction Behavior via Gradual Interventions and Measuring Property Gradients
Niklas Penzel, Joachim Denzler · 2025
Read Locally Explaining Prediction Behavior via Gradual Interventions and Measuring Property Gradients by Niklas Penzel, Joachim Denzler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05424.txt.
Introduction
Niklas Penzel and Joachim Denzler present a 2025 arXiv paper on explaining single predictions by gently changing the input. Most explanation tools report associations, not causes. Their framework uses modern image editing models to slide one trait at a time, blush on a cheek, say, or age in a face, and measures how the prediction glides in response. A new score captures the strength of that response.
Tests first proved the method finds planted biases in a synthetic setup. Real studies followed: skin lesion classifiers, the inside story of network training, and a large vision-language model probed with real edits. The authors argue property-level interventions reveal behavior that heat maps miss.
Catalog record
| CREATOR | Niklas Penzel, Joachim Denzler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05424
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Locally Explaining Prediction Behavior via Gradual Interventions and Measuring Property Gradients Niklas Penzel, Joachim Denzler Abstract Deep learning models achieve high predictive performance but lack intrinsic interpretability, hindering our understanding of the learned prediction behavior. Existing local explainability methods focus on associations, neglecting the causal drivers of model predictions. Other approaches adopt a causal perspective but primarily provide global, model-level explanations. However, for specific inputs, it’s unclear whether globally identified factors apply locally. To address this limitation, we introduce a novel framework for local interventional explanations by leveraging recent advances in image-to-image editing models. Our approach performs gradual interventions on semantic properties to quantify the corresponding impact on a model’s predictions using a novel score, the expected property gradient magnitude. We demonstrate the effectiveness of our approach through an extensive empirical evaluation on a wide range of architectures and tasks. First, we validate it in a synthetic scenario and demonstrate its ability to locally identify biases. Afterward, we apply our approach to investigate medical skin lesion classifiers, analyze network training dynamics, and study a pre-trained CLIP model with real-life interventional data. Our results highlight the potential of interventional explanations on the property level to reveal new insights into the behavior of deep models. 1 1 1 Project page: https://propgrad.github.io/ 1 Introduction (a) Fur color intervention on a light-furred dog. (b) Background intervention for a light-furred dog. (c) Changes in model behavior for fur color intervention. (d) Changes in model behavior for background intervention. Figure 1 : Fig. 1a and Fig. 1b demonstrate our approach to study model predictions through gradual interventions here for two properties: fur color and background illumination in a cat vs. dog classification task. Using [ 16 ] , we synthetically adjust these properties, i.e., darkening the fur or the background, while tracking prediction changes. Specifically, Fig. 1c and Fig. 1d show the resulting shifts in model outputs for three networks: an unbiased one, one trained only on dark-furred cats and light-furred dogs, and one trained on the opposite bias. The red line marks the decision threshold, where predictions flip between classes during the intervention. Modern deep learning models are complex data-centric systems that achieve high predictive performance but lack intrinsic interpretability. Hence, many explainability (XAI) methods were proposed to interpret trained model behavior, especially for vision models. Post-hoc XAI includes local methods, often generating pixel-wise attributions, e.g., [ 72 , 44 , 62 , 77 , 75 ] and global methods focused on human interpretable concepts or properties [ 35 , 82 , 60 , 11 , 68 , 9 , 18 , 45 , 33 ] . Unfortunately, global insights can be deceiving for individual inputs. On the local level, properties can be occluded or overshadowed by independent visual elements. To address this limitation, we introduce a novel approach to locally explain neural network prediction behavior based on input interventions at the property level. We propose to leverage recent breakthroughs in image-to-image editing models, e.g., [ 5 , 46 , 16 ] . These models are trained conditionally using Classifier-Free Guidance (CFG) scaling [ 28 ] . Hence, we can gradually control the alignment with the corresponding interventional instruction during inference. Our key insight is that this paradigm facilitates the smooth transition between two property states (see Fig. 1 , top row), also for complex features. Consequently, by utilizing CFG scaling, we intervene on semantic properties for an individual input and study the shift in prediction behavior of trained neural networks (see Fig. 1 , bottom row). To quantify the impact of a property on a model, we propose approximating the corresponding expected property gradient magnitude. This score is naturally connected to the intuition of measuring the change along the property axis in Fig. 1 . Additionally, our expected property gradient magnitude can be seen as an extension of the causal concept effect [ 20 ] for gradual interventions. Finally, to ensure robustness, we suggest a corresponding permutation significance test. We empirically validate our framework for local interventional explanations on a wide variety of architectures and tasks. First, we explore a biased scenario (cats vs. dogs [ 10 ] ) where we synthetically correlate a property (fur color) with the label to validate that our approach can locally identify the causal factor (see Fig. 1 ). Additionally, our gradual interventions allow for a direct quantification of how the outputs change in response to shifts in the selected property, a key distinction from local methods highlighting image regions, e.g., [ 72 , 44 , 62 , 77 , 75 , 14 , 21 , 56 , 34 ] . Further, our derived score quantitatively outperforms baselines for indicating locally biased behavior during interventions. We corroborate these findings for real-world skin lesion classification, analyzing a known bias. Afterward, we study the training dynamics of eight modern classifiers by tracking a property correlated with the label. Here we find oscillations also in later epochs, depending on the weight initialization. Finally, we demonstrate that our model-agnostic approach works with diverse sources of interventions by capturing real-life interventional data to analyze a CLIP [ 58 ] model. Our key contributions can be summarized as follows: (1) We introduce a new framework for local network prediction explanations based on gradual interventions, e.g., by leveraging Classifier-Free Guidance (CFG) scaling [ 28 ] . (2) We derive a novel score to quantify the shift in model behavior with respect to a property by approximating the expected property gradient magnitude. (3) We provide a corresponding hypothesis test to verify statistical significance. (4) We conduct experiments on a wide range of architectures and tasks to demonstrate the effectiveness of our approach. 2 Related Work Many methods to derive local explanations aim to find important regions in the input, e.g., [ 83 , 72 , 44 , 62 , 77 , 75 ] . A subset of these methods, most closely related to our approach, employs input perturbations to estimate importance. These perturbations or interventions are often done by replacing patches and, therefore, occluding parts of the input. Such occlusion patches can be constructed using noise, e.g., [ 83 , 54 ] . Other approaches use similar image regions [ 84 ] or generative infilling [ 8 , 37 , 34 ] . Similarly related are methods that use causal terminology and generate visual counterfactual explanations [ 76 ] by posing questions of the form “Which parts of the input need to change to result in a specific prediction?” Examples of this approach include [ 14 , 21 , 56 , 34 , 3 ] . Nevertheless, such visual explanations necessitate additional semantic interpretations by experts to identify specific properties responsible for the measured importance. In contrast, our approach explains the prediction behavior directly on the level of semantic properties. Related in that regard are, often global, explanation methods, e.g., [ 38 , 25 , 35 , 82 , 60 , 11 , 68 , 9 , 18 , 45 , 69 , 33 ] , which discover abstract concepts learned by a trained model. However, these methods require direct access to the model parameters or probing datasets. Further, while they are often explorative, they have difficulty determining whether a certain property is unused and can