FINCH: Locally Visualizing Higher-Order Feature Interactions in Black Box Models

\authororcid Anna Kleinau0000-0002-3415-6316, Bernhard Preim, and Monique Meuschke · 2025

Read FINCH: Locally Visualizing Higher-Order Feature Interactions in Black Box Models by \authororcid Anna Kleinau0000-0002-3415-6316, Bernhard Preim, and Monique Meuschke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16445.txt.

Introduction

Anna Kleinau and co-authors build a visual tool that shows how features team up inside black-box models in this 2025 paper. Most explainers score single features, missing the interplay that drives predictions. Their tool, FINCH, works case by case and uses color and highlighting to expose feature subsets that matter together. Extra views help users judge how far to trust the model and its explanation.

Case studies show the tool travels across domains. A study with machine learning experts rated it helpful and usable. The authors argue that seeing higher-order interactions, not just lone features, is key to honest explanations of complex models.

Catalog record

CREATOR\authororcid Anna Kleinau0000-0002-3415-6316, Bernhard Preim, and Monique Meuschke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16445

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FINCH: Locally Visualizing Higher-Order Feature Interactions in Black Box Models \authororcid Anna Kleinau0000-0002-3415-6316, Bernhard Preim, and Monique Meuschke Abstract In an era where black-box AI models are integral to decision-making across industries, robust methods for explaining these models are more critical than ever. While these models leverage complex feature interplay for accurate predictions, most explanation methods only assign relevance to individual features. There is a research gap in methods that effectively illustrate interactions between features, especially in visualizing higher-order interactions involving multiple features, which challenge conventional representation methods. To address this challenge in local explanations focused on individual instances, we employ a visual, subset-based approach to reveal relevant feature interactions. Our visual analytics tool Finch uses coloring and highlighting techniques to create intuitive, human-centered visualizations, and provides additional views that enable users to calibrate their trust in the model and explanations. We demonstrate Finch in multiple case studies, demonstrating its generalizability, and conducted an extensive human study with machine learning experts to highlight its helpfulness and usability. With this approach, Finch allows users to visualize feature interactions involving any number of features locally. \onlineid 0 \vgtccategory Research \vgtcpapertype please specify \authorfooter all authors are with Otto-von-Guericke University Magdeburg, Germany. \teaser Our visual analytics tool Finch generates dependency plots that can locally visualize feature interactions up to high orders. Introduction keywords Explaineble AI, Feature Interactions, HCAI. As AI grows due to the increasing availability of data, high-dimensional datasets with numerous instances are becoming the norm. Explainable AI (xAI) is evolving alongside this trend; however, traditional xAI methods are reaching their limits as modern machine learning models increasingly rely on interactions among countless features. Explaining features independently or focusing solely on one or two-feature interactions is insufficient. New methods are needed to make higher-order feature interactions —complex interactions among multiple features— understandable for humans, as these often contain the true predictive power of AI models. Our focus is on local explanations that clarify the prediction for individual instances. For example, in a decision support system, which feature interactions drive the current recommendation? This need becomes evident when considering the following example by Tsang et al. [ 30 ] , who describes the prediction function f(x)=x+y+xy . If relevance is assigned only to individual features, how do we attribute the interaction between x and y ? Similar issues arise in practice; Wang et al. [ 31 ] found that domain experts prefer to see feature interactions, such as the relationship between systolic and diastolic blood pressure. Moreover, from a security perspective, Xin et al. [ 32 ] show that partial dependence plots focused on a single feature can be manipulated to conceal discriminatory behavior. These examples underscore the growing need for more effective methods to characterize feature interactions. Our visual analytics tool FINCH was designed to tackle this challenge for high-dimensional, tabular data. Its intuitive user interface allows machine learning experts to easily trace how multiple features interact and see how each additional feature influences the outcome. This is achieved through a subset-based approach using visualizations similar to the well-known partial dependence plots [ 14 ] . Instead of visualizing all possible interactions in a feature space, our approach focuses on the local interactions relevant to the current data instance. In summary, we make the following contributions: • We introduce a novel approach for visualizing higher-order feature interactions at the instance level. • We developed an interactive online tool tailored for machine learning experts. • We conduct a comprehensive evaluation of our tool through both case studies and a user study. 1 Background xAI methods are classically divided into local and global explanation methods [ 7 ] . Global methods aim to explain the entire model at once by describing the overall relevance of input features for predictions. Feature interactions are visualized to show how one or two features interact with the prediction. In contrast, local methods focus on explaining the model’s behavior for a specific instance, highlighting the relevant features for that instance’s prediction and visualizing their influence as negative or positive. This section reviews two of the most frequently used methods. 1.1 PDPs - Partial Dependence Plots Partial Dependence Plots (PDPs) [ 14 ] are a global explanation method that visualizes the influence of one or two features on a prediction, typically using a line chart or heatmap. The x-axis represents the values of a feature, while the y-axis shows the mean prediction. PDPs work well with smaller datasets because they use a mutation-based approach to generate sufficient data points. To calculate the average prediction for a feature value, all available data instances are set to this value, while keeping their other features at their original value. A visualization similar to PDPs is the ICE plot. While calculated similarly, each instance is given its own curve instead of averaging all instances to produce a single curve. This results in a more detailed but also more cluttered visualization. 1.2 SHAP plots SHAP plots [ 24 ] are a local explanation method based on game theory that visualizes the positive and negative influences of features on the current instance. Each feature is represented by a bar, where the length depicts the strength of its influence, and the direction and color indicate whether the influence is positive or negative. This visualization shows the influence of each feature individually without displaying their interactions, which can be confusing when variables are highly correlated. In such cases, their combined influence might be assigned to one feature or split between them, leading to potentially misleading variations in the final SHAP plot. Visualizations similar to SHAP, but with different calculation methods, include LIME plots, which are created by learning a model around the local instance [ 28 ] . 2 Related Work In this section, we introduce the emerging concept of regional explanations that inspired our tool, and discuss prior work on interpreting feature interactions and visual analytics in xAI. 2.1 Regional methods Set between local and global methods, an emerging category of explanation methods are regional methods, which focus on specific subgroups of the data. Repid [ 16 ] analyzes the ICE curves of a feature by identifying subgroups with different behaviors based on the shapes of individual instances’ curves. A decision tree divides these groups while maintaining interpretable labels for easier understanding. While Repid requires selecting an initial feature for exploring interactions, Gadget [ 17 ] offers a more general approach by recursively decomposing the entire feature space to identify simple subspaces with minimal interactions, allowing for straightforward models in explanations. Like Repid , Gadget uses decision trees to guide the decomposition process. Unlike Finch , Repid and Gadget do not identify feature interactions relevant to a specific instance; instead, they search for feature interactions globally by identifying subgroups. 2.2 Feature Interaction Interpretation Few approaches exist for interpreting or visualizing higher-order feature interactions involving more than two features. Tsang et al. [ 30 ] provide a comprehensive review of feature interactions, emphasizing the need for improved interactive visualizations, which Finch aims to address. Zhang et al. [ 33 ] take a mathematical approach, assuming product separability of feature interactions. Their method identifies these interactions and determines the most likely mathematical formula to represent them. Unlike Finch ,

Read the complete text