The Most Important Features in Generalized Additive Models Might Be Groups of Features
Tomas Bosschieter, Luis França, Jessica Wolk, Yiyuan Wu, Bella Mehta, M.D, Joseph Dehoney, Orsolya Kiss, Fiona C. Baker, Qingyu Zhao, Rich Caruana, Kilian M. Pohl · 2025
Read The Most Important Features in Generalized Additive Models Might Be Groups of Features by Tomas Bosschieter, Luis França, Jessica Wolk, Yiyuan Wu, Bella Mehta, M.D, Joseph Dehoney, Orsolya Kiss, Fiona C. Baker, Qingyu Zhao, Rich Caruana, Kilian M. Pohl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19937.txt.
Introduction
Bosschieter and co-authors argue that in explainable models, the story often belongs to groups of features, not lone ones. Additive models score each feature's effect on its own, which can hide signals that only fire together, such as brain measures that travel in packs. Their method rates a whole group's weight at once, with no retraining, groups defined after the fact, overlaps allowed, and a meaning akin to explained variation.
Tests on synthetic data show the measure behaving sensibly across regimes. In a brain dataset the method surfaced drivers of depressive symptoms that single features missed, and in hip-surgery data it clarified how social factors jointly shape outcomes. Group weight, they conclude, gives the fuller picture doctors need.
Catalog record
| CREATOR | Tomas Bosschieter, Luis França, Jessica Wolk, Yiyuan Wu, Bella Mehta, M.D, Joseph Dehoney, Orsolya Kiss, Fiona C. Baker, Qingyu Zhao, Rich Caruana, Kilian M. Pohl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-19937
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Most Important Features in Generalized Additive Models Might Be Groups of Features Tomas Bosschieter, Luis França, Jessica Wolk, Yiyuan Wu, Bella Mehta, M.D, Joseph Dehoney, Orsolya Kiss, Fiona C. Baker, Qingyu Zhao, Rich Caruana, Kilian M. Pohl Abstract While analyzing the importance of features has become ubiquitous in interpretable machine learning, the joint signal from a group of related features is sometimes overlooked or inadvertently excluded. Neglecting the joint signal could bypass a critical insight: in many instances, the most significant predictors are not isolated features, but rather the combined effect of groups of features. This can be especially problematic for datasets that contain natural groupings of features, including multimodal datasets. This paper introduces a novel approach to determine the importance of a group of features for Generalized Additive Models (GAMs) that is efficient, requires no model retraining, allows defining groups posthoc, permits overlapping groups, and remains meaningful in high-dimensional settings. Moreover, this definition offers a parallel with explained variation in statistics. We showcase properties of our method on three synthetic experiments that illustrate the behavior of group importance across various data regimes. We then demonstrate the importance of groups of features in identifying depressive symptoms from a multimodal neuroscience dataset, and study the importance of social determinants of health after total hip arthroplasty. These two case studies reveal that analyzing group importance offers a more accurate, holistic view of the medical issues compared to a single-feature analysis. keywords Machine learning, interpretability, feature importance, healthcare, neuroscience 1 Introduction As machine learning models become ever-more ubiquitous in healthcare, there is a growing necessity for model interpretability. While it has proven difficult to reach a consensus on the meaning of interpretability [ 1 ] , one particularly common instrument is determining the importance of individual features (aka feature importance or feature attribution), which in and of itself also does not have one single definition. Common definitions of feature importance include the average contribution of a feature to prediction in a separable model [ 2 , 3 , 4 ] , average impurity decrease in tree-based models [ 5 , 6 ] , and the average increase in the model’s error or loss after shuffling the values of a feature (i.e., permutation feature importance) [ 7 , 8 ] . Providing an interpretation based on the importance of individual features is often challenging due to feature correlation and plausibility issues [ 9 ] . For a group of correlated features, their joint signal could be crucial for prediction, while the individual features in the group might be attributed low feature importance [ 10 ] . As a result, there could be signal in groups of correlated features that might be lost when analyzing individidual features by themselves. Even worse, these features could be filtered out during feature selection and thereby diminish the signal for inference [ 6 ] . This could be particularly troublesome on data containing natural groupings of features, such as modalities in multimodal data. Furthermore, interactions between features (in a group) are not captured through feature importance [ 11 ] . One way to address these issues is via group importance (aka grouped feature importance), i.e., by grouping features together and determining the importance of the group [ 12 , 13 ] . Group importance has previously been proposed using LASSO [ 14 ] , the Penalty Decomposition [ 15 ] , Shapley values [ 16 ] , and permutation testing [ 17 , 8 ] . Unfortunately, most of these methods come with computational challenges, mainly originating from the need to retrain the underlying machine learning model or compute expensive gradients. To address these challenges, we propose a robust, versatile, and computationally efficient method for computing group importance for Generalized Additive Models (GAMs), a very broad class of interpretable models that includes linear, logistic, and generalized linear models. GAMs with interactions [ 3 , 2 ] are of the form g(\mathbb{E}(y))=f_{1}(x_{1})+\cdots+f_{k}(x_{k})+\sum_{i\neq j}f_{i,j}(x_{i},x_{j}) , where g is a link function, y the target, x_{i} are features, \beta_{0} is an intercept, and f_{i}(x_{i}) and f_{i,j}(x_{i},x_{j}) are shape functions (aka component functions). In this setting, the (individual) importance I_{x_{j}} of a feature x_{j} is typically computed as the average absolute contribution of the corresponding shape function f_{j}(\cdot) [ 4 ] , i.e., I_{x_{j}}=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}|f_{j}(t_{j})|, (1) where t\in\mathcal{T} represents samples t in training data \mathcal{T} . For a group of features G=\{x_{i_{1}},\ldots,x_{i_{k}}\} , we extend this definition and propose I_{G}\coloneqq\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}|f_{i_{1}}(t_{i_{1}})+\cdots+f_{i_{k}}(t_{i_{k}})|, (2) with mathematical justification given in Section 2.2 . This definition of group importance has several properties we consider valuable, including: • groups can be defined posthoc (and thus do not affect the model) and may be overlapping, • computing group importance is computationally inexpensive due to the separability of the GAM, • it remains meaningful when there are a lot of (correlated) features, unlike, e.g., group permutations, • for a group consisting of one feature, its group importance equals the feature’s individual importance, • adding useless features, i.e. random noise, to a group does not change its group importance (up to small error \epsilon ), • one group could be all features in the dataset, yielding a relative scale on which to measure the importance of features and feature groups akin to percent of variance explained in statistics, i.e., percent of total importance ‘explained’ by a group of features. We implement our method with an Explainable Boosting Machine (EBM), a GAM-based model that achieves state-of-the-art accuracy while providing interpretability [ 2 , 4 ] . EBMs have recently gained popularity for their application in fields such as concrete compressive strength [ 18 ] , brain tumors [ 19 ] , Parkinson’s disease [ 20 ] , and revealing healthcare biases that result in significant loss of life [ 21 ] . We evaluate the properties of our definition of group importance on three synthetic experiments and two real-world datasets that contain natural groupings of features. In the first case study, we identify the depressive symptom of negative valence [ 22 , 8 ] on a neuroscience dataset consisting of measurements from functional brain networks and groups (aka domains) such as sleep, traumatic experiences, biographic data, and social support. In the second case study, we focus on the influence of community-level social determinants of health in predicting mortality for total hip arthroplasty using a large-scale healthcare dataset ( >100 k samples). These determinants, encompassing environmental, economic, and social conditions, are paramount in understanding health outcomes. Our main contributions are as follows: • Proposing a definition for group importance for Generalized Additive Models that is computationally inexpensive. • Showing that group importance offers enhanced interpretability in addition to individual feature importance, especially for correlated features, as demonstrated on two real-world datasets. • Demonstrating group importance can be used as a low-cost, yet effective method for feature selection, and defining a parallel between group importance and explained variance. 2 Methods We first describe Explainable Boosting Machines (EBMs) as a specific implementation of a Generalized Additive Model (GAM), and how feature importance is typically computed in this setting. Then, we extend this approach to define group importance. Subsequently, we turn to feature selection and briefly discuss the parallel between group importance and explained variation in statistics.