iLOCO: Distribution-Free Inference for Feature Interactions
Camille Olivia Little, Lili Zheng, Genevera I. Allen · 2025
Read iLOCO: Distribution-Free Inference for Feature Interactions by Camille Olivia Little, Lili Zheng, Genevera I. Allen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-06661.txt.
Introduction
Camille Little and co-authors build a score for how pairs of traits join to drive a model's calls, with sound doubt bands. The work dates to 2025. Trait scores help read what a model leans on. Most miss how traits act in pairs or groups, though.
Their score drops one trait of a pair at a time and reads the shift in loss. Fresh gains in leave-one-out math give doubt bands that need no spread guess. A team-learn trick makes the score and bands cheap to run. Tests on made and real sets beat past tools. The work gives the first doubt-backed read of trait pairs.
Catalog record
| CREATOR | Camille Olivia Little, Lili Zheng, Genevera I. Allen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-06661
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
iLOCO: Distribution-Free Inference for Feature Interactions Camille Olivia Little, Lili Zheng, Genevera I. Allen Abstract Feature importance measures are widely studied and are essential for understanding model behavior, guiding feature selection, and enhancing interpretability. However, many machine learning fitted models involve complex interactions between features. Existing feature importance metrics fail to capture these pairwise or higher-order effects, while existing interaction metrics often suffer from limited applicability or excessive computation; no methods exist to conduct statistical inference for feature interactions. To bridge this gap, we first propose a new model-agnostic metric, interaction Leave-One-Covariate-Out (iLOCO), for measuring the importance of pairwise feature interactions, with extensions to higher-order interactions. Next, we leverage recent advances in LOCO inference to develop distribution-free and assumption-light confidence intervals for our iLOCO metric. To address computational challenges, we also introduce an ensemble learning method for calculating the iLOCO metric and confidence intervals that we show is both computationally and statistically efficient. We validate our iLOCO metric and our confidence intervals on both synthetic and real data sets, showing that our approach outperforms existing methods and provides the first inferential approach to detecting feature interactions. Keywords: Interactions, feature importance, interpretability, minipatch ensembles, LOCO inference, distribution-free inference 1 Introduction Machine learning systems are increasingly deployed in critical applications, necessitating human-understandable insights. Consequently, interpretable machine learning has become a rapidly expanding field [ 32 , 12 ] . For predictive tasks, one of the most important aspects of interpretation is assessing feature importance. While much focus has been placed on quantifying the effects of individual features, the real power of machine learning lies in its ability to model higher-order interactions and complex feature representations [ 16 ] . However, there is a notable gap in interpretability: few methods provide insights into the higher-order feature interactions that truly drive model performance. Understanding how features interact to influence predictions is a critical component of interpretable machine learning, particularly in scientific applications or to make downstream business decisions. In drug discovery, for example, uncovering how chemical properties interact can guide the identification of effective compounds in large-scale screening processes [ 38 ] . In material science, feature interactions reveal relationships that enable the development of innovative materials with desired properties [ 3 , 30 ] . In business, analyzing interactions between customer demographics and purchasing behaviors can inform personalized marketing strategies and enhance customer retention [ 29 ] . In genomics, interactions between genes or “epistasis”, is believed to play a major role in driving complex diseases such as asthma, diabetes, multiple sclerosis, and various cardiovascular diseases [ 9 , 33 , 31 , 27 , 17 , 56 , 41 , 50 ] . These application areas underscore the need for machine learning tools that can accurately detect and quantify such feature interactions, providing interpretable and actionable insights for potentially high-dimensional data. In addition to detecting feature interactions, it is crucial to quantify the uncertainty associated with these interactions to ensure they are not simply noise. Uncertainty quantification has emerged as a focus in machine learning, addressing the need for more reliable and interpretable models [ 25 , 44 , 4 , 37 , 24 , 7 ] . While much of this work has traditionally centered on quantifying uncertainty in predictions [ 25 , 4 ] , there has been a growing interest in extending these techniques to feature importance [ 62 , 57 , 58 , 10 , 53 ] . To the best of our knowledge, uncertainty quantification has not yet been developed for feature interactions. Motivated by these challenges, we introduce the interaction Leave-One-Covariate-Out (iLOCO) metric and inference framework, which addresses key limitations of existing methods. Our framework provides a model-agnostic, distribution-free, statistically and computationally efficient solution for quantifying feature interactions and their uncertainties across diverse applications. 1.1 Related Works While feature importance has been extensively studied [ 39 , 2 , 28 , 32 ] , work on feature interactions remains relatively limited. Shapley-based approaches, such as FaithSHAP, extend the game-theoretic framework to quantify pairwise interactions by decomposing predictions into main and interaction effects [ 46 , 43 , 35 ] . The H-statistic, based on partial dependence, measures the proportion of output variance attributable to interactions [ 13 ] . Although both of these methods are model-agnostic methods and theoretically grounded, their computational cost increases exponentially with the number of features and observations, limiting their scalability. Recent work has also begun exploring interactions in large language models (LLMs), using attribution techniques to understand how combinations of input tokens influence prediction [ 22 ] . Model-specific alternatives have been developed for certain model classes, such as Iterative Forests for random forests [ 5 ] , Integrated Hessians for neural networks [ 21 ] , and linear regression-based methods [ 26 , 8 , 34 ] . Several local interaction detection/attribution methods have also been proposed [ 47 , 48 ] , focusing on explanations for individual instances instead of global interpretation. Further, no prior work provides uncertainty quantification for model-agnostic feature interaction. These limitations motivate the development of scalable metrics that can be applied across a wide range of models, associated with rigorous statistical inference procedures. There is a growing body of research on uncertainty quantification for model-agnostic feature importance in machine learning [ 25 , 58 , 53 , 52 , 1 , 15 , 40 , 51 ] ; however, this work does not extend to feature interactions. Our goal is to develop confidence intervals for our feature interaction metric that reflect its statistical significance and uncertainty. Our method builds upon the Leave-One-Covariate-Out (LOCO) metric and inference framework originally proposed by [ 25 ] and later studied by [ 62 , 60 ] . Since computational cost is a major challenge for interaction methods, we improve efficiency by adopting a fast inference strategy based on minipatch [ 60 , 55 , 45 ] ensembles, which simultaneously subsample both observations and features. 1.2 Contributions Our work focuses on two main contributions. First, we introduce iLOCO, a distribution-free, model-agnostic method for quantifying feature interactions that can be applied to any predictive model without relying on assumptions about the underlying data distribution. Moreover, we introduce an efficient way of estimating this metric via minipatch ensembles. Second, we develop rigorous distribution-free inference procedures for interaction effects, enabling confidence intervals for the interactions detected. Collectively, our contributions offer notable advancements in quantifying feature interaction importance and even higher-order interactions in interpretable machine learning. 2 iLOCO 2.1 Review: LOCO Metric To begin, let us first review the widely used LOCO metric that is used for quantifying individual feature importance in predictive models [ 25 ] . For a feature j , LOCO measures its importance by evaluating the change in prediction error when the feature is excluded. Formally, given a prediction function f:\mathcal{X}\to\mathbb{R} and an error measure \text{Err}() , the LOCO importance of feature j is: \displaystyle\Delta_{j}(\mathbf{X},\mathbf{Y})=\mathbb{E}\left[\mathrm{Error}(Y,f^{-j}(X^{-j};\mathbf{X}^{-j},\mathbf{Y}))-\mathrm{Error}(Y,f(X;\mathbf{X},\mathbf{Y}))\,\middle|\,\mathbf{X},\mathbf{Y}\right] (1) where f^{-j} is the prediction function trained without feature j . A large positive \Delta_{j} value suggests the importance of the feature by indicating performance degradation when the feature is excluded. While LOCO is effective at measuring the importance of individual features, it fails to capture feature interactions. To address this challenge, we propose