Marginal vs Conditional VIM
Mohammad Kaviul Anam Khan · 2025
Read Marginal vs Conditional VIM by Mohammad Kaviul Anam Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16988.txt.
Introduction
Two ways to rank what drives a model are set side by side by Mohammad Kaviul Anam Khan and co-authors in a 2025 arXiv paper. The head gives the longer title, on marginal and conditional importance from learning models. Black-box models hide which inputs sway them. A shuffle test on true means gives one rank, but tied inputs bend it. The team picks apart that bend by source.
A conditional rank, built to thin the bend, joins the shelf. Both ranks tie in a square way to a causal effect score. Users gain a fairer read of model drivers.
Catalog record
| CREATOR | Mohammad Kaviul Anam Khan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Marginal vs Conditional VIM Mohammad Kaviul Anam Khan Abstract Interpreting black-box machine learning models is challenging due to their strong dependence on data and inherently non-parametric nature. This paper reintroduces the concept of importance through “Marginal Variable Importance Metric” (MVIM), a model-agnostic measure of predictor importance based on the true conditional expectation function. MVIM evaluates predictors’ influence on continuous or discrete outcomes. A permutation-based estimation approach, inspired by Breiman, (2001) and Fisher et al., 2019a () , is proposed to estimate MVIM. MVIM estimator is biased when predictors are highly correlated, as black-box models struggle to extrapolate in low-probability regions. To address this, we investigated the bias-variance decomposition of MVIM to understand the source and pattern of the bias under high correlation. A Conditional Variable Importance Metric (CVIM), adapted from Strobl et al., (2008) , is introduced to reduce this bias. Both MVIM and CVIM exhibit a quadratic relationship with the conditional average treatment effect (CATE). Marginal and Conditional Importance Measures from Machine Learning Models and Their Relationship with Conditional Average Treatment Effect Mohammad Kaviul Anam Khan 1 1 1 The Hospital for Sick Children. Email: [email protected] , Olli Saarela 2 2 2 Dalla Lana School of Public Health University ot Toronto. Email: [email protected] and Rafal Kustra 2 2 2 Dalla Lana School of Public Health University ot Toronto. Email: [email protected] January 26, 2025 1 Introduction Using modern machine learning techniques, health outcomes for patients can be accurately predicted or classified ( Howell et al.,, 2012 ; Hawken et al.,, 2010 ; Fisher et al., 2019b, ; Cruz and Wishart,, 2006 ; Molnar,, 2020 ) . For example, an enhanced number of patient characteristics can be used to successfully predict the survival time of cancer patients using artificial neural networks (ANN) ( Liestbl et al.,, 1994 ; Ripley and Ripley,, 2001 ) . Studies have also used black-box machine learning models to analyze genetic data ( Libbrecht and Noble,, 2015 ) . However, their applicability in the various scientific fields can be hampered due to the lack of interpretability of the predictors relationship with the outcome from the models. Often, in clinical or health science research, parametric statistical models are preferred over black-box models because they are easily interpretable. Although some work has been published to interpret outputs from machine learning methods, they are mostly data driven and do not have any statistical or causal interpretation, which limits their use in many scientific domains, e.g., in clinical and public health. In the last few decades, some developments have been made to explain the output of black-box machine learning models ( Molnar,, 2020 ) . The popular methods include the Variable Importance Metric (VIM; Breiman,, 2001 ), LIME (Local Interpretable Model-agnostic Explanations; Ribeiro et al.,, 2016 ), SHAP (SHapley Additive exPlanations; Lundberg and Lee,, 2017 ) and leave one covariate out (LOCO) based estimators ( Rinaldo et al.,, 2019 ; Verdinelli and Wasserman,, 2024 ) . For example, in a recent study ( Weller et al.,, 2021 ) the authors were able to find risk factors for suicidal thoughts and behavior using multiple machine learning models using SHAP. However, it is not yet clear whether these methods can quantify a causal relationship between predictors and outcomes. Breiman, (2001) proposed the VIM for random forests to identify important variables for prediction. The VIM provides the ranking of the predictors based on changes in the prediction error. The VIM approach is widely applicable and can be used effectively for conventional linear models, generalized linear models, and generalized additive models ( Gregorutti et al.,, 2017 ; Fisher et al., 2019a, ; Hooker et al.,, 2021 ) . Hooker, (2007) showed the VIM equates to the Sobol indices ( Fel et al.,, 2021 ) when all predictors are independent. Numerous studies, however, have found that the permutation-based importance of random forests can produce highly misleading diagnostics, particularly when there is a strong dependence between features / predictors ( Strobl et al.,, 2007 ; Strobl et al.,, 2008 ; Hooker et al.,, 2021 ) . Using simulation studies, Hooker et al., (2021) advocated the use of alternative measures such as conditional permutation-based VIM ( Strobl et al.,, 2008 ; Debeer and Strobl,, 2020 ) to interpret the importance of variables, instead. ( Chamma et al.,, 2024 ) showed theoretically and empirically that conditional permutation importance (CPI) overcomes the limitations of standard permutation importance by providing accurate type I error control. Kaneko, (2022) proposed a technique based on cross-validation to calculate VIM that minimizes the effect of predictor correlations using simulations. However, most of these simulation scenarios were built on assuming simple functional relationships between outcomes and predictors. It is unclear, though, how VIM would perform in more complex scenarios, such as nonlinear and nonadditive functional relationships between predictors and outcome in presence of high correlation between the predictors. Furthermore, the question still remains about the use of such an importance measure and whether it is possible to use this measure to discern the causal relationship (if any) between the predictor and the outcome or if the VIM can be represented as a statistical parameter. Diaz et al., (2015) argued that identifying predictor importance and prediction of an outcome have different goals, and the VIM proposed by Breiman, (2001) does not have any clinical or causal relevance. Van der Laan and Rose, (2011) created a new definition of variable importance based on a targeted causal parameter of interest, which has been used very effectively for many real-life applications. Fisher et al., 2019a () , defined Model Reliance (MR) based on the idea of VIM. Gregorutti et al., (2017) ; Hooker et al., (2021) have also shown similar representations of such a statistical parameter. Fisher et al., 2019a () further showed such a metric can be represented as a statistical parameter and also have a causal interpretation. They further proposed techniques for estimating MR and their probabilistic bounds. Fisher et al., 2019a () ’s MR proposal was based on some pre-defined models or a model class. It is unclear if this MR or VIM can be represented as a statistical parameter in cases where the true relationship between an outcome and the predictors is completely unknown. One objective of this study is to relate permutation-based and conditional permutation-based importance measures to causal parameters. This work extends existing variable importance methods, while examining their statistical properties across various simulated scenarios, and focuses on complex predictors-outcome relationship and modern black-box model exemplified by popular extension of Gradient Boosting Machine (Friedman cite) implemented in XGboost [cite]. Additionally, it addresses the bias introduced by correlated predictors in permutation-based measures and demonstrates how conditional permutation-based measures can mitigate this bias. The study also explores the relationship between permutation-based importance measures and leave-one-covariate-out (LOCO) estimators. Finally, it illustrates how, in simple scenarios, marginal permutation-based importance can be derived from conditional permutation-based importance measures. The rest of the paper is structured as follows: in Section 2 we provide the definition of Marginal Variable Importance Metric (MVIM) and its relation to causal parameter. In Section 3 we showed the bias-variance decomposition of MVIM along with some simulation results. In Section 4 we defined the Conditional Variable Importance Metric (CVIM) and showed its relationship with causal parameter.