Variable Importance in Generalized Linear Models – A Unifying View Using Shapley Values

Sinan Acemoglu, Christian Kleiber, Jörg Urban · 2026

Read Variable Importance in Generalized Linear Models – A Unifying View Using Shapley Values by Sinan Acemoglu, Christian Kleiber, Jörg Urban (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00773.txt.

Introduction

Sinan Acemoglu, Christian Kleiber, and Jörg Urban ask a question every regression user faces: which variables matter most? For ordinary linear models, Shapley values, borrowed from game theory, neatly split the familiar R-squared among the predictors. Beyond linear models there is no accepted fit measure, only a crowd of pseudo R-squared values with murky meanings.

The authors lay out the properties a fit measure needs so that Shapley decompositions can be read as relative, and even absolute, variable importance. They propose a pseudo R-squared built on Kullback-Leibler divergence, which takes a convenient form for generalized linear models. That choice unifies and extends earlier proposals, giving applied researchers in many fields a principled importance measure.

Catalog record

CREATORSinan Acemoglu, Christian Kleiber, Jörg Urban
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Variable Importance in Generalized Linear Models – A Unifying View Using Shapley Values Sinan Acemoglu, Christian Kleiber, Jörg Urban Abstract Variable importance in regression analyses is of considerable interest in a variety of fields. There is no unique method for assessing variable importance. However, a substantial share of the available literature employs Shapley values, either explicitly or implicitly, to decompose a suitable goodness-of-fit measure, in the linear regression model typically the classical R 2 R^{2} . Beyond linear regression, there is no generally accepted goodness-of-fit measure, only a variety of pseudo- R 2 R^{2} s. We formulate and discuss the desirable properties of goodness-of-fit measures that enable Shapley values to be interpreted in terms of relative, and even absolute, importance. We suggest to use a pseudo- R 2 R^{2} based on the Kullback-Leibler divergence, the Kullback-Leibler R 2 R^{2} , which has a convenient form for generalized linear models and permits to unify and extend previous work on variable importance for linear and nonlinear models. Several examples are presented, using data from public health and insurance. Keywords. Averaging over orderings; Dominance analysis; Goodness of fit; Hierarchical partitioning; Kullback-Leibler divergence; Regression 1 Introduction Assessing the importance of explanatory variables in regression analyses is of considerable interest in a number of areas. There is a large literature spanning several decades, highly fragmented and characterized by parallel developments and rediscoveries in numerous fields, among them various behavioral and social sciences, the medical sciences, ecology, and business administration. Excellent surveys are available from 1 , 18 , and 15 ; 16 . For the linear regression model, a prominent method, apparently originating from 26 , determines variable importance via a representation of the classical R^{2} as a weighted average involving all possible subsets of regressors in the model. At the time it did not attract much attention, presumably because of the substantial computational burden. (Note that a model with 10 regressors – a model of rather modest size by current standards – already involves 2^{10}=1024 regressions.) The beauty of the proposal in 26 is that it is easy to interpret and that it produces a ‘fair’ decomposition of the value of an overall goodness-of-fit measure, R^{2} , into the contributions of the individual regressors. The method has been rediscovered and extended several times using different terminologies, among them averaging over orderings ( 21 ; 22 , a term that we will use below, see) , hierarchical partitioning ( 9 ) and dominance analysis ( 4 ; 2 ) . More specifically, hierarchical partitioning and dominance analysis extend the idea of 26 by permitting fairly arbitrary goodness-of-fit measures for evaluating the contributions of predictors. They also go beyond linear regression. It was pointed out by 39 that hierarchical partitioning, and hence averaging over orderings, amounts to using the Shapley value, a concept from cooperative game theory, to decompose a measure of fit of the regression. An early paper explicitly using the Shapley value framework is 27 , still confined to linear regression. Advances in statistical computing and increasing computational power have made Shapley value computations more feasible in recent years, and the availability of flexible software for nonlinear models has motivated the search for measures of variable importance beyond linear regression. Indeed, 9 already suggested to use hierarchical partitioning to measure the contributions of predictors in classical nonlinear models via a decomposition of \chi^{2} statistics. Later, 3 applied dominance analysis in logistic regression, but now with certain pseudo- R^{2} measures. To the best of our knowledge, empirical examples for nonlinear models are still largely confined to binary response models, although some software implementations offer functionality beyond this setting, notably Poisson regression. However, there are problems with certain proposals and corresponding implementations, as we shall discuss below. In this paper, we pursue three goals: (i) the formulation of desirable properties of goodness-of-fit measures that allow an interpretation of Shapley values in terms of relative and even absolute importance, (ii) the systematic use of Shapley values for a reasonably large class of models, here GLMs and certain extensions thereof, and (iii) the consistent use of Shapley values with goodness-of-fit measures that generalize classical variants while at the same time opening the methodology to further settings. We show that the goodness-of-fit measures that are used to assess the contributions of predictors need to be chosen carefully, as their properties are critical to the interpretation of the resulting Shapley values. We therefore formulate desirable properties of goodness-of-fit measures that help to interpret the resulting Shapley values as measures of relative and even of absolute importance. Currently, there does not appear to be a generally accepted procedure for assessing variable importance in generalized linear models (GLMs). Therefore, we propose to use Shapley values along with a fit measure that meets certain requirements and exists for all GLMs, namely the Kullback-Leibler R^{2} , hereafter denoted as R^{2}_{\text{KL}} . It was introduced by 8 and is closely related to the deviance and the classical likelihood ratio statistic. Specifically, for the linear regression model R^{2}_{\text{KL}} reduces to the classical R^{2} , while for binary response models it reduces to McFadden’s likelihood ratio index, perhaps the most prominent of all pseudo- R^{2} measures in this setting. Thus, our proposal extends Shapley value decompositions from linear regression models to the much larger class of GLMs (and even some related models), thereby providing a unified approach to variable importance for a range of nonlinear regression models. We present several empirical examples to illustrate the methodology, using data from public health and insurance. The remainder of this paper is organized as follows: In Section 2 we summarize the necessary background on the Shapley value. In Section 3 we discuss desirable properties of a goodness-of-fit measure for which Shapley values are computed. We establish that R^{2}_{\text{KL}} possesses these properties in Section 4 . Section 5 provides examples involving Poisson and geometric regression as well as the Poisson hurdle model, a model with two linear predictors but with GLM building blocks. These examples also illustrate the tradeoffs in the choice of the fit measure. Section 6 concludes. 2 Shapley values in regression The Shapley value is a solution concept from cooperative game theory, introduced by 36 . A convenient reference for the game-theoretical terminology and background is 13 . A cooperative game (P,v) in coalitional form is described by a finite set of players, P=\{1,\dots,p\} , and a characteristic function v:2^{P}\to\mathbb{R} that assigns a real number v(S) to each element S of the power set 2^{P} . S\subseteq P is called a coalition, P the grand coalition, and v(S) can be interpreted as the payoff that coalition S can secure when its members act as a unit. In cooperative game theory, a standard condition for the characteristic function v is v(\varnothing)=0 ; i.e., the empty set or coalition \varnothing secures a payoff of zero. We may refer to this as ‘zero-normalization’. The Shapley value \varphi_{i}(P,v) for player i\in P and a given characteristic function v is \varphi_{i}(P,v)\penalty\ =\penalty\ \sum_{S\subseteq P\setminus\{i\}}\underbrace{\frac{|S|!\penalty\ (p-|S|-1)!}{p!}}_{\rm{weight}}\underbrace{\left(v(S\cup\{i\})-v(S)\right)}_{\text{\parbox{113.81102pt}{\centering marginal contribution of\\ player $i$ to coalition $S$\@add@centering}}}, (1) where |S| represents the cardinality of the set S . Hence \varphi_{i}(P,v) is the average marginal contribution of player i ; the average is formed over all subsets S

Read the complete text