Interpretable Kernels

Patrick J.F. Groenen, Michael Greenacre · 2025

Read Interpretable Kernels by Patrick J.F. Groenen, Michael Greenacre (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15932.txt.

Introduction

Kernel methods let learning models bend their fits, but they hide what the model does with the raw traits. Patrick Groenen and Michael Greenacre reopen that black box in 2025. Their case is the wide table, where traits outnumber the rows of data.

In that case, they show, the kernel answer can be restated in plain terms. The same forecasts come from a weighted sum of the raw traits, with a shrink rule that uses a special yardstick. That restatement keeps the forecast exact while the meaning comes back. A user can once more say which traits push a forecast up or down. The work matters where a model must be explained, not just run, from credit to care.

Catalog record

CREATORPatrick J.F. Groenen, Michael Greenacre
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15932

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interpretable Kernels Patrick J.F. Groenen, Michael Greenacre Abstract The use of kernels for nonlinear prediction is widespread in machine learning. They have been popularized in support vector machines and used in kernel ridge regression, amongst others. Kernel methods share three aspects. First, instead of the original matrix of predictor variables or features, each observation is mapped into an enlarged feature space. Second, a ridge penalty term is used to shrink the coefficients on the features in the enlarged feature space. Third, the solution is not obtained in this enlarged feature space, but through solving a dual problem in the observation space. A major drawback in the present use of kernels is that the interpretation in terms of the original features is lost. In this paper, we argue that in the case of a wide matrix of features, where there are more features than observations, the kernel solution can be re-expressed in terms of a linear combination of the original matrix of features and a ridge penalty that involves a special metric. Consequently, the exact same predicted values can be obtained as a weighted linear combination of the features in the usual manner and thus can be interpreted. In the case where the number of features is less than the number of observations, we discuss a least-squares approximation of the kernel matrix that still allows the interpretation in terms of a linear combination. It is shown that these results hold for any function of a linear combination that minimizes the coefficients and has a ridge penalty on these coefficients, such as in kernel logistic regression and kernel Poisson regression. This work makes a contribution to interpretable artificial intelligence. 1 Introduction The last two decades have shown a dramatic improvement of prediction accuracy by artificial intelligence methods through machine learning. The availability of big data combined with highly nonlinear models has led to improved prediction capabilities. However, in many applications, such nonlinear predictions cannot be used due to regulatory requirements. For example, a bank cannot use directly the results from a deep neural net “black box” algorithm for credit scoring because the regulator requires the bank to be able to explain their credit scoring rules. Therefore, explainable artificial intelligence (abbreviated as XAI) has received much attention. Ali et al. 2023 and Dwivedi et al. 2023 give overviews of XAI, classifying the area into different types, where the present contribution would be included in the category of “model explainability”. Furthermore, our approach falls in what Rudin 2019 calls models that are “inherently interpretable” rather than explaining black box models. This paper focuses on kernel methods that are used for nonlinear predictions, where our main contribution lies in re-expressing the kernel penalty term such that a linear interpretation in the features becomes possible. The use of kernels has become a popular and attractive tool to allow for nonlinear predictions. It has gained popularity in machine learning through the method of support vector machines and can be applied in many other techniques that use linear prediction. We refer to Hastie et al. 2008 , Section 5.8, who give a simplified introduction to kernel methods based on the explanation in Wahba 1990 . We provide below a linear algebra interpretation. The main idea is to map each row \mathbf{x}_{i}^{\sf T} of the n\times p matrix \mathbf{X} of features into the vector \bm{\phi}_{i}^{\sf T} of an enlarged feature space represented by the n\times r matrix \bm{\Phi} , where r\gg p . A well-known example is the so-called second degree nonhomogeneous polynomial kernel that is equivalent to choosing \bm{\phi}_{i}^{\sf T} as the vector of all main and two-variable interaction effects, which is a special case of the features in polynomial regression. Then, the linear prediction is done in the enlarged feature space of the \bm{\phi}_{i}^{\sf T} by \bm{\phi}_{i}^{\sf T}\bm{\beta} , which is equivalent to a nonlinear prediction in the original space of the \mathbf{x}_{i}^{\sf T} . As the dimensionality r of the space spanned by \bm{\phi}_{i}^{\sf T} increases towards or beyond the number of observations (rows) n , then some form of shrinkage is needed to avoid overfitting. For very large r , the estimation of explicit coefficients \bm{\beta} of the linear combination \bm{\eta}=\bm{\Phi}\bm{\beta} becomes infeasible. This situation is one where the so-called kernel trick can be used: that is, instead of computing the solution for \bm{\beta} , the optimization can be done directly over the predictions in \bm{\eta} , making use of the n\times n kernel matrix \mathbf{K}=\bm{\Phi\Phi}^{\sf T} . Mappings from \mathbf{X} to \bm{\Phi} that satisfy the so-called Mercer conditions ( Mercer 1909 ; Campbell 2002 ) have very simple expressions for k_{ij}(\mathbf{x}_{i},\mathbf{x}_{j}) as a function of \mathbf{x}_{i} and \mathbf{x}_{j} only (which we will also denote by k_{ij} to simplify notation). For example, the radial basis function (RBF) kernel (or Gaussian kernel) has k_{ij}=\exp({-s\|\mathbf{x}_{i}-\mathbf{x}_{j}\|^{2}}) (with s>0 a scaling hyper-parameter) and the nonhomogeneous polynomial kernel of degree d has k_{ij}=(1+\mathbf{x}_{i}^{\sf T}\mathbf{x}_{j})^{d} . One of the main disadvantages of the use of kernels is that only the resulting predictions \bm{\eta}=\bm{\Phi}\bm{\beta} are found, without knowing \bm{\beta} . Thus, for many kernels no model interpretation is given in terms of the derived features in \bm{\Phi} or in the original features in \mathbf{X} . This paper offers a general solution to this problem. Here, it is shown that if p\geq n the kernel solution can also be expressed in terms of the original features, that is, \bm{\eta}=\bm{\Phi}\bm{\beta}=\mathbf{X}\bm{\gamma} , so that the coefficients in \bm{\gamma} can be interpreted similarly as in regression. In this case, both linear combinations and both kernel penalties are exactly the same. For p<n , we propose to use a least-squares approximation of the kernel penalty that preserves the prediction by \bm{\eta}=\mathbf{X}\bm{\gamma} and thus the linear interpretation through the coefficients in \bm{\gamma} . A simple diagnostic is proposed, the kernel accounted for (KAF), that gives the proportion of the kernel penalty that is accounted for by the approximation. Next to the explainability of the kernel, a second important conclusion of this paper is that the nonlinear prediction through kernels critically depends on the kernel penalty and is equivalent to a linear prediction with a quadratic penalty on the coefficients, with contributions to the penalty not only on the size of the individual coefficients (as for the ridge penalty) but also on their combinations. A third contribution is the realization that the linear interpretation proposed in this paper can be applied to any prediction model that uses a kernel in combination with a quadratic kernel penalty. Examples are support vector machines, kernel ridge regression, kernel logistic regression, kernel Poisson regression, amongst others. With this generality, we contribute to explainable AI. The present paper can be seen as a contribution following the call of Crawford et al. 2018 who provided general approximation for kernels and what they call effect size analogs through projections in Bayesian Approximate Kernel Regression. They write: “It should be clear that a variety of projection procedures can be specified corresponding to various priors and loss functions, and a systematic study elucidating which projections are efficient and robust is of great interest.” We follow a linear algebra derivation leading to coefficients that are exactly the same in cases of a wide matrix of predictors, and approximate otherwise.

Read the complete text