MIBoost: A Gradient Boosting Algorithm for Variable Selection After Multiple Imputation

Robert Kuchen · 2025

Read MIBoost: A Gradient Boosting Algorithm for Variable Selection After Multiple Imputation by Robert Kuchen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21807.txt.

Introduction

Robert Kuchen introduces MIBoost, a boosting method that selects variables once across many imputed datasets at the same time, in work from twenty twenty-five. Missing data plague real analyses, and multiple imputation is the standard patch. Model selection across those patched copies, though, lacks consensus. Pooling separate models performs poorly by known results. Unified-loss tricks already work for penalised regression methods.

MIBoost extends that trick to component-wise gradient boosting. One selection mechanism rules all imputed copies jointly. A matching cross-validation routine tunes the algorithm fairly. Simulations show prediction on par with established rivals. An R package ships the method for practitioners. Boosting joins the principled missing-data toolbox.

Catalog record

CREATORRobert Kuchen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MIBoost: A Gradient Boosting Algorithm for Variable Selection After Multiple Imputation Robert Kuchen Abstract Statistical learning methods for automated variable selection, such as LASSO, elastic nets, or gradient boosting, have become increasingly popular tools for building powerful prediction models. Yet, in practice, analyses are often complicated by missing data. The most widely used approach to address missingness is multiple imputation, which creates several completed datasets. However, there is an ongoing debate on how to perform model selection in the presence of multiple imputed datasets. Simple strategies, such as pooling models across datasets, have been shown to have suboptimal properties. Although more sophisticated methods exist, they are often difficult to implement and therefore not widely applied. In contrast, two recent approaches modify the regularization methods LASSO and elastic nets by defining a single loss function, resulting in a unified set of coefficients across imputations. Our key contribution is to extend this principle to the framework of component‑wise gradient boosting by proposing MIBoost , a novel algorithm that employs a uniform variable‑selection mechanism across imputed datasets. Simulation studies suggest that our approach yields prediction performance comparable to that of these recently proposed methods. Keywords: Gradient boosting, Multiple imputation, Variable selection, Component‑wise gradient boosting, Prediction 1 Introduction Multiple imputation is widely considered the gold standard for handling missing data. When the main interest lies in causal inference, coefficients and standard errors obtained from each imputed dataset can be easily aggregated using Rubin’s pooling rules [ 1 ] . However, performing variable selection for prediction purposes with multiple imputed datasets remains methodologically challenging, as the selected models can differ substantially across datasets. Many previously proposed methods for obtaining a single unified model are complex and insufficiently implemented in software packages. Simply selecting all variables chosen in any of the datasets often leads to overly complex models [ 2 ] , so many users resort to ad hoc solutions, such as including only variables selected in a minimum proportion (e.g., 50%) of datasets. While this heuristic may yield satisfactory results in some cases, it suffers from strong dependence on an arbitrary threshold and can be unstable in small samples or in the presence of correlated predictors [ 3 ] . To address this problem, [ 4 ] proposed a framework that leverages the popular variable selection methods LASSO and elastic nets [ 5 ] . They suggest a framework comprising two different approaches, which both rely on minimizing a single loss function, ultimately resulting in a unified model. This framework is implemented in the R package miselect [ 6 ] . Another statistical‑learning method often considered an alternative to LASSO and elastic nets is component‑wise gradient boosting [ 7 ] , which also implicitly results in coefficient shrinkage and variable selection, making it well‑suited for high‑dimensional data. Whereas LASSO and elastic nets are limited to generalized linear models, boosting accommodates diverse loss functions and nonlinear effects, making it highly versatile. However, no method to date ensures uniform variable selection when applying component-wise gradient boosting to multiply imputed data. This paper extends the idea of defining a single loss function across and proposes a novel algorithm for component‑wise gradient boosting that enforces consistent variable selection across all imputed datasets. Our main contributions are as follows: • We propose MIBoost , a novel boosting algorithm specifically designed for prediction after multiple imputation, which couples base‑learner selection across imputations to improve model stability and interpretability. • We develop a cross‑validation strategy for MIBoost that avoids data leakage by separating training and validation imputations and aggregates performance across imputations in a principled way. • We conduct a comprehensive simulation study assessing the predictive accuracy, variable selection performance, and computational efficiency of MIBoost compared to existing boosting‑based and imputation‑based approaches. The remainder of this paper is structured as follows: Section 2 provides background on existing methods for variable selection after multiple imputation and on component‑wise gradient boosting. Section 3 introduces the proposed MIBoost algorithm together with its cross‑validation framework, MIBoostCV . Section 4 presents a comprehensive simulation study comparing the performance of MIBoost with established methods. Finally, Section 5 discusses our findings and outlines future research directions. 2 Background 2.1 Statistical and Component-Wise Gradient Boosting Statistical gradient boosting [ 8 ] is a machine learning algorithm that minimizes a specified differentiable loss function \rho by incrementally constructing an additive model in a stagewise manner [ 9 , 10 ] . Given n observations, we define the empirical loss as \hat{R}(\bm{\eta})=\sum_{i=1}^{n}\rho\big(y_{i},\eta(\bm{X}_{i})\big), which quantifies the discrepancy between the additive predictor \hat{\bm{\eta}} and the observed responses \bm{y} . Boosting achieves minimization of this loss by iteratively descending \hat{R}(\bm{\eta}) along the direction of its negative gradient. Specifically, at each iteration, we compute \bm{u}=-\nabla_{\bm{\eta}}\hat{R}(\bm{\eta}), which corresponds to the negative gradient of the empirical loss with respect to the current additive predictor (often referred to as pseudo-residuals ). In the special case of squared error loss, these pseudo-residuals coincide with the ordinary residuals. Design Matrix and base-learners The additive predictor \hat{\bm{\eta}} is a function of the design matrix \bm{X} , which contains information on a set of c original candidate covariates (e.g., age , sex ). Centering these covariates to a mean of zero is recommended for numerical stability and comparability. From these, we derive a collection of p candidate functions ( c\leq p ), so that we can express the design matrix as \bm{X}=[\bm{X}_{1},\dots,\bm{X}_{p}]=\begin{bmatrix}\bm{x}_{1}^{T}\\ \vdots\\ \bm{x}_{n}^{T}\end{bmatrix}, where \bm{X}_{j} denotes the vector of values of the j ‑th candidate function across all observations, and \bm{x}_{i} the i ‑th row containing all candidate function values for observation i . Candidate functions may be simple transformations (e.g., age , age 2 ) or more flexible effects, such as penalized splines or random intercepts. Building on these candidate functions, we define base‑learners as simple prediction rules (e.g., univariate regressions or smoothers) that estimate their association with the outcome. In this paper we focus on component‑wise gradient boosting [ 7 ] , in which each base-learner corresponds only to a single candidate function and thus to a single candidate covariate. Furthermore, at each iteration, only a single base‑learner is selected and updated, while all others remain unchanged. Formally, at each boosting iteration, denoted by t in the following, we compute, for each of the p base‑learners, the least‑squares projection of the current negative gradient vector \bm{u}^{[t]} (the pseudo‑residuals) onto the function space of that base-learner: \hat{h}_{\beta_{r}}^{[t]}=\arg\min_{h\in\mathcal{H}_{r}}\sum_{i=1}^{n}\big(u_{i}^{[t]}-h(x_{i,r})\big)^{2}, where \mathcal{H}_{r} denotes the function space associated with base-learner r . The estimated function \hat{h}_{\beta_{r}}^{[t]} is referred to as the empirical base-learner for candidate function r in iteration t . For a simple linear base‑learner, this corresponds to an ordinary least‑squares regression of the pseudo‑residuals on an intercept and the r -th candidate function, yielding the fitted function \hat{h}_{\beta_{r}}^{[t]}(\bm{X}_{r})=\hat{\beta}_{0,r}^{[t]}\bm{1}+\hat{\beta}_{r}^{[t]}\bm{X}_{r}. In each iteration, the base-learner that best fits the pseudo‑residuals is selected: r^{*}=\arg\min_{r}\sum_{i=1}^{n}\big(u_{i}^{[t]}-\hat{h}_{\beta_{r}}^{[t]}(x_{i,r})\big)^{2}. The additive predictor is then updated by moving a small step \nu (typically \nu=0.1 ) in the direction of the selected base-learner: \bm{\hat{\eta}}^{[t]}=\bm{\hat{\eta}}^{[t-1]}+\nu\hat{h}_{\beta_{r^{*}}}^{[t]}. Note that the same base‑learner can be selected multiple times. Consequently, with only one base‑learner updated per iteration, a small stopping iteration t_{stop} typically leaves some covariates excluded and, due to the small step length, results in coefficient shrinkage. This makes component-wise gradient boosting conceptually related to penalized regression

Read the complete text