Tree-based methods for estimating heterogeneous model performance and model combining
Ruotao Zhang, Constantine Gatsonis, Jon Steingrimsson · 2025
Read Tree-based methods for estimating heterogeneous model performance and model combining by Ruotao Zhang, Constantine Gatsonis, Jon Steingrimsson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01905.txt.
Introduction
Ruotao Zhang, Constantine Gatsonis, and Jon Steingrimsson present a 2025 statistics paper on finding groups of people for whom a prediction model works better or worse. Scores are usually reported for a whole population, which can hide weak spots in smaller groups. Their tree-based methods search the data for splits that widen the gap in performance between groups.
They extend the idea from single trees to forests and boosting ensembles, where many trees vote together. The ensembles can also blend several models, giving more weight where each performs best. Tests on simulated data and a real lung cancer screening dataset show how the methods expose uneven performance.
Catalog record
| CREATOR | Ruotao Zhang, Constantine Gatsonis, Jon Steingrimsson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tree-based methods for estimating heterogeneous model performance and model combining Ruotao Zhang, Constantine Gatsonis, Jon Steingrimsson Abstract Abstract: Model performance is frequently reported only for the overall population under consideration. However, due to heterogeneity, overall performance measures often do not accurately represent model performance within specific subgroups. We develop tree-based methods for the data-driven identification of subgroups with differential model performance, where splitting decisions are made to maximize heterogeneity in performance between subgroups. We extend these methods to tree ensembles, including both random forests and gradient boosting. Lastly, we illustrate how these ensembles can be used for model combination. We evaluate the methods through simulations and apply them to lung cancer screening data. 1 Introduction The performance of a predictive model is typically assessed using overall measures for the population under consideration. However, model performance often varies across different covariates, resulting in certain subgroups that show superior or inferior performance compared to the overall population [ 1 ] . Consequently, overall performance measures may not accurately represent performance within specific subgroups. When model-derived predictions are used for individualized decision making, it is essential that the measures of model performance reflect the subgroup of interest. Traditionally, variation in performance across subgroups is explored by estimating model performance within a limited set of predefined subgroups. The issue of differential model performance is frequently highlighted as a key aspect of fairness, or the lack thereof, in machine learning and artificial intelligence [ 2 , 3 ] . Data-driven tree-based methods have previously been developed to discover subgroups with differential treatment effects without requiring pre-specification of subgroups [ 4 , 5 , 6 ] . In contrast, methods to identify subgroups with differential model performance are less explored. To the best of our knowledge, the only existing data-driven method for identifying such subgroups is presented in [ 7 ] . These authors focus specifically on longitudinal data and use splitting decisions based on interaction tests within a shared random effects model. Consequently, their approach applies only to specific data structures, and the validity of their splitting criteria relies on the correct specification of this shared random effects model. In the first part of this manuscript, we develop two general tree-based methods for the data-driven discovery of subgroups with differential model performance, applicable to any measure of model performance. These methods produce interpretable subgroup structures by recursively partitioning the covariate space into disjoint subgroups that maximize between-subgroup heterogeneity or within-subgroup homogeneity of model performance. Our proposed methods accommodate scenarios where the number of covariates exceeds the sample size and do not require pre-specification of potential subgroups. While the resulting binary tree structure is easily interpretable, it may exhibit high variability. Ensemble methods often outperform single trees in related contexts; thus, we additionally extend our approach to random forest and gradient boosting frameworks to estimate conditional measures of model performance for any given predictive model. In the second part of the manuscript, we develop ensemble-based methods for model combination, building upon the methods introduced in the first part. There exists extensive literature on model combination, often emphasizing dynamic model selection. For instance, [ 8 , 9 ] estimated model classification error rates in local regions around specific covariate patterns (regions of competence) and selected the optimal model(s) for prediction at these covariate patterns. [ 10 ] provided a comprehensive overview of dynamic model selection approaches. Typically, regions of competence are defined using algorithms such as clustering or K-Nearest Neighbors. Other methods focused on weighting models include the super learner [ 11 ] and the mixture of experts [ 12 ] . Both methods estimate weights from data (instead of applying fixed rules such as majority voting), but neither is explicitly designed to combine pre-trained predictive models. Moreover, while the super learner provides a uniform set of weights across all covariate patterns, the mixture of experts approach allows for covariate-dependent weights. Our proposed approach can be utilized with any measure of model performance and eliminates the subjective choice regarding the size of the region of competence. The paper is structured as follows. In Sections 3.1 and 3.2 , we propose two extensions of the classification and regression tree algorithm [ 13 ] to identify subgroups with differential model performance. In Section 5 , we extend the single-tree methods to ensemble methods. In Sections 5.1 and 5.2 , we apply these ensemble methods to estimate covariate-dependent weights for model combination. Finally, we evaluate the finite-sample performance of the proposed algorithms through simulations and an application to lung cancer screening data. 2 Notation Let X=(X^{(1)},\dots,X^{(p)})\in\mathcal{X} be a p -dimensional covariate vector, and Y\in\mathcal{Y} an outcome. A set w is a subgroup if w\subseteq\mathcal{X} . We assume that the data consist of n i.i.d. observations \mathcal{D}=\{(X_{i},Y_{i}):i=1,\dots,n\} sampled from the joint distribution of (X,Y) . We are interested in data-driven discovery of subgroups with differential model performance for a specific measure of model performance and for an already fit prediction model that we denote by h(X) . This incorporates prediction models h(X) that were fit using external data and prediction models that were fit using a training set independent of \mathcal{D} . For reasons that will become clear later, we separate measures of model performance into those that can be calculated for a single observation i and those that cannot. Examples of models performance measures that can be calculated for a single observation i are loss-based measures as for a given loss function L(Y,h(X)) the loss associated with observation i is L(Y_{i},h(X_{i})) . On the other hand, the area under the ROC curve (AUC) for a subgroup w is defined as AUC(w)=\mathbb{E}[I\{h(X)>h(X^{\prime})\}\mid X\in w,X^{\prime}\in w,Y=1,Y^{\prime}=0], where (Y,X) and (Y^{\prime},X^{\prime}) is a random pair of observations from the subgroup w with the former being a case (Y=1) and the latter being a control (Y^{\prime}=0) . As AUC(w) is defined using a pair of observations, it is an example of a measure of model performance that cannot be calculated for each individual observation. For a given measure of model performance, We use \mu_{i} to denote the observed model performance for observation i (if available) and \widehat{\mu}(w) to denote the estimated model performance in subgroup w . For example, for loss-based measures, we have \mu_{i}=L(Y_{i},h(X_{i})) and \widehat{\mu}(w)=\frac{1}{\sum_{i=1}^{n}I(X_{i}\in w)}\sum_{i=1}^{n}I(X_{i}\in w)L(Y_{i},h(X_{i})) . 3 Tree-based methods for discovering subgroups In this section we present two methods for discovering subgroups with differential model performance. 3.1 Regression tree with transformed outcome The original Classification and Regression Tree (CART) algorithm [ 13 ] has three main steps: initial tree building, tree pruning, and final tree selection. The final output of the CART algorithm is a set of subgroups and subgroup-specific estimators where the subgroups are chosen to minimize within-subgroup heterogeneity in the outcome. Following the same principle, we modify the CART algorithm to discover subgroups that minimize within-subgroup heterogeneity in model performance. The first step is to grow a large initial tree. The tree-growing process starts with all observations in \mathcal{D} in a single group. For a covariate X^{(j)} and a cut point c , define the pair of subgroups L_{j,c}=\{X\in\mathcal{X}:X^{(j)}\leq c\} and R_{j,c}=\{X\in\mathcal{X}:X^{(j)}>c\} . To make splitting decisions, the algorithm cycles through all possible covariate and split-point pairs (j,c) and finds the pair that optimizes a splitting statistic. Once