Model Class Selection
Ryan Cecil, Lucas Mentch Department of Statistics, University of Pittsburgh · 2025
Read Model Class Selection by Ryan Cecil, Lucas Mentch Department of Statistics, University of Pittsburgh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11355.txt.
Introduction
Ryan Cecil and Lucas Mentch introduce model class selection, a generalization of the recent idea of selecting a set of near-optimal models rather than one winner. Classical selection picks a single model by a fixed criterion, while set selection keeps every model that is plausibly optimal. Their step up asks which whole collections of models, among several rival collections, contain at least one optimal model.
Under mild conditions, data-splitting methods are shown to solve the problem in general. That yields a formal way to test whether simpler, interpretable model classes can match complex black-box methods on a given data set. Simulated and real-data experiments illustrate the approach across cases.
Catalog record
| CREATOR | Ryan Cecil, Lucas Mentch Department of Statistics, University of Pittsburgh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model Class Selection Ryan Cecil, Lucas Mentch Department of Statistics, University of Pittsburgh Abstract Classical model selection seeks to find a single model within a particular class that optimizes some pre-specified criteria, such as maximizing a likelihood or minimizing a risk. More recently, there has been an increased interest in model set selection (MSS), where the aim is to identify a (confidence) set of near-optimal models. Here, we generalize the MSS framework further by introducing the idea of model class selection (MCS). In MCS, multiple model collections are evaluated, and all collections that contain at least one optimal model are sought for identification. Under mild conditions, data splitting based approaches are shown to provide general solutions for MCS. As a direct consequence, for particular datasets we are able to investigate formally whether classes of simpler and more interpretable statistical models are able to perform on par with more complex black-box machine learning models. A variety of simulated and real-data experiments are provided. Keywords: Model Selection, Rashomon Effect, Model Confidence Set, Universal Inference 1 Introduction In statistical and machine learning (ML) studies, it is standard to use model evaluation metrics such as AIC, BIC, step-wise selection, holdout, or cross-validation to identify a single empirically optimal model ( Hastie et al. 2009 ) . These methods are used to compare the predictive performances of various modeling algorithms often associated with different model classes. For example, suppose a data scientist wishes to construct a model that can predict if a loan applicant will default in less than a year based on the available data at their company. Given the data, they know strategies for fitting various types of models such as an additive linear model, random forest, and neural network. They are likely unsure, however, of which type of model will perform best. A common approach is thus to simply identify the model type that minimizes some empirical metric (e.g. cross-validation error) and re-fit on the full dataset to obtain a final model for inference or prediction. Oftentimes, many well-performing models exist that optimize the chosen model selection criteria. This concept, called the Rashomon effect, may lead researchers with different datasets to construct entirely different models ( Breiman 2001 ) . In recognition of this effect, procedures for model set selection (MSS) have been growing in popularity. Such MSS methods generally seek to identify an entire collection of similarly optimal models (i.e. an approximate Rashomon set ) rather than relying on only the single one identified via classical model selection methods (e.g. see Fisher et al. 2019 , Wasserman et al. 2020 , Kissel & Mentch 2024 ). Frequently, these sets of models are computed such that they are guaranteed to contain the optimal model(s) of a given type with a desired level of confidence under certain conditions and sample sizes. The size of the Rashomon set can be seen as providing a measure of the uncertainty of the model selection problem. In addition, the computed model set can be used to explore the data and identify important predictors ( Fisher et al. 2019 ) . Recently, ML methods have achieved significant breakthroughs in a variety of fields, surpassing benchmark performances once thought to be unattainable. These advances have been driven in large part by the modern growth of computational resources, allowing researchers to train large scale models with potentially billions of parameters. Compute at scale, however, often comes with a cost. Unlike classical statistical models that are interpretable, have quantifiable uncertainty, and are able to be used to conduct valid inference, the theoretical underpinnings of ML approaches are not fully developed ( Hunter & Holmes 2023 ) . Recent work in mathematics, statistics, and computer science have made significant strides towards addressing these issues (e.g. Abdar et al. 2021 , Berner et al. 2022 , Mentch & Hooker 2016 , Rudin et al. 2022 ). However, there still exists a significant trade-off in interpretability and theoretical guarantees when one moves from simple, well-studied classes of models (e.g. linear models) to complex classes (e.g. random forests or neural networks) to improve predictive accuracy. This is often an issue in studies focused on inference or causality and settings where the model is employed in high-stakes environments such as healthcare or criminal justice ( Wexler 2017 , Varshney & Alemzadeh 2017 , Rudin 2019 ) . On the other hand, it is also somewhat common for practitioners to rule out classical models a priori whenever the data is assumed to be too large or complex. Recent research, however, suggests that this may be frequently misguided as there are numerous real-world settings in which simpler, interpretable models can perform on par with ML alternatives ( Rudin 2019 , Semenova et al. 2022 , Semenova et al. 2023 , Boner et al. 2024 ) . When this is the case, researchers may naturally prefer the simpler class in most settings. On the other hand, if complex ML models significantly outperform simpler alternatives, then there may be a preference for performance over interpretability. To our knowledge, little theoretical attention seems to have been given to formally detecting when these trade-offs exist. To address this, in this work, we introduce model class selection (MCS) methods that are able to determine with high probability when no model within a simple class can match the performance gains realized by more complex ML models. We formally investigate this problem in Section 2 by proposing a general MCS hypothesis testing framework. In Section 3 , we extend data splitting based tools to this general MCS setting. In Sections 4 and 5 , we consider alternative approaches to MCS and examine the power of our proposed methodology in simulated and real-world settings before concluding with a discussion in Section 6 . 2 Constructing Methods for MCS Denote the data as {\cal D}^{(n)}=\left\{Z_{1},\dots,Z_{n}\right\} where Z_{1},\dots,Z_{n}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}P are independent and identically distributed with support over the set \mathcal{Z} . Let \mathcal{H}\subseteq\{h\;|\;h:\mathcal{Z}\rightarrow\mathcal{Y}\} denote a model class, defined as a set of measurable functions mapping from the data space \mathcal{Z} to a prediction space \mathcal{Y} . Define \ell:\mathcal{H}\times\mathcal{Z}\rightarrow\mathbb{R}^{+} to be a measurable function that specifies the error of a model h\in\mathcal{H} on some data point z\in\mathcal{Z} . The goal of classical model selection is to choose a model h\in\mathcal{H} that minimizes the risk R(h)=\mathbb{E}_{Z_{0}}[\ell(h,Z_{0})] , defined as the expected loss of h at an independent test point Z_{0}\sim P . To formally establish the goals of MCS, for j=1,...,d where d\geq 2 , let T_{j} denote a non-empty index set corresponding to models h_{t}\in\mathcal{H} for any t\in T_{j} . Define \mathcal{H}_{j}\coloneq\left\{h_{t}:t\in T_{j}\right\} for j=1,...,d to be the model classes of interest. Denoting [d]\coloneq\left\{1,...,d\right\} , our goal is to form a confidence set for \Theta=\Theta\left(P\right)\coloneq\argmin_{j\in[d]}\inf_{t\in T_{j}}R\left(h_{t}\right) which is the set of all indices corresponding to model classes containing at least one model that achieves the optimal risk. In other words, given r\in[d] , if r\in\Theta , then the model class \mathcal{H}_{r} contains a model that performs as well or better than any other model from any class. On the other hand, if r\not\in\Theta , then the model class \mathcal{H}_{r} should not be considered if the goal is to choose a model with the best possible performance. Although the problem above