Beyond Point Estimates: Distributional Uncertainty in Machine Learning Performance Evaluation
Christoph Lehmann, Yahor Paromau · 2025
Read Beyond Point Estimates: Distributional Uncertainty in Machine Learning Performance Evaluation by Christoph Lehmann, Yahor Paromau (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16931.txt.
Introduction
Christoph Lehmann and Yahor Paromau argue that single scores hide too much about model quality. Their paper appeared in two thousand twenty-five. Accuracy or error is often reported as one fixed number. Yet training involves chance at many steps. Data splits, starting values, and tuning searches all vary. The same recipe can yield a spread of results.
The authors treat each metric as a random quantity with its own spread. They study quantiles of that spread and the confidence ranges around them. Real cases in classification and regression anchor the work. Simulations check the methods under known truth. Small samples get special care, since practice often allows few runs. The paper gives teams a more honest way to report performance.
Catalog record
| CREATOR | Christoph Lehmann, Yahor Paromau |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16931
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Point Estimates: Distributional Uncertainty in Machine Learning Performance Evaluation Christoph Lehmann, Yahor Paromau Abstract Machine learning models are often evaluated using point estimates of performance metrics such as accuracy, F1 score, or mean squared error. Such summaries fail to capture the inherent variability induced by stochastic elements of the training process, including data splitting, initialization, and hyperparameter optimization. This work proposes a distributional perspective on model evaluation by treating performance metrics as random quantities rather than fixed values. Instead of focusing solely on aggregate measures, empirical distributions of performance metrics are analyzed using quantiles and corresponding confidence intervals. The study investigates point and interval estimation of quantiles based on real-data use cases for classification and regression tasks, complemented by simulation studies for validation. Special emphasis is placed on small sample sizes, reflecting practical constraints in machine learning, where repeated training is computationally expensive. The results show that meaningful statistical inference on the underlying performance distribution is feasible even with sample sizes in the range of 10–25, while standard nonparametric confidence interval remain applicable under these conditions. The proposed approach provides a more detailed characterization of variability and uncertainty compared to mean-based evaluation and enables a more differentiated comparison of models. In particular, it supports a risk-oriented interpretation of model performance, which is relevant in applications where reliability is critical. The presented methods are easy to implement and broadly applicable, making them a practical extension to standard performance evaluation procedures in machine learning. Keywords— machine learning, model evaluation, distributional uncertainty, performance metrics, quantile estimation, confidence intervals, bootstrap methods, nonparametric statistics 1 Introduction In machine learning training, outcomes are influenced by numerous factors, including the train-test split, optimizer choice, initial weights, hyperparameter optimization etc. In the sense of scientific meaning, these factors are so-called confounding factors and analyzing those is considered good practice, as it enables a comprehensive understanding of the experimental conditions. In contrast, while confounding factors are examined in the typical machine learning workflow, this is often done in an unsystematic manner, with a primary focus on achieving the best single value for a target metric of interest. But simply selecting the best single value does not reflect the training process as an experiment, as even with a fixed choice of (hyper)parameters, results can vary due to the probabilistic nature of the modeling approach, for example due to weights initialization or data augmentation. The objective of this work is to represent results of machine learning training as a distribution of the target metric of interest (such as e.g. accuracy or root mean squared error) conditional on the confounding factors. Drawing from a distribution of some target metric of interest can be realized by using seed-controlled train runs of a machine learning pipeline. Thereby, one confounding factor is varied based on seeds, as e.g. initial weights of a model, the train-test data split etc. The resulting single values can be interpreted as repeated measurements from the distribution of the target metric of interest for a given confounding factor, that allows to quantify its impact on the target metric of interest. A distributional view allows for the evaluation of the training process in terms of uncertainty that can be quantified from a statistical perspective. Generating such repeated measurements can be quite costly, as running machine learning training needs time and computing resources. Therefore, a small to medium amount of such measurements is assumed throughout this contribution. This amount of measurements is referred to as the sample size n which will be in the order of 10-50. Note that this sample size n needs to be clearly separated from the data volume (train or test data size) within some machine learning training. The presented methodology does not consider the data size that is used for the training itself. Figure 1 shows examples of empirical distributions of the root mean squared error for a regression problem and the accuracy rate of a classification problem. Each of these empirical distributions is generated based on approximately 1\,000 seed-controlled train runs for different confounding factors. (a) Empirical distributions of root mean squared error for regression. (b) Empirical distributions of accuracy for classification. Figure 1: Examples of empirical distributions of TMoIs based on approx. 1\,000 seed-controlled train runs. Figure 1(a) shows two regression approaches for the same data (this is from the real-data use case Superconductors, see Section 4.1 ): deep neural network (DNN, fully connected) and gradient boosting trees (GBT). For both approaches, the different train runs are generated by seed-controlled train-test splits. Figure 1(b) illustrates classifier training runs using a convolutional neural network applied to the same dataset (from the real-data use case Simpsons characters, see Section 4.1 ). One set of training runs was generated through seed-controlled hyperparameter optimization (HO), specifically using the Tree-Structured Parzen Estimator, where the seeds control initialization. The other set of runs was generated through seed-controlled data augmentation. In practice, hundreds or even thousands of repetitive measurements are rarely available. As a consequence, the actual resulting empirical distributions consisting of observations from 10 or 25 repeated measurements typically will not look so smoothly as shown in Figure 1 . Hence, the contribution at hand demonstrates how to work with modestly sized samples to gain insights into the unknown underlying distribution. Such analyses are a typical task of inferential statistics and involve considerations of uncertainty. This contribution aims to bridge the gap between statistics and machine learning by integrating statistical perspectives into machine learning practice. When interconnecting statistics and machine learning, clear definitions of terms are crucial. Therefore, the following central terms are defined as follows. • Theoretical Distribution: An idealized or assumed model for how probability mass is distributed. Such a model can be characterized by parameters or quantities as mean, variance, quantiles, etc. The theoretical true values of these quantities are typically unknown and need to be estimated based on data. • Estimation: The process of deriving or approximating unknown quantities based on observed data. Estimation can result in a single value (point estimation) or a range of values (interval estimation). • Uncertainty: The variability of an estimator due to sampling or algorithmic stochasticity. • Confidence Interval: Estimation that provides a range of plausible values for a statistical quantity to reflect the estimation uncertainty. Note that the above terms mainly reflect the frequentist perspective of statistics. Bayesian approaches, in contrast, treat the quantity of interest as a random variable and rely on prior information to produce a posterior distribution. While Bayesian methods can provide posterior distributions for performance metrics, this contribution focuses on frequentist approaches, which are more familiar to practitioners with classical statistical training from engineering or medicine. The core contributions of this article are as follows: 1. Highlighting the stochastic nature of TMoIs through multiple training runs, providing an engineering perspective on the number of repetitions needed for stable model evaluation. 2. Conducting unusually large-scale repeated measurements (500–1000 runs per setting) to ensure robust empirical insights. 3. Characterizing the empirical distribution of TMoIs using quantiles, which allows a risk-oriented perspective on performance. 4. Quantifying the statistical uncertainty of estimated quantiles via CIs. 5. Demonstrating that the proposed approach applies to all continuous TMoIs, including classification metrics (accuracy, precision, F1, etc.) and regression metrics (root mean squared error,