Determining the K K in K K -fold cross-validation

Kenichiro McAlinn & Kōsaku Takanashi · 2025

Read Determining the K K in K K -fold cross-validation by Kenichiro McAlinn & Kōsaku Takanashi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12698.txt.

Introduction

Kenichiro McAlinn and Kōsaku Takanashi question a habit every data analyst knows: splitting data for cross-validation by convention, such as eighty percent for training and twenty percent for testing. That split sets training size and test certainty, yet the usual choices are rules of thumb. The authors formalize the trade-off by scoring predictive performance while explicitly penalizing uncertainty in the evaluation itself, a quantity that needs stated assumptions.

They derive formulas for that uncertainty and turn them into a rule for choosing the hold-out size from an assumed noise level. Tests with linear regression and random forests across several domains, plus a high-dimensional genomics case, show the best split depends on the data, the model, and the noise assumption. The conclusions drawn from the same dataset can shift with the split, so the authors urge making the choice explicit rather than automatic.

Catalog record

CREATORKenichiro McAlinn & Kōsaku Takanashi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Determining the K K in K K -fold cross-validation Kenichiro McAlinn & Kōsaku Takanashi Abstract Cross-validation is a standard technique used across science to test how well a model predicts new data. Data are split into K K ”folds,” where one fold (i.e., hold-out set) is used to evaluate a model’s predictive ability. Researchers typically rely on conventions when choosing K K , commonly K = 5 K=5 , or 80 : 20 80{:}20 split, even though the choice of K K can affect inference and model evaluation. Principally, this K K should be determined by balancing the predictive accuracy (bias) and the uncertainty of this accuracy (variance), which forms a tradeoff based on the size of the hold-out set. More training data means more accurate models, but fewer testing data lead to uncertain evaluation, and vice versa. The challenge is that this evaluation uncertainty cannot be estimated directly from data. We propose a procedure to determine the optimal K K by deriving a finite-sample upper bound on the evaluation uncertainty and adopting a utility-based approach to make this tradeoff explicit. Analyses of real-world datasets using linear regression and random forest demonstrate this procedure in practice, providing insight into implicit assumptions, robustness, and model performance. Critically, the results show that the optimal K K depends on both the data and the model, and that conventional choices implicitly make assumptions about the fundamental characteristics of the data. Our framework makes these assumptions explicit and provides a principled, transparent way to select K K based on the data and model rather than convention. By replacing a one-size-fits-all choice with context-specific reasoning, it enables more reliable comparisons of predictive performance across scientific domains. Keywords : Cross-validation; Model evaluation; Prediction uncertainty; Bias–variance tradeoff Introduction In typical data analysis, the researcher has a plethora of models or hyperparameters to choose from. For example, a researcher often has access to a set of models that generate drastically different inferences or predictions, and the researcher must choose one of these models to make informed decisions. Which model to use– i.e., model uncertainty– is a central issue in statistics and machine learning. To assess model performance, a common technique used is cross-validation (CV), where a certain portion of the data is randomly allocated to train the model, and the remaining portion is used to evaluate the model through its predictions. Cross-validation, developed simultaneously and independently by Seymour Geisser and Mervyn Stone ( 7 ) , is popular, and used across a broad breadth of fields (e.g., machine learning, econometrics, genomics, and epidemiology, to name just a few), because the concept and implementation are simple, while not being constrained to a specific data type or a domain, and theoretically known to produce unbiased estimates of the prediction loss ( 9 ; 12 ; 3 , see, e.g.,) . In simple terms, cross-validation tests how well a model trained on part of the data predicts the rest. Repeating this process multiple times provides an estimate of how the model will generalize to unseen data. The most common version is K -fold cross-validation, where the data are split into K equal-sized folds, and the test set is constructed by removing one of them. In practice, K is determined by convention (e.g., 5-fold CV, or 80:20 split), and does not typically differ across domain, dataset, or model. Even though this choice of K directly affects selection and inference, there is no principled way to determine this K with theoretical justification. Consider a concrete example: a researcher analyzing a dataset might find that Model A outperforms Model B using 5-fold CV, but Model B beats Model A using 10-fold CV. This could occur because Model B is more complex than Model A, requiring more data to produce good forecasts, but it could also occur because, particularly if the total sample size is small, the test data happened to favor Model B. In either case, the researcher could select the wrong model and reach incorrect scientific conclusions if the number of folds is poorly chosen. More importantly, a researcher faced with different results based on the number of folds does not have a principled way to discern which to use. The difficulty in determining the hold-out size is due to the inherent tradeoff that exists in cross-validation. The more training data one has, the better the model estimation will be. Conversely, the more test data one has, the better the performance assessment of the model will be. Given a fixed total sample size, a tradeoff exists between the accuracy of the model and the uncertainty of the model evaluation. Consider an extreme case where all of the data are allocated to the training set. The model is the most accurate, using all the data to train, but there is no model evaluation (due to not having any test samples), and the uncertainty regarding the assessment is the largest. At the other extreme, if all data are allocated to the test set, the model cannot learn from the data and can only predict based on prior assumptions. The evaluation uncertainty, however, is the smallest because all of the data are used to evaluate. This tradeoff suggests that an optimal balance exists between having sufficient training data for learning and enough test data for reliable evaluation. This simple thought experiment illustrates why a universal choice of K cannot exist, since the optimal split depends on how much uncertainty one can tolerate in evaluating model accuracy. Our goal is to formalize this tradeoff to enable a systematic choice of K . More precisely, this tradeoff is between the prediction loss of the model and its uncertainty regarding the prediction loss; similar to the bias-variance tradeoff regarding model complexity. While this tradeoff has been intuited, it has never been rigorously explicated due to the impossibility of a universal unbiased estimator for the latter uncertainty ( 6 ) , except for specific problem settings ( 1 , e.g.,) , and only asymptotic results have been derived for the purpose of testing model performance ( 2 ; 4 ; 10 ) . While these results provide valuable asymptotic insights, they do not yield a usable criterion for choosing K in a given dataset with finite samples. For our theoretical contribution, we derive a finite sample upper bound (the theoretical maximum) to this uncertainty, rigorously linking it to the prediction loss. Given this uncertainty bound, the optimal test set size can be derived by balancing this tradeoff. We express this tradeoff using a mean-variance utility (a combined score of accuracy and reliability) of the von Neumann-Morgenstern utility function. The optimal test set size maximizes this utility function, given the researcher’s belief in the irreducible error in the data-generating process, which acts as a tuning parameter. A key insight of our approach is recognizing that any choice of K implicitly assumes the level of irreducible noise in the data-generating process. Thus, choosing K=5 assumes one noise level and K=10 assumes another. Current practice makes these choices by convention without examining what they imply about the fundamental characteristics of the data. We make these implicit assumptions explicit, transforming convention into a transparent, data-informed decision. In the following, we formalize this tradeoff, derive theoretical

Read the complete text