A Stationary-Distribution Theory for Triplet-Based Plateau Search in Random Forest Ensemble-Size Selection
Andrey A. Dukhovny [email protected], Andrey M. Lange [email protected] · 2026
Read A Stationary-Distribution Theory for Triplet-Based Plateau Search in Random Forest Ensemble-Size Selection by Andrey A. Dukhovny [email protected], Andrey M. Lange [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30837.txt.
Introduction
Andrey Dukhovny and Andrey Lange build a theory for a practical question in random forests: how many trees are enough? Adding trees reduces the wobble of a finite forest but costs training and prediction time. Plateau-based tuning compares out-of-bag scores at three geometrically spaced tree counts and moves the middle one, yet once other settings settle, that middle count does not converge to a fixed value. It fluctuates in a stationary pattern instead.
The pair model the central count as a birth-death Markov chain on a geometric grid and derive its stationary distribution from local balance. The equilibrium centre grows like the inverse square of the tolerance, and the spread scales the same way. The leading relative spread, however, is independent of the tolerance and set by the grid spacing and update rule.
Catalog record
| CREATOR | Andrey A. Dukhovny [email protected], Andrey M. Lange [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Stationary-Distribution Theory for Triplet-Based Plateau Search in Random Forest Ensemble-Size Selection Andrey A. Dukhovny [email protected], Andrey M. Lange [email protected] Abstract The number of trees is a central computational parameter in Random Forests: increasing it reduces finite-ensemble variability but increases training and prediction cost. Plateau-based tuning adapts this parameter through local comparisons of out-of-bag scores at a geometric triplet of tree counts. After the remaining hyperparameters have stabilized, however, the central triplet point need not converge to a deterministic value; instead, it fluctuates around a stationary regime. This paper develops a stationary-distribution theory for this process. The central ensemble size B t B_{t} is modeled as a birth–death Markov chain on a geometric grid, and its stationary distribution is derived through local balance. Under a leading centered folded-normal approximation, equilibrium equations are obtained for the original update rule and a symmetric modified variant, implying that the stationary center B ∗ = O ( ε − 2 ) B_{*}=O(\varepsilon^{-2}) as ε ↓ 0 \varepsilon\downarrow 0 . The stationary spread is also characterized. A local Gaussian approximation and a Fokker–Planck interpretation give grid-level variance constants. After conversion to the ensemble-size scale, σ B , ∗ = O ( ε − 2 ) \sigma_{B,*}=O(\varepsilon^{-2}) , while the variance is O ( ε − 4 ) O(\varepsilon^{-4}) . The leading relative spread is independent of ε \varepsilon and controlled by the scale factor and update rule. These results interpret plateau-based Random Forest tuning as a stochastic process rather than a deterministic stopping rule. \declaretheorem [name=Proposition]myproposition \declaretheorem [name=Theorem]mytheorem Andrey A. Dukhovny and Andrey M. Lange † † heading: 2026 1- Andrey A. Dukhovny and Andrey M. Lange † † shortheadings: Stationary Theory for Plateau Search / Dukhovny and Lange † † firstpage: 1 † † editor: To be assigned keywords random forest, ensemble-size selection, plateau search, stationary distribution, birth–death Markov chain 1 Introduction The setting is supervised learning on tabular data with n observations and p input features. After standard preprocessing of categorical variables, such data can be represented by a design matrix X\in\mathbb{R}^{n\times p} , whose rows correspond to observations and columns to features, together with a target vector y . Tree-based ensembles remain state-of-the-art for tabular data, consistently outperforming deep learning in many benchmark studies ( Grinsztajn et al. 2022 ; Shwartz-Ziv and Armon 2022 ; Borisov et al. 2024 ) . Among them, Gradient Boosting ( Friedman 2001 ; Chen and Guestrin 2016 ) often delivers superior predictive accuracy, while Random Forest ( Breiman 2001 ; Biau and Scornet 2016 ) offers greater stability. This stability stems from averaging over many randomized trees: the Monte Carlo component of the ensemble prediction decreases as the number of trees grows. The independence of trees also enables parallel training and the use of out-of-bag (OOB) scores, which provide an internal performance estimate without cross-validation. Beyond prediction, Random Forests provide variable importance measures (VIMs), such as the Mean Decrease in Impurity (MDI), which naturally arise from the tree-building procedure ( Breiman 2001 ; Louppe et al. 2013 ) . These VIMs are widely used for feature selection, network inference, and scientific discovery ( Strobl et al. 2007 ; Kursa et al. 2010 ; Ewald et al. 2024 ) . However, stabilizing VIMs in high-dimensional settings with correlated features may require substantially more trees than stabilizing the predictive score itself ( Lange et al. 2025 ; Tolosi and Lengauer 2011 ) . A well-calibrated predictive score is therefore a necessary, though not sufficient, condition for trustworthy VIMs. This observation reinforces the motivation of the present line of work: before one can reliably assess variable importance, one must first obtain a Random Forest with stable and sufficiently accurate predictive performance. The number of trees T is therefore a central computational parameter of Random Forests. Increasing T reduces finite-ensemble variability, but it also increases training and prediction cost. Standard hyperparameter optimization (HPO) methods, such as TPE ( Bergstra et al. 2011 ) or Hyperband ( Li et al. 2018 ) , require the user to specify a search range [T_{\min},T_{\max}] . Because adding trees does not induce the usual overfitting behavior of many other hyperparameters, the selected value of T tends to be driven toward the upper boundary T_{\max} . Raising T_{\max} shifts the selected value further toward the boundary, offering no guarantee that the chosen bound is either sufficient or computationally efficient. Early-stopping heuristics avoid an explicit upper bound by monitoring incremental score improvements, but they may stop prematurely when OOB score fluctuations make the observed improvement appear small. In a recent paper, Porvatov et al. 2026 introduced a triplet-based plateau search procedure that adapts the number of trees without requiring a fixed T_{\max} . At each HPO trial, the non- T hyperparameters are sampled in the usual way, while the ensemble size is represented by a geometric triplet L=B/\mathrm{sf} , B , and R=B\cdot\mathrm{sf} , with a fixed scale factor \mathrm{sf}>1 . The geometric construction keeps the relative separation between neighboring ensemble sizes approximately constant, since R-B=(\mathrm{sf}-1)B and hence (R-B)/B=\mathrm{sf}-1 . This is important because, for a fixed additive step R=B+\Delta , the relative separation satisfies (R-B)/B=\Delta/B\to 0 as B\to\infty , making neighboring forests increasingly difficult to distinguish by their OOB scores. Fixed increments of this type, for example \Delta=10 , were used in earlier ensemble-size studies ( Latinne et al. 2001 ; Lange et al. 2025 ) . Thus, the scale factor \mathrm{sf} acts as a resolution parameter for the plateau comparison. The forest is trained sequentially up to R trees, and the OOB scores at L , B , and R are recorded. The relative score gaps d_{L}=\frac{|S_{B}-S_{L}|}{|S_{B}|},\qquad d_{R}=\frac{|S_{R}-S_{B}|}{|S_{B}|} (1) indicate how close the central ensemble size B is to the plateau region. They are compared with a user-specified tolerance \varepsilon , typically of the order 10^{-3} . If both inequalities d_{L}\leq\varepsilon and d_{R}\leq\varepsilon hold, the ensemble is considered stable but potentially unnecessarily large, and the triplet is shifted left for the next trial. In the original rule, if d_{R}>\varepsilon , the improvement from B to R is still above the tolerance, and the current ensemble is treated as insufficient; the triplet is therefore shifted right. In the remaining original-rule case, d_{L}>\varepsilon and d_{R}\leq\varepsilon , the triplet remains at the current level. A symmetric modified rule is also analyzed below; in this variant, the mixed case d_{L}\leq\varepsilon<d_{R} is assigned to staying rather than to a right shift. Importantly, trials that trigger a right shift are not considered later for selecting the trial with the highest score, because they are deemed too unstable to be trusted. This adaptive mechanism eliminates the need for an arbitrary upper bound T_{\max} and avoids the systematic underestimation bias of one-shot early stopping. Moreover, it jointly optimizes the ensemble size with other hyperparameters such as tree depth and m_{\mathrm{try}} , recognizing that the required number of trees interacts with the remaining hyperparameters ( Probst et al. 2019 ; Bernard et al. 2009 ) . However, this adaptivity changes the nature of the problem. The central triplet point B_{t} evolves across HPO trials according to random OOB score comparisons. This randomness has several sources. First, the OOB scores are computed from finite data and therefore inherit sampling variability. Second,