OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence
Stamatis Mastromichalakis · 2026
Read OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence by Stamatis Mastromichalakis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06613.txt.
Introduction
Mastromichalakis presents OptiRoulette, a meta-optimizer picking update rules during training instead of fixing one. It combines locking, random sampling from a pool, learning-rate scaling during swaps, and swap-out. Built as a torch-ready component packaged for setup, it reports seed results on five suites: SVHN, Tiny ImageNet, Caltech-256, CIFAR-100, and CIFAR-100-C. Against an AdamW baseline, It improves test accuracy on all five suites, with advantage in closing reliability at higher targets. It reaches targets in all runs while the rival reaches none.
Catalog record
| CREATOR | Stamatis Mastromichalakis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35463 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence Stamatis Mastromichalakis Abstract This paper presents OptiRoulette, a stochastic meta-optimizer that selects update rules during training instead of fixing a single optimizer. The method combines warmup optimizer locking, random sampling from an active optimizer pool, compatibility-aware learning-rate scaling during optimizer transitions, and failure-aware pool replacement. OptiRoulette is implemented as a drop-in, torch.optim.Optimizer -compatible component and packaged for pip installation. We report completed 10-seed results on five image-classification suites: CIFAR-100, CIFAR-100-C, SVHN, Tiny ImageNet, and Caltech-256. Against a single-optimizer AdamW baseline, OptiRoulette improves mean test accuracy from 0.6734 to 0.7656 on CIFAR-100 (+9.22 percentage points), 0.2904 to 0.3355 on CIFAR-100-C (+4.52), 0.9667 to 0.9756 on SVHN (+0.89), 0.5669 to 0.6642 on Tiny ImageNet (+9.73), and 0.5946 to 0.6920 on Caltech-256 (+9.74). Its main advantage is convergence reliability at higher targets: it reaches CIFAR-100/CIFAR-100-C 0.75, SVHN 0.96, Tiny ImageNet 0.65, and Caltech-256 0.62 validation accuracy in 10/10 runs, while the AdamW baseline reaches none of these targets within budget. On shared targets, OptiRoulette also reduces time-to-target (e.g., Caltech-256 at 0.59: 25.7 vs 77.0 epochs). Paired-seed deltas are positive on all datasets; CIFAR-100-C test ROC-AUC is the only metric not statistically significant in the current 10-seed study. † † runningauthor: Stamatis Mastromichalakis † † runningtitle: OptiRoulette † † authors: Independent Researcher, tmnetworks.gr [email protected] https://orcid.org/0000-0003-1713-5078 † † affiliation: † † affiliation: Key words: stochastic optimizer selection, LR scaling, optimizer pool, dynamic optimizer switching, convergence acceleration, deep neural network optimization Disclaimer: The OptiRoulette name refers exclusively to a machine-learning optimizer and has no affiliation, sponsorship, or technical relation to roulette manufacturers, casinos, or any physical/software gambling products or services. Author: Stamatis Mastromichalakis Official Repository: GitHub Repository 1 Introduction Optimizer choice remains a first-order determinant of deep-network training efficiency and final quality. In current practice, however, training typically relies on a single fixed optimizer (e.g., SGD-family or Adam-family) across all epochs, despite well-known stage-dependent behavior: adaptive methods often provide strong early progress, while non-adaptive methods may provide stronger late-stage generalization. This stage mismatch motivates optimizer policies that can evolve during training rather than remain static. Prior work has explored several directions toward this goal. One line uses one-way transitions from adaptive to non-adaptive updates, such as SWATS Keskar and Socher 2017 and bounded-adaptive schemes such as AdaBound Luo et al. 2019 . Another line combines update dynamics through wrappers like Lookahead Zhang et al. 2019 . A more recent line explicitly models multi-optimizer interleaving and switching; for example, IOMT ( Chen et al. 2025 ) and DOIT ( Chen et al. 2026 ) use surrogate performance models and acquisition-style selection rules to choose optimizers over training stages. In parallel, meta-learning approaches treat optimizer design itself as a learned problem (e.g., LSTM-based optimizers in Andrychowicz et al. 2016 ). These advances are important, but they also expose a practical tradeoff: richer optimizer-selection policies can improve adaptation, yet they may increase system complexity, introduce additional tuning surfaces, and reduce plug-and-play usability in standard training pipelines. OptiRoulette is designed around a complementary point in this design space: a lightweight stochastic interleaving policy that remains optimizer-native and deployment-oriented, while still exploiting stage-dependent optimizer diversity. Specifically, OptiRoulette is proposed as a standalone optimizer that combines (i) warmup locking, (ii) stochastic epoch-level sampling from an active optimizer pool, (iii) compatibility-aware learning-rate scaling during optimizer transitions, and (iv) failure-aware pool replacement. The optimizer is implemented as a torch.optim.Optimizer -compatible component intended for drop-in usage and packaging. This paper makes four contributions. First, it formalizes the OptiRoulette optimization process as a stochastic optimizer-selection mechanism over an evolving active set. Second, it provides a theory-grounded interpretation of why the warmup-plus-interleaving regime can accelerate convergence in practice. Third, it reports completed 10-seed evidence across five image-classification suites (CIFAR-100, CIFAR-100-C, SVHN, Tiny ImageNet, and Caltech-256), with emphasis on time-to-target convergence, which is the main competitive advantage observed in this study (Section 4.1 ). Fourth, it positions the resulting milestone behavior against current literature and public-archive reporting conventions. 2 OptiRoulette Optimizer Design 2.1 State and Components Let \mathcal{O}=\{o_{1},\dots,o_{K}\} be the set of configured optimizers. At each training epoch e , OptiRoulette maintains: 1. an active set \mathcal{A}_{e}\subseteq\mathcal{O} (from the pool manager), 2. a blocked set \mathcal{B}_{e}\subseteq\mathcal{O} , 3. a current optimizer o_{e}\in\mathcal{A}_{e} . The optimizer state machine is split into phases: 1. Warmup phase : optimizer is locked to a predefined optimizer (SGD in our experiments), 2. Roulette phase : optimizer is sampled randomly from candidates. 2.2 Random Selection Rule Candidate optimizers at epoch e are: \mathcal{C}_{e}=\mathcal{A}_{e}\setminus\mathcal{B}_{e}, (1) with fallback to \mathcal{O}\setminus\mathcal{B}_{e} if \mathcal{C}_{e} is empty. With avoid_repeat=true , the previous optimizer is removed from candidates when possible: \mathcal{C}^{\prime}_{e}=\begin{cases}\mathcal{C}_{e}\setminus\{o_{e-1}\},&\text{if }|\mathcal{C}_{e}|>1,\\ \mathcal{C}_{e},&\text{otherwise}.\end{cases} (2) The selected optimizer is sampled uniformly: o_{e}\sim\mathrm{Unif}(\mathcal{C}^{\prime}_{e}). (3) In the reported experiments, switching granularity is epoch ; therefore one optimizer is selected per epoch (except during warmup lock), and all batches in that epoch use the same optimizer. 2.3 Warmup and Phase Transition Warmup can terminate in two ways: 1. fixed epoch cutoff ( warmup_epochs ), 2. plateau criterion on validation accuracy slope. In the completed runs, fixed warmup is used with warmup_epochs=17 , warmup_optimizer=sgd , and dropafter_warmup=true . Hence SGD is forced during early training and then excluded from roulette candidates after warmup. 2.4 Reward and Pool Replacement After each epoch, the evaluation loop computes a reward for the selected optimizer: r_{e}=0.7\cdot(\mathrm{valAcc}_{e}-\mathrm{valAcc}^{\text{last}}_{o_{e}})+0.3\cdot(\mathrm{valAcc}_{e}-\mathrm{valAcc}^{\text{global-last}}), (4) then clips r_{e}\in[-1,1] . Pool replacement is triggered by failure conditions: 1. consecutive low rewards below a threshold, 2. catastrophic validation drop relative to best observed validation accuracy. The replacement active optimizer is selected from backup candidates, optionally using compatibility constraints and preferred swap rules. Learning rates can be adjusted across swaps via compatibility-aware scaling rules. 2.5 Why OptiRoulette Can Converge Faster The observed acceleration can be interpreted as a stage-wise stochastic preconditioning effect. Let g_{t}=\nabla_{\theta}\mathcal{L}(\theta_{t}) and let optimizer o_{t} apply update map \phi_{o_{t}} with internal state s_{t}^{(o_{t})} and base learning rate \eta_{o_{t}} . The OptiRoulette update can be written as \theta_{t+1}=\theta_{t}-\eta_{o_{t}}\,\phi_{o_{t}}(g_{t},s_{t}^{(o_{t})}). (5) Conditioned on selection probabilities \pi_{t}(i)=\Pr[o_{t}=i] over active candidates, the expected update is \mathbb{E}[\Delta\theta_{t}\mid g_{t}]=-\sum_{i\in\mathcal{A}_{t}}\pi_{t}(i)\,\eta_{i}\,\phi_{i}(g_{t},s_{t}^{(i)}), (6) which behaves as a mixture of optimizer-specific descent geometries rather than a single fixed preconditioner. In the reported configuration, this mixture is preceded by an explicit 17-epoch warmup ( warmup_epochs=17 ) with warmup_optimizer=sgd at LR 0.1 , then SGD is removed ( dropafter_warmup=true ). This creates a two-stage dynamic: 1. Fast basin entry : high-step SGD warmup rapidly moves from random initialization to a useful attraction region. 2. Refinement regime : after warmup, selection is over {Nadam, Adam, AdamW, Ranger, Adan, Lion} with substantially smaller base LRs (Nadam/Adam/AdamW: 10^{-3} , Ranger: 7\cdot 10^{-4} , Adan: 1.2\cdot 10^{-4} , Lion: 10^{-5} ), reducing oscillation while preserving optimizer diversity. Relative to warmup LR 0.1 , this induces an immediate post-warmup step-size contraction of approximately 10^{2} – 10^{4} , which helps convert rapid early progress into stable late-stage refinement. Compatibility-aware LR scaling further regularizes transitions: 1. high-to-low-family transition scale: 0.01 ( from_high_to_low ), 2. low-to-high-family transition scale: 10.0 ( from_low_to_high ), 3. special cases: AdaHessian LR override 0.15 , Lion LR cap 3\cdot 10^{-4} . These rules are designed to limit destructive step-size discontinuities across optimizer families and to keep switched dynamics inside a stable