OrthoBO : Orthogonal Bayesian Hyperparameter Optimization
Maresa Schröder, Pascal Janetzky, Michael Klar, Stefan Feuerriegel · 2026
Read OrthoBO : Orthogonal Bayesian Hyperparameter Optimization by Maresa Schröder, Pascal Janetzky, Michael Klar, Stefan Feuerriegel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06454.txt.
Introduction
Maresa Schröder, Pascal Janetzky, Michael Klar, and Stefan Feuerriegel tackle a hidden failure mode in Bayesian optimization, a method for tuning models when each test is expensive. Even with a correct surrogate model, Monte Carlo noise in estimating the acquisition score can flip which candidate looks best, leading to poor choices.
Their remedy is an orthogonal acquisition estimator that subtracts a carefully weighted control term, cutting estimation variance while preserving the target. The full OrthoBO framework adds ensemble surrogates and a log transformation. Theory and experiments show lower variance, steadier rankings, and strong results on tuning neural networks, including fine-tuning tasks.
Catalog record
| CREATOR | Maresa Schröder, Pascal Janetzky, Michael Klar, Stefan Feuerriegel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06454
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OrthoBO : Orthogonal Bayesian Hyperparameter Optimization Maresa Schröder, Pascal Janetzky, Michael Klar, Stefan Feuerriegel Abstract Bayesian optimization is widely used for hyperparameter optimization when model evaluations are expensive; however, noisy acquisition estimates can lead to unstable decisions. We identify acquisition estimation noise as a failure mode that was previously overlooked: even when the surrogate model and acquisition target are correctly specified, finite-sample Monte Carlo error can perturb acquisition values. This can, in turn, flip candidate rankings and lead to suboptimal BO decisions. As a remedy, we aim at variance reduction and propose an orthogonal acquisition estimator that subtracts an optimally weighted score-function control variate, which yields an acquisition residual orthogonal to posterior score directions and which thus reduces Monte Carlo variance. We further introduce OrthoBO : a Bayesian optimization framework that combines our orthogonal acquisition estimator with ensemble surrogates and an outer log transformation. We show theoretically that our estimator preserves the target, leads to variance reduction, and improves pairwise ranking stability. We further verify the theoretical properties of OrthoBO through numerical experiments where our framework reduces acquisition estimation variance, stabilizes candidate rankings, and achieves strong performance. We also demonstrate the downstream utility of OrthoBO in hyperparameter optimization for neural network training and fine-tuning. 1 Introduction Hyperparameter optimization (HPO) is a fundamental component of modern machine learning systems, and has been used across a wide range of domains, including biology [ 5 , 49 ] , chemistry [ 67 ] , manufacturing [ 1 ] , medicine [ 44 ] , and physics [ 57 ] . Because evaluating hyperparameter candidates can require substantial computation, Bayesian optimization (BO) is frequently used to identify optimal hyperparameter configurations under limited evaluation budgets (e.g., 19 , 53 , 55 , 59 , 65 ). BO fits a probabilistic surrogate model of the objective and selects new configurations by maximizing an acquisition function such as expected improvement (EI) [ 34 ] . 1 1 1 For a thorough introduction to BO, see, e.g., 24 , 53 . In our work, we focus on a source of instability that is central to BO decisions but has received comparatively little attention: the estimation of the acquisition value itself . Acquisition functions are often treated as deterministic once a surrogate model has been fitted. In many practical BO pipelines, however, the acquisition value is obtained by marginalizing over uncertain surrogate parameters, aggregating across surrogate models, or using Monte Carlo (MC) approximations (e.g., 10 , 30 , 48 , 55 ). As a result, the quantity used to rank candidate configurations is itself a noisy estimate. This may hurt the performance of BO because BO acts on rankings : even small estimation errors can flip the order of two candidates with similar acquisition values and lead the BO algorithm to evaluate many suboptimal configurations (see Fig. 1 ). At a technical level, we treat the estimation of the acquisition function as a MC estimation problem [ 10 , 29 , 30 , 55 , e.g.,] and propose a novel orthogonalized estimator of the acquisition value that serves as a variance-reduction technique, thus improving robustness to sampling errors and stabilizing acquisition rankings. Our notion of orthogonality stems from the orthogonal ML literature (e.g., 12 , 20 , 35 , 39 ) in that our estimator offers robustness due to variance reduction guarantees Note that our notion of robustness differs from that in existing ‘robust’ regression methods, which aim to protect the surrogate model against corrupted or heavy-tailed observations (e.g., 40 ). Figure 1 : Failure mode in Bayesian HPO: estimation noise in the acquisition value can flip rankings and lead to suboptimal configurations. Example. Consider tuning a classification model on a large-scale image dataset, where each evaluation requires training on many images and assessing performance under an imbalanced class distribution. Two hyperparameter configurations may differ only slightly (e.g., a deeper network with stronger dropout versus a shallower network with weaker regularization). If estimates of the acquisition values are subject to noise, BO may spend a suboptimal configuration simply because it was ranked first due to sampling noise. Our contribution is complementary to existing work on robust BO. Prior methods have mainly focused on improving the surrogate model, for example by accounting for kernel misspecification, hyperparameter uncertainty, unreliable posterior uncertainty, trust-region localization, or ensemble surrogates [ 8 , 11 , 15 , 38 , 45 , 48 , 59 ] . Other work has improved the acquisition formula or its numerical optimization [ 2 ] . However, these approaches do not directly address the variance of the estimated acquisition values that drive BO decisions. In contrast, our method improves the statistical quality of the estimated acquisition values used for candidate selection through a novel variance reduction technique based on orthogonal scores. OrthoBO : We build on our orthogonal acquisition estimator and introduce OrthoBO , a BO framework for robust hyperparameter optimization. OrthoBO uses our orthogonalized acquisition estimation as a core component and combines it with ensemble surrogates 38 , 48 and an outer log transformation [ 2 ] . Here, each component serves a distinct role: our orthogonalization stabilizes acquisition estimation, ensembling addresses structural misspecification and the outer log transformation improves numerical conditioning during acquisition maximization. Our OrthoBO offers several theoretical guarantees. (i) Our proposed orthogonalized acquisition estimator is unbiased for the marginal EI target and achieves lower variance than the corresponding naïve MC estimator ( \rightarrow Theorem 4.1 ). (ii) We further show that the lower acquisition estimation variance improves the stability of pairwise acquisition rankings ( \rightarrow Proposition 4.3 ). Intuitively, this reduces the probability of falsely selecting a candidate based on MC noise, and leads thus to performance improvements in practice. Our main contributions are: (1) We propose a novel, orthogonalized estimator for hyperparameter acquisition, reducing estimation variance without changing the underlying acquisition target. (2) We develop OrthoBO , a BO framework that explicitly targets acquisition estimation noise, structural surrogate misspecification, and numerical instabilities. (3) We provide theoretical guarantees for variance reduction and ranking stability, and demonstrate improved empirical performance on various HPO benchmarks. Importantly, OrthoBO is broadly applicable to essentially all BO problems. 2 Related work 2 2 2 We provide an extended related work in Supplement A . Therein, we also provide a broader review of BO under different types of misspecification and orthogonal ML. BO for hyperparameter tuning. BO is a standard approach for HPO (e.g., 19 , 55 , 65 ), particularly when evaluations of the objective function are expensive. In practice, BO relies on a surrogate model to approximate the objective and guide the search. A common choice for the surrogate model are Gaussian processes (GPs) 27 , 34 , 55 ; other alternatives include tree-based and density-based surrogates such as tree-structured Parzen estimators (TPE) 6 , 7 , 62 . Subsequent work has addressed scaling challenges through multi-fidelity methods, batch BO, and trust-region approaches (e.g., 15 , 17 , 26 , 63 ). However, no method has explicitly studied or addressed acquisition estimation noise as a failure mode. Acquisition functions. Surrogate models are used by acquisition functions, which are heuristics to evaluate the utility of hyperparameter candidates before querying the (expensive) objective function [ 4 ] . The performance of the