Generative Bayesian Hyperparameter Tuning

Hedibert Lopes, Nick Polson, Vadim Sokolov · 2025

Read Generative Bayesian Hyperparameter Tuning by Hedibert Lopes, Nick Polson, Vadim Sokolov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20051.txt.

Introduction

Hedibert Lopes, Nick Polson, and Vadim Sokolov reframe hyperparameter tuning as a generative problem. Tuning picks regularization strength, model size, and robustness knobs. Cross-validation can cost too much at scale. Full Bayesian learning stalls on the cost of sampling the posterior. Their approach joins two ideas. One is to use randomized, weighted objectives as stand-ins for Bayesian posteriors. The other is to amortize the repeated optimization across many settings.

A learned transport map does the amortizing. It takes hyperparameters, with random weights, straight to the matched optimizer. That forms a generator look-up table for estimators. Grids or smooth ranges of settings can then be scored fast. The table serves both predictive tuning and rough Bayesian uncertainty checks. Links to weighted M-estimation round out the frame.

Catalog record

CREATORHedibert Lopes, Nick Polson, Vadim Sokolov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20051

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Bayesian Hyperparameter Tuning Hedibert Lopes, Nick Polson, Vadim Sokolov Abstract Hyper-parameter selection is a central practical problem in modern machine learning, governing regularization strength, model capacity, and robustness choices. Cross-validation is often computationally prohibitive at scale, while fully Bayesian hyper-parameter learning can be difficult due to the cost of posterior sampling. We develop a generative perspective on hyper-parameter tuning that combines two ideas: (i) optimization-based approximations to Bayesian posteriors via randomized, weighted objectives (weighted Bayesian bootstrap), and (ii) amortization of repeated optimization across many hyper-parameter settings by learning a transport map from hyper-parameters (including random weights) to the corresponding optimizer. This yields a “generator look-up table” for estimators, enabling rapid evaluation over grids or continuous ranges of hyper-parameters and supporting both predictive tuning objectives and approximate Bayesian uncertainty quantification. We connect this viewpoint to weighted M M -estimation, envelope/auxiliary-variable representations that reduce non-quadratic losses to weighted least squares, and recent generative samplers for weighted M M -estimators. 1 Introduction Hyper-parameters such as regularization strengths, robustness/quantile levels, and architectural or optimization settings have an outsized impact on predictive performance and calibration. In practice, these choices are frequently made by cross-validation (CV), despite well-known statistical and computational drawbacks in some regimes ( Stone 1974 ) . Bayesian hyper-parameter learning offers a principled alternative, but can be challenging in modern models because posterior computation is expensive and must typically be repeated for each hyper-parameter configuration. This paper develops a generative viewpoint on hyper-parameter tuning ( Nareklishvili et al. 2024 ; Polson and Sokolov 2025b ; Nareklishvili et al. 2023c ) . We start from optimization formulations of regularized estimation and their MAP interpretations, and then leverage randomized, weighted objectives to obtain scalable approximate posteriors via the weighted Bayesian bootstrap (WBB) ( Newton et al. 2021 ) . The key additional step is amortization: rather than re-solving the optimization problem from scratch for each hyper-parameter value, we learn a transport map (generator) from hyper-parameters and random weights to the corresponding optimizer. This idea is closely related in spirit to hyper-networks and amortized inference, where one learns to predict parameters or approximate posterior objects as a function of context ( Ha et al. 2017 ; Bhadra et al. 2024 ) . Viewed as a hyper-parameter optimization strategy, the learned map can be used to rapidly evaluate predictive criteria over hyper-parameter grids (a drop-in replacement for expensive CV loops) or to support gradient-based outer-loop tuning, complementing black-box Bayesian optimization ( Snoek et al. 2012 ; Bergstra and Bengio 2012 ; Baker et al. 2022 ) and gradient-based/bilevel approaches ( Maclaurin et al. 2015 ; Franceschi et al. 2018 ) . Viewed as approximate Bayesian inference, combining WBB with a generator yields fast sampling of parameter draws across hyper-parameter settings. The remainder of the paper develops the core ingredients and illustrates how envelope/auxiliary-variable representations can turn difficult losses into weighted least-squares inner steps, aligning them with the same computational template as WBB and the generative map. 1.1 Existing literature Two strands of literature are especially close to our unifying perspective. First, Golub et al. 1979 develop generalized cross-validation (GCV) as an analytic, rotation-invariant surrogate for leave-one-out cross-validation in ridge regression and related linear smoother settings. Their key observation is that when fitted values are linear in y through a hat matrix A(\lambda) , one can estimate predictive risk using only residuals and the effective degrees of freedom \mathrm{tr}\{A(\lambda)\} , yielding a tuning rule that avoids explicit refitting across held-out folds and does not require estimating \sigma^{2} . In modern terms, GCV provides an explicit outer criterion U(\cdot) for \lambda in a setting where the inner optimizer map \lambda\mapsto\hat{\theta}(\lambda) is available in closed form. Second, Shin et al. 2023 ; Shin et al. 2024 propose the Generative Multi-purpose Sampler (GMS), which constructs a generator G(w,\lambda,\eta) that outputs solutions of weighted (penalized) M -estimators across bootstrap and cross-validation weight patterns and across tuning parameters. A central contribution is their training objective: instead of learning G by regressing onto precomputed optimizers (which would require many expensive inner solves), they minimize an integrated weighted objective value over draws of (w,\lambda,\eta) , so that generator training and M -estimation co-evolve via a single optimization loop. They also demonstrate how this enables computationally intensive procedures (iterated bootstrap, bootstrapped CV, solution paths) at scale. Our contribution is to place these ideas into a single paradigm suitable for modern models trained by SGD: a randomized weighted inner objective, an outer tuning criterion defined as an expectation over the induced randomness, and an optional transport map/generator to amortize repeated optimization. In this view, GCV is the deterministic linear-smoother special case where both the optimizer map and the outer risk surrogate are analytic, while GMS is the weighted M -estimation special case where the generator is trained by criterion minimization. The rest of the paper develops this unification in the context of WBB-based approximate posteriors and fast uncertainty summaries. Contributions and scope. This paper contributes a unifying template that connects classical analytic tuning (GCV), weighted M -estimation perturbation schemes (bootstrap/CV weights), and generator-based amortization. We provide a concrete algorithm box that makes the mapping (\omega,\lambda,\eta)\mapsto\hat{\theta} operational for SGD-trained models, and we demonstrate its performance in both controlled ridge regression and modern deep-learning settings (MNIST). We show how criterion-based training can avoid expensive optimizer-label generation while still enabling fast evaluation of tuning objectives and uncertainty summaries. Code to reproduce all experiments is available at https://github.com/VadimSokolov/Generative-Bayesian-Hyperparameter-Tuning . 2 Generative Bayesian Hyperparameter Tuning 2.1 A unified method that generalizes GCV and GMS We now state a generic method that subsumes (i) classical analytic tuning rules such as GCV for linear smoothers ( Golub et al. 1979 ) and (ii) modern generator-based weighted M -estimation as in GMS ( Shin et al. 2023 ; Shin et al. 2024 ) . The method has three components: an inner (possibly randomized) training objective, an outer tuning criterion, and an optional transport map (generator) that amortizes repeated solutions of the inner problem. Inner problem (randomized weighted training objective). Let h=(\lambda,\eta) denote a vector of hyper-parameters: \lambda is a regularization strength and \eta collects any additional hyper-parameters that index the data-fit term (e.g. robustness/quantile parameters, temperature parameters, augmentation strengths, etc.). Let \omega\sim\pi(\omega) be random weights (including deterministic \omega\equiv\mathbf{1} as a special case). Define the weighted objective L(\theta;\omega,h)\;=\;\frac{1}{n}\sum_{i=1}^{n}\omega_{i}\,\ell_{\eta}(y_{i}\mid\theta)\;+\;\lambda\,\phi(\theta),\qquad\hat{\theta}(\omega,h)\in\arg\min_{\theta}L(\theta;\omega,h). (1) In neural networks, \hat{\theta}(\omega,h) is computed approximately by SGD on reweighted mini-batches. In convex problems, \hat{\theta}(\omega,h) may be available in closed form. Transport map / generator (amortizing the optimizer). We introduce a parametric map g_{\varphi} intended to approximate the optimizer: g_{\varphi}(\omega,h)\approx\hat{\theta}(\omega,h). This can be trained either (A) by supervised regression to precomputed optimizers \hat{\theta}(\omega^{(b)},h^{(b)}) , or (B) by direct minimization of the criterion value (GMS-style) by plugging g_{\varphi} into L(\cdot;\omega,h) and optimizing \varphi by backpropagation. Outer problem (what “hyper-parameter tuning” means). The outer goal is to choose h to optimize a criterion U that measures out-of-sample quality and/or uncertainty. A general template is \min_{h}\;J(h)\;=\;\mathbb{E}_{\omega\sim\pi(\omega)}\Big[U\big(\hat{\theta}(\omega,h),h\big)\Big]\;\approx\;\frac{1}{M}\sum_{m=1}^{M}U\big(g_{\hat{\varphi}}(\omega^{(m)},h),h\big), (2) where U might be a validation negative log-likelihood, a proper scoring rule, a risk estimate, or a proxy for marginal-likelihood objectives. The approximation on the right is the computational payoff: once g_{\hat{\varphi}} is trained, we can evaluate (and often differentiate) J(h) without repeatedly re-running SGD for each h . Special case

Read the complete text