Hyperparameter Optimization via Interacting with Probabilistic Circuits

Jonas Seng, Fabrizio Ventola, Zhongjie Yu, Kristian Kersting · 2025

Read Hyperparameter Optimization via Interacting with Probabilistic Circuits by Jonas Seng, Fabrizio Ventola, Zhongjie Yu, Kristian Kersting (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17804.txt.

Introduction

Jonas Seng and co-authors present a new way to tune the settings of a machine learning model with human input, published in 2025. Tuning, called hyperparameter optimization, is often done by Bayesian methods. Few of those methods let a user feed in beliefs, and the ones that do may not reflect those beliefs well.

Their method uses probabilistic circuits as the stand-in model for the search. These circuits encode a joint distribution over settings and scores, and allow exact inference and sampling. A new selection rule then draws candidate settings directly, with no extra inner search. User beliefs are built into that rule by design. Theory and tests show strong results, beating prior interactive methods.

Catalog record

CREATORJonas Seng, Fabrizio Ventola, Zhongjie Yu, Kristian Kersting
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE151 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17804

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hyperparameter Optimization via Interacting with Probabilistic Circuits Jonas Seng, Fabrizio Ventola, Zhongjie Yu, Kristian Kersting Abstract Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, to date, only a few allow to include human feedback. Existing interactive Bayesian optimization (BO) methods incorporate human beliefs by weighting the acquisition function with a user-defined prior distribution. However, in light of the non-trivial inner optimization of the acquisition function prevalent in BO, such weighting schemes do not always accurately reflect given user beliefs. We introduce a novel BO approach leveraging tractable probabilistic models named probabilistic circuits (PCs) as a surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores. They enable exact conditional inference and sampling. Based on conditional sampling, we construct a novel selection policy that enables an acquisition function-free generation of candidate points (thereby eliminating the need for an additional inner-loop optimization) and ensures that user beliefs are reflected accurately in the selection policy. We provide a theoretical analysis and an extensive empirical evaluation, demonstrating that our method achieves state-of-the-art performance in standard HPO and outperforms interactive BO baselines in interactive HPO. Ap Ap 1 Introduction Hyperparameters crucially influence the performance of machine learning (ML) algorithms and must be set carefully to fully unleash the algorithm’s potential ( Bergstra and Bengio, 2012 ; Hutter et al., 2013 ; Probst et al., 2019 ) . Hyperparameter optimization (HPO) algorithms ( Bischl et al., 2023 ) aim to efficiently traverse a predefined search space to find good configurations quickly and avoid unpromising regions. Generally, HPO is framed as optimizing an expensive black-box function since the true functional form of the objective is commonly unknown, and the evaluation of hyperparameter configurations is costly, as it requires training ML models several times. Bayesian optimization (BO) methods have proven to be well-suited for HPO since they are sample efficient and converge on good configurations quickly. Typically, BO algorithms approximate the black-box objective with a surrogate model based on observations made during optimization and use the surrogate in an acquisition function to select the next candidate configuration, balancing exploration of the search space and exploitation of the surrogate ( Shahriari et al., 2016 ; Wang et al., 2022 ) . Figure 1: Interactive Bayesian Hyperparameter Optimization. (Left) We devise an interactive BO method by employing PCs as surrogates encoding a joint distribution over hyperparameters and evaluation scores (omitted for clarity). PCs allow users to directly condition the surrogate on their beliefs during tractable candidate generation, thereby reflecting user knowledge accurately. (Right) Accurately reflecting user beliefs is crucial for interactive HPO to fully leverage user knowledge. In contrast to \pi BO and BOPrO, IBO-HPC (our method) precisely reflects the user prior provided for hyperparameter R (resolution). See App. A for details. Although the recent advancements in HPO could facilitate the design and optimization of ML models for non-experts, in most cases, hyperparameters are still tuned manually ( Bouthillier and Varoquaux, 2020 ) . Given that many ML practitioners perform hyperparameter tuning purely based on their knowledge, experience, and intuition, integrating this valuable knowledge to guide HPO algorithms during optimization can substantially foster the search and mitigate its cost. Moreover, it makes HPO more flexible and interactive, bringing it closer to the recently envisioned goal of human-centered AutoML ( Lindauer et al., 2024 ) . For example, in Fig. 1 (Left), three hyperparameters of a CNN are optimized (depth multiplier N , width multiplier W , and resolution R ). During an HPO run, a user might realize that values around N=16 and W=3 yield high-performing models. Hence, a user can guide the HPO algorithm with the obtained knowledge (here, N=16 and W=3 ) without restarting the optimization from scratch. This can considerably increase the convergence speed and quality of the final solution by focusing the optimization on remaining hyperparameters (here R , details in App. A ). Recent works by Souza et al. (2021) and Hvarfner et al. (2022) allow users to infuse knowledge into a BO framework via user-defined prior distributions that are used to reshape the acquisition function according to user beliefs via multiplicative weighting. Although these approaches are valid and principled ways to guide an HPO task, their weighting schemes might not reflect user knowledge accurately. For example, in Fig. 1 (Right), the configurations selected by both BOPrO ( Souza et al., 2021 ) and \pi BO ( Hvarfner et al., 2022 ) during the first 20 iterations of optimization remarkably deviate from the given user prior (see App. A for details). This happens because acquisition functions are essentially black-boxes, often non-convex and hard to optimize ( Wilson et al., 2018 ) . Consequently, it is hard to anticipate the influence of user beliefs on the actual behavior of the BO algorithm. Note that although trivial incorporation of user beliefs, e.g., by simply setting certain hyperparameters when optimizing the acquisition function, can be accurate, the acquisition function remains a black-box and the influence of priors hard to anticipate. To integrate user feedback in HPO more reliably and accurately, we introduce Interactive Bayesian Optimization via Hyperparameter Probabilistic Circuits (IBO-HPC) . This novel BO method relies on probabilistic circuits (PCs) ( Choi et al., 2020 ) as a surrogate model. PCs encode joint distributions over a set of random variables and come with exact and tractable marginalization, conditioning, and sampling. We derive an acquisition function-free, purely data-driven selection policy that suggests new configuration candidates by leveraging PCs’ tractable conditional sampling mechanism. User beliefs can be provided at any time and are reflected accurately by directly conditioning the PC on them during candidate selection (see Fig. 1 ). Our Contributions: (1) We introduce a novel BO method ( IBO-HPC ) that does not require any inner-loop optimization of an acquisition function and enables a direct, accurate, flexible, and targeted incorporation of user knowledge into BO. (2) We formally define a general notion of interactive policy in BO, and show that IBO-HPC conforms to this notion and is guaranteed to reflect user knowledge accurately during optimization. (3) We analyze the convergence of IBO-HPC and show that it converges proportionally to the expected improvement. (4) We provide an extensive empirical evaluation of IBO-HPC showing that it is competitive with strong HPO baselines on standard HPO tasks and outperforms strong interactive baselines in interactive HPO. 2 Background & Related Work This section briefly introduces hyperparameter optimization (HPO) as well as relevant related work, including (interactive) Bayesian optimization (BO) and probabilistic circuits (PCs). HPO is formally defined as follows ( Kohavi and John, 1995 ; Hutter et al., 2019 ) . Definition 1 (Hyperparameter optimization (HPO)) . Given hyperparameters \bm{\mathcal{H}}=\{H_{1},\dots,H_{n}\} with associated domains \mathbf{H}_{1},\dots,\mathbf{H}_{n} , we define a search space \mathbf{\Theta}=\mathbf{H}_{1}\times\dots\times\mathbf{H}_{n} . For a given black-box evaluation function f:\bm{\Theta}\rightarrow\mathbb{R} , hyperparameter optimization aims to solve \bm{\theta}^{*}=\arg\min_{\bm{\theta}\in\bm{\Theta}}f(\bm{\theta}) . BO has effectively solved many practically relevant HPO tasks and will be introduced next. Interactive Bayesian Optimization BO aims to optimize a black-box objective function f:\mathbf{\Theta}\rightarrow\mathbb{R} which is costly to evaluate, i.e., to find the input \bm{\theta^{*}}=\arg\min_{\bm{\theta}\in\mathbf{\Theta}}f(\bm{\theta}) ( Shahriari et al., 2016 ) . BO typically leverages two key ingredients, a probabilistic surrogate model and a selection policy determining the next \bm{\theta}^{\prime} to be evaluated, and

Read the complete text