Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space

Kazuki Ishikawa, Ryota Ozaki, Yohei Kanzaki, Ichiro Takeuchi, Masayuki Karasuyama · 2025

Read Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space by Kazuki Ishikawa, Ryota Ozaki, Yohei Kanzaki, Ichiro Takeuchi, Masayuki Karasuyama (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09329.txt.

Introduction

Picking the best learning algorithm for a job, and tuning its knobs, is a search over a huge space where each candidate keeps its own private set of knobs. Kazuki Ishikawa and four co-authors fold all the knob sets into one shared hidden space in this 2025 paper. A single guide model can then learn from every trial.

Because trials of one algorithm now teach the guide about the others, the search needs far fewer runs. The team also pre-trains the folding with a guard against collapse, and adds a ranking step that picks the best pre-trained folding for a new data set. Tests on open benchmark data show the shared-space search beating methods that keep each algorithm in its own silo. The work targets automated machine learning pipelines.

Catalog record

CREATORKazuki Ishikawa, Ryota Ozaki, Yohei Kanzaki, Ichiro Takeuchi, Masayuki Karasuyama
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09329

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space Kazuki Ishikawa, Ryota Ozaki, Yohei Kanzaki, Ichiro Takeuchi, Masayuki Karasuyama Abstract Selecting the optimal combination of a machine learning (ML) algorithm and its hyper-parameters is crucial for the development of high-performance ML systems. However, since the combination of ML algorithms and hyper-parameters is enormous, the exhaustive validation requires a significant amount of time. Many existing studies use Bayesian optimization (BO) for accelerating the search. On the other hand, a significant difficulty is that, in general, there exists a different hyper-parameter space for each one of candidate ML algorithms. BO-based approaches typically build a surrogate model independently for each hyper-parameter space, by which sufficient observations are required for all candidate ML algorithms. In this study, our proposed method embeds different hyper-parameter spaces into a shared latent space, in which a surrogate multi-task model for BO is estimated. This approach can share information of observations from different ML algorithms by which efficient optimization is expected with a smaller number of total observations. We further propose the pre-training of the latent space embedding with an adversarial regularization, and a ranking model for selecting an effective pre-trained embedding for a given target dataset. Our empirical study demonstrates effectiveness of the proposed method through datasets from OpenML. 1 Introduction In recent years, machine learning (ML) has gained widespread adoption and has been utilized in various fields. However, to achieve high prediction performance, the precise selection of an appropriate ML algorithm and its hyper-parameters (HPs) is indispensable. There exist a large number of possible combinations of ML algorithms and HPs, which makes their efficient selection is challenging. The simultaneous selection of an ML algorithm and HPs is called Combined Algorithm Selection and Hyper-parameter optimization (CASH) problem [ Thornton et al.(2013) , Feurer et al.(2015a) ] . The CASH problem is formulated as the optimization of the prediction performance with respect to the pair of an ML algorithm and its HP setting. To solve a CASH problem, Bayesian optimization (BO) [ Brochu et al.(2010) , Snoek et al.(2012) , Bergstra et al.(2011) ] is often used. BO constructs a probabilistic surrogate model (typically, the Gaussian process) to approximate the objective function in the search space (e.g., to predict the validation accuracy of an ML model in the HP space), by which a decision making considering the current uncertainty of the surrogate model becomes possible. In general, since different ML algorithms have different HP spaces, different surrogate models are usually required to estimate for each one of ML algorithms [ Nguyen et al.(2020) ] . In this approach, each surrogate model is independently learned and information of observations is not shared across ML algorithms. Because of this observation separation, this approach requires sufficient observations for all candidate ML algorithms to build an effective surrogate model. On the other hand, applying BO in the joint (concatenated) space of different HP spaces [ Hutter et al.(2011) ] has been also studied. However, the joint space becomes high dimension and the default value setting is required for ‘non-active’ ML algorithms whose justification is unclear. Further details of related work is discussed in Section 4 . In this study, we propose a BO based CASH solver in which a surrogate model is learned in a shared latent space. The basic idea is to embed observations obtained by different ML algorithms into a common latent space. This approach enables the surrogate model 1) to be constructed by using information of all candidate ML algorithms, and 2) to avoid the difficulties in the joint space approach such as high dimensionality. Since most of HPs control the model flexibility in different manners, we assume that a common latent representation can be created for HP spaces of many different ML algorithms through an appropriate transformation. We further introduce a pre-training framework for the latent space construction so that the search can be stably accelerated even at the early iterations of BO. Figure 1: Overview of Proposed Framework. Figure 1 illustrates the entire framework of the proposed method, which comprises three main components. Each of these components is a major contribution of this paper: • The first component is the BO with a surrogate model in the shared latent space for the CASH optimization, illustrated in Fig. 1 (a). By combining the multi-task Gaussian process [ Álvarez et al.(2012) ] and deep kernel [ Wilson et al.(2016) ] , we build the surrogate model in such a way that observations in different HP spaces can be shared. • The second component is the pre-training of the latent space embedding, illustrated in Fig. 1 (b). To encourage information sharing among different ML algorithms, we introduce domain adversarial learning [ Ganin et al.(2016) ] as a regularizer in the pre-training, which prevents observations from different ML algorithms from being isolated in the latent space. • The pre-training can be performed for a variety of past datasets, called source datasets, beforehand. In the third component, shown as (c) in Fig 1 , we construct a ranking model that recommends the best pre-trained embedding model for a given target dataset by using a technique in learning to rank [ Liu(2011) ] . We empirically demonstrate the effectiveness of the proposed framework through datasets in OpenML [ Vanschoren et al.(2013) ] . 2 Problem Setting We assume that there exist M candidate machine learning (ML) algorithms {\mathcal{A}}=\{A^{(1)},\ldots,A^{(M)}\} , in which the m -th ML algorithm A^{(m)} has a hyper-parameter (HP) space \boldsymbol{\Lambda}^{(m)} . This means that our search space consists of a pair of an ML algorithm A^{(m)} and an HP vector \boldsymbol{\lambda}^{(m)} , denoted as \Xi=\{(A^{(m)},\boldsymbol{\lambda}^{(m)})\mid A^{(m)}\in{\mathcal{A}},\boldsymbol{\lambda}^{(m)}\in\boldsymbol{\Lambda}^{(m)},m\in[M]\} . A given dataset {\mathcal{D}} is partitioned into the training dataset {\mathcal{D}}_{\mathrm{train}} and the validation dataset {\mathcal{D}}_{\mathrm{valid}} , respectively. Suppose that \mathrm{Acc}(A^{(m)},\boldsymbol{\lambda}^{(m)},{\mathcal{D}}_{\mathrm{train}},{\mathcal{D}}_{\mathrm{valid}}) is an evaluation score (such as the validation classification accuracy) on {\mathcal{D}}_{\mathrm{valid}} of an ML algorithm A^{(m)} trained by {\mathcal{D}}_{\mathrm{train}} with an HP \boldsymbol{\lambda}^{(m)} . The optimization problem for identifying the pair (A^{*},\boldsymbol{\lambda}^{*})\in\Xi that maximizes \mathrm{Acc}(A^{(m)},\boldsymbol{\lambda}^{(m)},{\mathcal{D}}_{\mathrm{train}},{\mathcal{D}}_{\mathrm{valid}}) is called Combined Algorithm Selection and Hyperparameter optimization (CASH) problem [ Thornton et al.(2013) , Feurer et al.(2015a) ] : \displaystyle(A^{*},\boldsymbol{\lambda}^{*})=\mathop{\text{argmax}}_{(A^{(m)},\boldsymbol{\lambda}^{(m)})\in\Xi}\mathrm{Acc}(A^{(m)},\boldsymbol{\lambda}^{(m)},{\mathcal{D}}_{\mathrm{train}},{\mathcal{D}}_{\mathrm{valid}}). (1) In general, it is often difficult to analytically represent a relation between \mathrm{Acc} and (A^{(m)},\boldsymbol{\lambda}^{(m)}) , and therefore, this problem is often regarded as a black-box optimization problem. 3 Proposed Method Here, we introduce our proposed framework for efficiently solving ( 1 ). In Section 3.1 , we describe our BO with a latent space surrogate model (Fig. 1 (a)). The pre-training (Fig. 1 (b)) of the latent space embedding is described in Section 3.2 . The ranking model (Fig. 1 (c)) for the selection of a pre-trained model is shown in Section 3.3 . The algorithm of the proposed method is shown in Algorithm 1 , whose details are described throughout this section. Algorithm 1 Proposed Method Require : Initial observations {\mathcal{O}} , Max iteration T , Meta-features \boldsymbol{x}_{s}^{\mathrm{meta}} and pre-trained MLPs for source datasets \{{\mathcal{D}}^{\prime}_{s}\}_{s\in[S]} , Ranking model f_{\mathrm{rank}} 1 Compute meta-feature \boldsymbol{x}^{\mathrm{meta}} for target dataset {\mathcal{D}} 2 Select the best source dataset {\mathcal{D}}^{\prime}_{s_{\rm{best}}} by ranking model f_{\mathrm{rank}} : s_{\mathrm{best}}\leftarrow\mathop{\text{argmax}}_{s\in[S]}f_{\mathrm{rank}}(\boldsymbol{x}^{\mathrm{meta}},\boldsymbol{x}_{s}^{\mathrm{meta}}) \triangleright Recommendation of pre-trained model (Fig. 1 (c)) ​​​​​​ 3 Set \{\phi^{(m)}\}_{m\in[M]} as MLPs trained beforehand by

Read the complete text