MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning
Lam Ngo, Huong Ha, Jeffrey Chan, Hongyu Zhang · 2025
Read MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning by Lam Ngo, Huong Ha, Jeffrey Chan, Hongyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06847.txt.
Introduction
Lam Ngo, Huong Ha, Jeffrey Chan, and Hongyu Zhang widen a search tool for hard design tasks. The work came out in 2025. The tool is Bayesian search, used to tune costly black-box systems. Most forms of it need smooth knobs that turn by small steps. Many real tasks do not fit that mold at all.
Some picks come from a fixed list, like which stuff to use or which build to pick. High counts of knobs make the search worse. Their tool, MOCA-HESP, cuts the space into curved zones. It tries more than one way to turn list picks into numbers. A bandit rule learns on the fly which code works best. The frame is a meta tool, so other search tools can plug in. The team built three such tools on top of it.
Catalog record
| CREATOR | Lam Ngo, Huong Ha, Jeffrey Chan, Hongyu Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06847
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning Lam Ngo, Huong Ha, Jeffrey Chan, Hongyu Zhang Abstract High-dimensional Bayesian Optimization (BO) has attracted significant attention in recent research. However, existing methods have mainly focused on optimizing in continuous domains, while combinatorial (ordinal and categorical) and mixed domains still remain challenging. In this paper, we first propose MOCA-HESP, a novel high-dimensional BO method for combinatorial and mixed variables. The key idea is to leverage the hyper-ellipsoid space partitioning (HESP) technique with different categorical encoders to work with high-dimensional, combinatorial and mixed spaces, while adaptively selecting the optimal encoders for HESP using a multi-armed bandit technique. Our method, MOCA-HESP, is designed as a meta-algorithm such that it can incorporate other combinatorial and mixed BO optimizers to further enhance the optimizers’ performance. Finally, we develop three practical BO methods by integrating MOCA-HESP with state-of-the-art BO optimizers for combinatorial and mixed variables: standard BO, CASMOPOLITAN, and Bounce. Our experimental results on various synthetic and real-world benchmarks show that our methods outperform existing baselines. Our code implementation can be found at https://github.com/LamNgo1/moca-hesp . † † paperid: 4477 1 Introduction In recent years, high-dimensional BO has emerged as a crucial area of research, as many expensive black-box optimization problems can have hundreds of dimensions. Applications of high-dimensional BO include but are not limited to hyperparameter tuning of machine learning models [ 41 , 10 , 48 ] , neural architecture search [ 21 , 40 ] , drug discovery [ 22 , 13 ] , supply chain management [ 19 , 49 ] and optimal system design [ 9 ] . Various research works have tackled the high-dimensional BO problem for continuous spaces [ 10 , 34 , 42 , 50 , 30 , 31 ] . However, in many real-world scenarios, objective functions may include combinatorial (ordinal and/or categorical) and mixed variables [ 19 , 32 , 9 , 22 , 35 ] . This presents significant challenges, as BO methods that are tailored for continuous variables cannot be directly applied to solve problems involving these combinatorial and mixed variables [ 39 , 46 , 35 , 7 ] . There are multiple reasons. First, categorical variables lack a natural ordering or rank, preventing standard kernels from capturing the relationships between categorical variables and other variables, thus degrading the surrogate model’s predictive accuracy. Second, since ordinal and categorical spaces are discrete, gradient-based optimizers are not effective for optimizing the acquisition function. Third, conventional approaches such as using one-hot encoding to transform categorical variables into numerical values are not scalable as they can significantly increase the problem dimensionality, affecting the computational costs. Although there exist research works that tackle the problem of BO over high-dimensional combinatorial and mixed search spaces [ 32 , 39 , 46 , 5 , 35 ] , due to the aforementioned challenges, achieving promising performance for these types of inputs is still an open question. In this paper, we propose a novel high-dimensional BO meta-algorithm for combinatorial and mixed variables. To address high dimensionality, we leverage the hyper-ellipsoid space partitioning technique (HESP) [ 30 ] to define local regions with a high probability of containing the global optimum, within which BO can be performed. We propose a novel HESP technique designed for combinatorial and mixed problems, which integrates various encoding methods to transform combinatorial variables into continuous representations and employs a multi-armed bandit (MAB) strategy to adaptively select the most effective encoder. Our proposed method, M eta-Algorithm for High-dimensional O rdinal, CA tegorical and Mixed Bayesian Optimization via H yper- E llipsoid S pace P artitioning (MOCA-HESP), is designed as a meta-algorithm [ 47 , 42 , 30 ] that can incorporate various BO optimizers to solve high-dimensional BO problems with combinatorial and mixed variables. Furthermore, we develop three novel algorithms MOCA-HESP-BO, MOCA-HESP-Casmo and MOCA-HESP-Bounce, corresponding to the cases when we incorporate MOCA-HESP with state-of-the-art BO methods for high-dimensional combinatorial and mixed variables: standard BO, CASMOPOLITAN [ 46 ] , and Bounce [ 35 ] , respectively. For MOCA-HESP-BO, we integrate MOCA-HESP with the standard BO method, using it as the optimizer within MOCA-HESP. For MOCA-HESP-Casmo, we propose a novel technique to integrate CASMOPOLITAN’s core feature - the local region adaptation mechanism - into MOCA-HESP and define local regions satisfied both the Mahalanobis and the Hamming distance criteria. For MOCA-HESP-Bounce, we propose a novel technique to incorporate Bounce’s core feature - the subspace embedding technique - into MOCA-HESP’s local regions, allowing low-dimensional optimization. Notably, the MOCA-HESP meta-algorithm can flexibly adapt different settings, e.g., input data encoding, used by various BO optimizers, making it compatible with existing and future BO optimizers. Our experimental results on various synthetic and real-world combinatorial and mixed benchmark problems show that the MOCA-HESP methods outperform the respective BO optimizers and other baselines, demonstrating the effectiveness and efficiency of MOCA-HESP. We summarize our contributions as follows: • We propose a novel meta-algorithm, namely MOCA-HESP, for solving BO problems involving high-dimensional combinatorial and mixed variables. We develop a novel HESP strategy that leverages categorical encoders and an adaptive encoder selection mechanism to effectively transform combinatorial variables into numerical representations. To the best of our knowledge, MOCA-HESP is the first meta-algorithm for BO in high-dimensional combinatorial and mixed spaces. • We develop three practical methods, MOCA-HESP-BO, MOCA-HESP-Casmo, MOCA-HESP-Bounce, by proposing novel strategies to incorporate MOCA-HESP with state-of-the-art BO methods for high-dimensional combinatorial and mixed variables. • We conduct extensive experiments and analysis on synthetic and real-world combinatorial and mixed benchmark problems against the baselines, validating the efficiency of our proposed algorithm. 2 Related Work Figure 1: Illustration of MOCA-HESP meta-algorithm. (1) The mixed input data is encoded by an adaptively chosen encoder . (2) The hyper-ellipsoid local region, which is derived from the base local region (Eq. ( 3 )) and the distinct features of the BO optimizers, is constructed/updated, (3) From the hyper-ellipsoid’s local region, the BO process (based on the chosen BO optimizes) proposes the next observations in the encoded spaces. (4) A decoder is used to transform the encoded data for function evaluations. High-dimensional BO with Combinatorial and Mixed Variables. One popular approach is to enhance the capacity of surrogate models in modeling objective functions with these types of variables. For example, TPE [ 4 ] and SMAC [ 20 ] propose to use different surrogate models such as Tree Parzen Structure and Random Forest, which can directly model combinatorial and mixed variables. Garrido-Merchán and Hernández-Lobato 2020 Garrido-Merchán and Hernández-Lobato 2020 propose to modify the kernel by rounding the kernel input to better capture flat regions in combinatorial and mixed objective functions. COMBO Oh et al. 2019 tackles the problem by constructing a combinatorial graph, where each vertex represents a possible combination of categorical variables, and proposing a graph kernel to capture the interactions between categorical variables within this graph. Moss et al. 2020 Moss et al. 2020 propose the String kernel, which models the categorical variables as strings, hence performing the optimization in a string space. CoCaBO Ru et al. 2020 introduces the Overlapped kernel, which combines continuous and categorical kernels via a weighted average of addition and multiplication to capture the interaction between continuous and categorical variables, enhancing the modeling