OPBO: Order-Preserving Bayesian Optimization
Wei Peng, Jianchen Hu*, Kang Liu, Qiaozhu Zhai · 2025
Read OPBO: Order-Preserving Bayesian Optimization by Wei Peng, Jianchen Hu*, Kang Liu, Qiaozhu Zhai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18980.txt.
Introduction
Wei Peng, Jianchen Hu, Kang Liu, and Qiaozhu Zhai fix Bayesian optimization where it hurts most: high dimensions. Optimizing an expensive black box, like a simulation or a drug assay, usually relies on a Gaussian process surrogate, which collapses past a few hundred variables. Their method, OPBO, posted to arXiv in December 2025, keeps only what matters, the ranking of candidates, and learns it with a small neural network instead of fitting exact values.
Rather than hunting the single best point from an acquisition model, OPBO picks good-enough solutions from an ordinal set, cutting computation further. On problems past 500 dimensions it significantly beats both Gaussian-process and regression-network baselines, and the code is public. For engineers tuning vast design spaces, from chips to materials, an optimizer that survives high dimensions widens what is searchable at all.
Catalog record
| CREATOR | Wei Peng, Jianchen Hu*, Kang Liu, Qiaozhu Zhai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-18980
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OPBO: Order-Preserving Bayesian Optimization Wei Peng, Jianchen Hu*, Kang Liu, Qiaozhu Zhai Abstract Bayesian optimization is an effective method for solving expensive black-box optimization problems. Most existing methods use Gaussian processes (GP) as the surrogate model for approximating the black-box objective function, it is well-known that it can fail in high-dimensional space (e.g., dimension over 500). We argue that the reliance of GP on precise numerical fitting is fundamentally ill-suited in high-dimensional space, where it leads to prohibitive computational complexity. In order to address this, we propose a simple order-preserving Bayesian optimization (OPBO) method, where the surrogate model preserves the order , instead of the value , of the black-box objective function. Then we can use a simple but effective OP neural network (NN) to replace GP as the surrogate model. Moreover, instead of searching for the best solution from the acquisition model, we select good-enough solutions in the ordinal set to reduce computational cost. The experimental results show that for high-dimensional (over 500) black-box optimization problems, the proposed OPBO significantly outperforms traditional BO methods based on regression NN and GP. The source code is available at https://github.com/pengwei222/OPBO . 1 Introduction Black-box optimization refers to the problem of searching for an input so that the output of a black-box system is optimized, the design can only rely on past input and output data collected from stimulating the system. In such problems, the objective function is considered a black-box and we have no access to its structure or gradient information. In reality, it is usually extremely expensive to collect rich enough data to describe the high-dimensional black-box objective function. For example, the design space exploration problem in chip design requires a timely and good enough parameter configuration with only limited tested data given (the data can be unreliable since they are collected from a cycle-accurate simulator) [ 1 ] . The given dataset is only a small portion in the whole high-dimensional design space. Therefore, the main goal of black-box optimization is to find a solution close to the global optimum efficiently. The Bayesian optimization (BO) techniques for black-box functions have been developed in machine learning [ 2 , 3 ] , with applications including hyperparameter tuning in deep learning [ 2 ] , policy search in reinforcement learning [ 4 ] . BO has proven to be a well-established approach for black-box optimization [ 5 , 6 ] . It iteratively implement the following two steps: 1) build a surrogate model to approximate the unknown objective function; 2) use an acquisition function to balance exploration and exploitation, guiding the selection of the next evaluation point. Gaussian processes (GP) model is widely used as a surrogate model in BO, due to its ability to provide accurate prediction and quantification of the model uncertainty. While GP performs well in low-dimensional problems (e.g., dimension less than 20), it often scale poorly to high-dimensional problems [ 5 ] . This problem arises due to several reasons. First, its reliance on precise numerical relationships for curve fitting becomes a bottleneck. In high-dimensional space, the curse of dimensionality renders data points exceedingly sparse, making accurate interpolation or pointwise fitting extremely difficult and prone to overfitting noise. Second, its computational complexity is dominated by kernel matrix construction. This cubic scaling with data dimensionality creates a severe computational bottleneck for high-dimensional problems. The modeling limitations and computational complexity of GP in high-dimensional space limits its practical application. In contrast, neural network (NN) exhibit stronger adaptability when handling high-dimensional data [ 7 , 8 ] . On one hand, NN possesses superior representational capacity for modeling high-dimensional function. On the other hand, unlike GP, the computational complexity of NN scales almost linearly with data size. Previous studies have attempted to apply deep neural networks (DNN) to optimization tasks, such as black-box optimization in continuous search space [ 9 ] and contextual bandit problems in discrete space [ 10 ] . However, most existing methods are based on Bayesian neural networks (BNN), whose posterior inference relies on computationally expensive sampling techniques [ 11 ] . More importantly, neither BNN nor DNN resolve the problem of precise numerical fitting in high-dimensional data. This leads to notable limitations in modeling. In order to overcome the modeling difficulty in high-dimensional space, we note that when selecting the next sampling point via an acquisition function in BO, we care more about the relative rank quality rather than the exact value of the candidate solution. This point aligns closely with ranking neural networks (RNN) in [ 12 ] . By directly modeling preference relationships between samples through comparative learning, the RNN becomes insensitive to the absolute scale of function value and it can focus on capturing the relative orders instead. This characteristic makes RNN particularly suitable for high-dimensional optimization problems. RNN is commonly used in areas such as recommendation systems and information retrieval, with the core task of generating relative rankings of items rather than predicting absolute scores. Another related application work preserving the candidate order is [ 1 ] , which utilizes a transformation technique to preserve the order of the black-box function through a linear surrogate model for solving the design space exploration problem in processor design. Motivated by [ 12 ] and [ 1 ] , we propose a general order-preserving (OP) surrogate model for BO in this work. The proposed model directly learns the ordinal relationships between samples, rather than their precise numerical values. This shift from value to order yields a representation that is more robust to uncertainty and noise. Furthermore, we relax the objective of the acquisition model from finding the optimal solution to selecting the good-enough solutions. This approach prioritizes practical acceptability over theoretical optimality. We conducted systematic experiments on benchmark high-dimensional (with dimension over 500) black-box problems. The results demonstrate that the proposed OP surrogate model significantly outperforms GP and regression NN in performance. 2 Related Work BO is a key technique for solving black-box optimization problems [ 5 , 6 ] . The most commonly used surrogate model for BO is GP due to its flexibility in representing uncertainty and well-calibrated posterior distributions. The efficiency of the optimization process largely depends on the choice of acquisition function, such as Expected Improvement (EI) [ 13 ] and Upper Confidence Bound (UCB) [ 14 ] . BO has achieved significant success in low-dimensional (with dimension less than 20) problems with limited sample budgets. However, Standard GP-based BO encounters notable challenges in high-dimensional space, particularly when dealing with non-stationary/noisy functions or scaling to large numbers of observations. On one hand, the number of observed points becomes sparse in high-dimensional space, making it difficult to accurately estimate the posterior distribution, which leads to a notable decline in the performance of both the surrogate and acquisition models. On the other hand, high dimensios exacerbates the increase in cumulative regret (i.e., the accumulated gap between the observed function values and the global optimum), resulting in overall performance degradation. In order to overcome the difficulties of high-dimensional optimization, one mainstream approach based on GP involves the use of random embedding strategy [ 15 , 16 ] . This method leverages the decomposable nature of the objective function, expressing the original high-dimensional function as