GPTOpt: Towards Efficient LLM-Based Black-Box Optimization

Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konaković Luković · 2025

Read GPTOpt: Towards Efficient LLM-Based Black-Box Optimization by Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konaković Luković (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25404.txt.

Introduction

Jamison Meindl led seven co-authors in presenting SemanticOpt. Yunsheng Tian, Tony Cui, and Veronika Thost helped. Zhang-Wei Hong, Jie Chen, and Wojciech Matusik joined. Mina Konakovic Lukovic closes the author team. Their 2025 paper tackles expensive black-box tuning. Each test may cost much time or money.

Current optimizers mainly use numeric or categorical readings. Bayesian tuning is a leading example there. They ignore broader domain knowledge from experts. Papers and past tests also go unused. Language models can interpret such semantic information well. Yet even strong models solve tuning unreliably alone. SemanticOpt fine-tunes models on structured tuning trajectories. Natural language context augments those trajectories for training. It jointly uses numeric and semantic evidence. Predictions stay interpretable and aligned with Bayesian surrogates. Their benchmark pairs real problems with semantic information. SemanticOpt beat classic and language model methods on average.

Catalog record

CREATORJamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konaković Luković
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25404

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GPTOpt: Towards Efficient LLM-Based Black-Box Optimization Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konaković Luković Abstract Global optimization of expensive, derivative-free black-box functions demands extreme sample efficiency. Classical methods such as Bayesian Optimization (BO) can be effective, but they often require careful parameter tuning to each application domain. At the same time, Large Language Models (LLMs) have shown broad capabilities, yet state-of-the-art models remain limited in solving continuous black-box optimization tasks. We introduce GPTOpt , an LLM-based optimization method that equips LLMs with continuous black-box optimization capabilities. By fine-tuning large language models on extensive synthetic datasets derived from diverse BO parameterizations, GPTOpt leverages LLM pre-training to generalize across optimization tasks. On a variety of black-box optimization benchmarks, GPTOpt surpasses traditional optimizers, highlighting the capacity of LLMs for advanced numerical reasoning and introducing a flexible framework for global optimization without parameter tuning. 1 Introduction Black-box optimization under tight evaluation budgets is pivotal in many scientific and engineering settings. Since derivatives are often unavailable, classical gradient-based solvers such as Newton’s method or conjugate-gradient are not applicable. Practitioners therefore turn to gradient-free heuristics such as genetic algorithms, simulated annealing, and evolutionary strategies, which often require thousands of evaluations to converge ( Holland 1992 ; Kirkpatrick et al. 1983 ; Hansen & Ostermeier 2001 ) . When each experiment or simulation is slow or costly, such sample counts become prohibitive. Therefore, we focus on improving continuous black-box optimization under strict evaluation constraints. Bayesian optimization (BO) ( Shahriari et al. 2015 ) reduces this burden by fitting a probabilistic surrogate (typically a Gaussian process) and selecting queries via a heuristic acquisition function that balances exploration and exploitation. BO has driven progress in materials discovery ( Erps et al. 2021 ) , molecular design ( Griffiths & Hernández-Lobato 2020 ) , clinical prognosis ( Alaa & Schaar 2018 ) , and hyper-parameter tuning ( Turner et al. 2021 ) . However, BO’s success hinges on hand-chosen kernels, acquisition functions, and hyperparameters whose optimal settings vary across landscapes and are difficult to tune without expert insight or extra evaluations. We show this challenge in Figure 1 , where we show the gap between the performance of individual BO methods and the best of all tested BO methods averaged over multiple synthetic 5D test spaces. We see that selecting the best hyperparameters would greatly improve performance and remains a challenge of using BO in practice. Therefore, there is a need for a low-budget general-purpose global optimizer that can perform well without tuning. Figure 1: Performance comparison of best BO-based method to individual BO methods and SOTA LLM-based optimization over 10 5D synthetic functions. Performance is measured with a normalized regret score where higher is better. The large amounts of data and increasing computing power available today have led to the development of LLMs, which provide extraordinary capabilities over a wide range of tasks. While LLMs have been used for specific optimization tasks ( Yang et al. 2024 ; Lange et al. 2024 ) and assisting BO-based methods ( Liu et al. 2024 ) , their capabilities as standalone global optimizers remain limited. Experiments show that LLMs can solve basic optimization problems, Huang et al. 2024 but even state-of-the-art current LLMs remain far-behind classical methods in black-box optimization. We see these limitations in Figure 1 , as we implemented an LLM-based black-box optimization scheme using Gemini 2.5 Pro ( Comanici et al. 2025 ) . Similarly to Huang et al. 2024 , we find that the native LLM-based method is unable to compete with BO-based methods. We provide further details on this experiment in Appendix C.2.1 . Other works have shown that pretrained transformer-based models can used in a variety of data-driven decision making scenarios, such as machine translation ( Vaswani et al. 2017 ) , regression ( Song et al. 2024b ) , and robotic control ( Chen et al. 2021 ) . Recent approaches have begun exploring using pretrained models for optimization, including both transformer architectures and offline pretraining ( Chen et al. 2022b ; Krishnamoorthy et al. 2022 ; Maraval et al. 2024 ) . However, the full potential of LLMs in optimization remains untapped, in part due to the limited availability of massive high-quality optimization datasets ( Song et al. 2024c ) . Furthermore, demonstrating robust zero-shot generalization on completely unseen problems within low evaluation budgets remains an open challenge, as most pretrained approaches require training on similar data to which the model is tested. Consequently, traditional optimization, such as Bayesian optimization, continues to be the preferred choice in practical applications. Therefore, we aim to combine the power of LLMs with high-quality optimization data to achieve performance and flexibility beyond that of current methods and add new capabilities to LLMs. In this paper, we introduce GPTOpt , a LLM-based general-purpose optimizer for continuous black-box problems up to 10 D. We fine-tune Llama 3.2 3B on a diverse dataset of trajectories generated using various BO variants over a synthetic function space ( Dubey et al. 2024 ) . As a result, GPTOpt demonstrates a robust understanding of optimization dynamics and demonstrates the ability to add black-box optimization capabilities to LLMs. In summary, our contributions are as follows. 1. We develop a diverse synthetic function generator and collect large-scale optimization trajectories on these functions from 10 BO variants to be used as training data. 2. We design a fine-tuning approach that leverages the extensive pre-training of LLMs to add high-quality optimization capabilities. 3. Our evaluation demonstrates that GPTOpt achieves strong zero-shot generalization performance across a range of unseen global optimization benchmarks, surpassing the performance of existing state-of-the-art BO methods, while providing the opportunity for further extensions to our method. 2 Related work Bayesian optimization (BO): BO is a powerful global optimization technique due to its ability to efficiently handle expensive black-box functions by balancing exploration and exploitation with carefully designed acquisition functions, such as Expected Improvement (EI), LogEI ( Ament et al. 2023 ) , Upper Confidence Bound (UCB) and Vizier ( Golovin et al. 2017 ; Song et al. 2024d ) . Designed with different principles of trading-off exploration and exploitation, they suit different types of optimization problems. BO has been made accessible by a collection of open-source libraries and software including Spearmint ( Snoek et al. 2012 ) , BoTorch ( Balandat et al. 2020 ) , AutoOED ( Tian et al. 2021 ) , SMAC3 ( Lindauer et al. 2022 ) , HEBO ( Cowen-Rivers et al. 2022 ) , Openbox ( Jiang et al. 2024 ) , etc. However, it remains a challenge for users to determine the most appropriate configuration for their specific problem setting, as the selection is often heuristic and depends on the problem landscape. Pretrained optimization methods: The idea of learning to optimize has been studied relatively early in continuous gradient-based optimization ( Li & Malik 2016 ; Andrychowicz et al. 2016 ; Chen et al. 2022a ) . Many early works reformulate optimization as a sequence prediction problem, training recurrent neural network (RNN) to predict the next point to evaluate. Yet, gradients need to be provided as additional information and they mostly assume in-domain settings ( Chen et al. 2022a

Read the complete text