Large Scale Multi-Task Bayesian Optimization with Large Language Models
Yimeng Zeng, Natalie Maus, Haydn Thomas Jones, Jeffrey Tao, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez, Ryan Marcus, Osbert Bastani, Jacob R. Gardner · 2025
Read Large Scale Multi-Task Bayesian Optimization with Large Language Models by Yimeng Zeng, Natalie Maus, Haydn Thomas Jones, Jeffrey Tao, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez, Ryan Marcus, Osbert Bastani, Jacob R. Gardner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08131.txt.
Introduction
Yimeng Zeng and nine colleagues present a 2025 paper on making optimization smarter by letting a language model learn from past searches. Bayesian optimization is a workhorse for tuning expensive black-box problems, but transferring lessons across tasks has barely scaled past a few dozen. The authors close the loop: a language model is trained on the best solutions found so far, then proposes starting points for new tasks.
Those searches, in turn, produce fresh training data, and the loop repeats. Tested on database query tuning and antimicrobial peptide design across roughly fifteen hundred tasks, the model's starting guesses steadily improved. In the end its quick answers beat full from-scratch searches while using far fewer evaluations. The authors frame the loop as experience that compounds.
Catalog record
| CREATOR | Yimeng Zeng, Natalie Maus, Haydn Thomas Jones, Jeffrey Tao, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez, Ryan Marcus, Osbert Bastani, Jacob R. Gardner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08131
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Scale Multi-Task Bayesian Optimization with Large Language Models Yimeng Zeng, Natalie Maus, Haydn Thomas Jones, Jeffrey Tao, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez, Ryan Marcus, Osbert Bastani, Jacob R. Gardner Abstract In multi-task Bayesian optimization, the goal is to leverage experience from optimizing existing tasks to improve the efficiency of optimizing new ones. While approaches using multi-task Gaussian processes or deep kernel transfer exist, the performance improvement is marginal when scaling to more than a moderate number of tasks. We introduce a novel approach leveraging large language models (LLMs) to learn from, and improve upon, previous optimization trajectories, scaling to approximately 2000 distinct tasks. Specifically, we propose an iterative framework in which an LLM is fine-tuned using the high quality solutions produced by BayesOpt to generate improved initializations that accelerate convergence for future optimization tasks based on previous search trajectories. We evaluate our method on two distinct domains: database query optimization and antimicrobial peptide design. Results demonstrate that our approach creates a positive feedback loop, where the LLM’s generated initializations gradually improve, leading to better optimization performance. As this feedback loop continues, we find that the LLM is eventually able to generate solutions to new tasks in just a few shots that are better than the solutions produced by “from scratch” by Bayesian optimization while simultaneously requiring significantly fewer oracle calls. 1 Introduction Multi-task optimization seeks to use related, previously solved tasks to accelerate the optimization of new ones. Multi-task optimization appears naturally in a variety of domains where similar problems are encountered repeatedly, such as hyperparameter optimization, material science, database query optimization, and drug design. Formally, suppose we have tasks \{1,2,\ldots,T\} , each associated with its own objective function f_{t}(\mathbf{x}) . In multi-task Bayesian optimization (BO), we seek to find some \mathbf{x}^{*}_{t} such that \mathbf{x}_{t}^{*}=\underset{\mathbf{x}\in\mathcal{X}}{\arg\min}f_{t}(\mathbf{x}), (1) for each task t . We focus on the setting where, for each task, we have collected a dataset D_{t} of observations, and we wish to leverage this data when optimizing unseen test tasks. Multi-task BO methods have predominantly used Gaussian processes (GPs) and/or a variety of shared weight neural network feature extractors to jointly learn correlations across tasks ( Swersky et al. 2013 ; Perrone et al. 2018 ; Patacchiola et al. 2020 ; Hakhamaneshi et al. 2022 ) . A standard approach involves placing a multi-output GP over the input-task space, decomposing the kernel as an input kernel k(\mathbf{x},\mathbf{x}^{\prime}) and a task kernel k(t,t^{\prime}) . Despite their effectiveness, many of these methods—with the notable exception of recent work such as Wang et al. 2024c —tend to saturate in performance after tens of training tasks and do not extract additional optimization performance on new tasks when given hundreds or thousands of related tasks for training. We propose Bayesian Optimization with LLM Transfer ( BOLT ), a straightforward approach to multi-task BO that departs from the framework of building related task information into the BO surrogate model. Instead, as BO completes optimization for training tasks, we fine-tune a large language model (LLM) to, given a task description or context C[f_{t}] , generate solutions for that optimization problem that we can use as strong initialization for BO. This approach creates a self-reinforcing feedback loop: BO generates high-quality solutions that we can leverage to fine-tune the LLM; the fine-tuned LLM, in turn, produces better initializations that improve BO performance. Over time, the LLM learns to directly generate solutions that are highly competitive, enabling top-of- k -samples from the LLM (requiring just a few oracle calls) to outperform full “from scratch” BO runs (requiring a large number of oracle calls). This iterative improvement enables BOLT to scale to and still extract value from thousands of tasks. We validate BOLT on two diverse and challenging domains where many related optimization tasks are available. Application 1: Database query plan optimization. In database query plan optimization, the goal is to optimize execution plans in PostgreSQL to minimize runtime. Efficient query planning is critical for database performance, and traditional optimizers such as PostgreSQL’s built-in query planner rely on hand-crafted heuristics and cost models. We show that an LLM fine-tuned using BOLT can generate superior plans, sometimes outperforming PostgreSQL’s own optimizer in a single-shot setting. Application 2: Antimicrobial peptide (AMP) design. Antimicrobial peptides are small peptides that kill (pathogenic) bacteria. We use a software package, APEX ( Wan et al. 2024 ) , which can predict the minimum inhibitory concentration (MIC) of peptides against various pathogens. In this setting, each task is a distinct extinct peptide used as a template, and the goal is to enhance its antimicrobial activity through a limited number edits. We demonstrate that our approach continues to improve as more tasks are introduced and, in the few-shot setting, can eventually generate peptides that outperform those found through full “from-scratch” BO runs. Our experimental results highlight two key benefits of BOLT . First, unlike many existing multi-task BO methods, LLM-generated initialization continues to improve performance as the number of tasks grows, avoiding the saturation observed in GP-based methods. Impressively, by the end of training, the LLM-generated initializations alone nearly outperform competing approaches. Second, a sufficiently fine-tuned LLM can eventually produce solutions in a few-shot setting that rival or surpass full “from scratch” BO runs. Notably, in the database query optimization setting, the LLM becomes a better one-shot optimizer than PostgreSQL’s built-in query planner for certain query types. Contributions 1. We propose BOLT , a scalable and simple alternative to traditional multi-task BO, leveraging LLMs to generate strong initial solutions for new tasks. BOLT leverages a combination of high quality optimized solutions produced by BO and self augmentation for fine-tuning. 2. We validate BOLT through experiments on two challenging domains: database query optimization and antimicrobial peptide design. Results demonstrate that initializations generated by BOLT continue to improve performance as the number of tasks grows. 3. We provide empirical results demonstrating that, after sufficient fine-tuning, the LLM alone can eventually match or even outperform full “from scratch” Bayesian optimization runs with significantly fewer oracle calls. 2 Background Bayesian optimization (BO). Bayesian Optimization (BO) ( Močkus 1975 ; Snoek et al. 2012 ) is an iterative approach to optimize black-box functions in a sample-efficient manner. On each step of the optimization, a supervised probabilistic surrogate model (usually a Gaussian Process (GP) ( Rasmussen 2003 ) ) is conditioned on all data collected so far. Then, the surrogate model’s predictive posterior distribution p(y\mid\mathbf{x},D) is used to decide what data point(s) should be evaluated next, typically by maximizing some acquisition function , defined with respect to p(y\mid\mathbf{x},D) , which guides the exploration-exploitation trade off. Finally, the selected data points are evaluated on the black-box function and added to the dataset. This iterative process continues until the evaluation budget is reached. Structured optimization via latent space BO. BO has recently been applied to optimizing structured search spaces, such as molecular and amino acid sequences, by leveraging latent space Bayesian optimization. This approach incorporates a variational autoencoder (VAE) to map structured inputs into a continuous latent space, where BO is performed ( Eissman et al. 2018 ; Tripp et al. 2020 ; Grosnit et al. 2021 ; Siivola et al. 2021 ; Stanton et al. 2022 ; Maus et al.