Preference-Guided Diffusion for Multi-Objective Offline Optimization

Yashas Annadani, Syrine Belakaria, Stefano Ermon, Stefan Bauer, Barbara Engelhardt · 2025

Read Preference-Guided Diffusion for Multi-Objective Offline Optimization by Yashas Annadani, Syrine Belakaria, Stefano Ermon, Stefan Bauer, Barbara Engelhardt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17299.txt.

Introduction

Yashas Annadani and four co-authors tackle design problems with two goals at once in a 2025 paper. Offline tuning must work from a fixed pile of past designs and their scores, with no fresh trials allowed. The best answers form a front of trade-offs, each fine on one goal and weaker on the other.

A diffusion model dreams up new designs, steered by a judge trained on which of two designs beats the other. The judge still reads well past its training pile, so the search can reach fronts no past design touched. A diversity term keeps the front spread out, and tests beat older methods.

Catalog record

CREATORYashas Annadani, Syrine Belakaria, Stefano Ermon, Stefan Bauer, Barbara Engelhardt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Preference-Guided Diffusion for Multi-Objective Offline Optimization Yashas Annadani, Syrine Belakaria, Stefano Ermon, Stefan Bauer, Barbara Engelhardt Abstract Offline multi-objective optimization aims to identify Pareto-optimal solutions given a dataset of designs and their objective values. In this work, we propose a preference-guided diffusion model that generates Pareto-optimal designs by leveraging a classifier-based guidance mechanism. Our guidance classifier is a preference model trained to predict the probability that one design dominates another, directing the diffusion model toward optimal regions of the design space. Crucially, this preference model generalizes beyond the training distribution, enabling the discovery of Pareto-optimal solutions outside the observed dataset. We introduce a novel diversity-aware preference guidance, augmenting Pareto dominance preference with diversity criteria. This ensures that generated solutions are optimal and well-distributed across the objective space, a capability absent in prior generative methods for offline multi-objective optimization. We evaluate our approach on various continuous offline multi-objective optimization tasks and find that it consistently outperforms other inverse/generative approaches while remaining competitive with forward/ surrogate-based optimization methods. Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions that approximate the Pareto front well. 1 Introduction Several design problems in science and engineering require optimizing a black-box, expensive-to-evaluate function. For example, in antibiotic drug discovery, the goal is to identify novel molecules with high antibacterial activity [ 39 ] . This can be formulated as a single-objective optimization problem. However, in practice, most real-world design challenges involve balancing multiple conflicting objectives. For example, in drug discovery, in addition to maximizing antibacterial activity, we also want to minimize toxicity and production costs [ 38 ] . This constitutes a multi-objective experimental design problem. Prior work in both single and multi-objective optimization (MOO) has largely focused on adaptive experimental design using online methods such as Bayesian optimization [ 33 ] . These approaches rely on training surrogate models for each objective function and designing acquisition functions that are typically optimized via gradient-based techniques [ 3 ] or evolutionary algorithms to determine the next candidate for evaluation. This process is iteratively repeated to optimize the objectives. However, in many real-world applications, sequential evaluations—where inputs are tested one at a time or in small batches—are impractical. In some cases, we have only a single opportunity to evaluate the function, and we must allocate the entire evaluation budget efficiently. For example, in drug design, scientists cannot test molecules one by one in wet lab experiments due to the high cost, slow turnaround, and the inherently parallelizable nature of the process [ 38 ] . Instead, it is common to evaluate all candidate molecules in a single batch. This setting is referred to as offline black-box optimization [ 42 ] . While recent work has explored offline optimization in the single-objective setting [ 46 , 23 ] , where extensive prior data is leveraged to model the objective function and identify potential optima, the MO case remains relatively underexplored. Offline black-box optimization presents distinct challenges compared to traditional online optimization. Since the algorithm cannot iteratively refine the learned model using newly acquired data, it must effectively leverage the available dataset to generalize beyond observed data points. This is particularly challenging because the true optima are often expected to lie outside the existing dataset, requiring robust extrapolation. Additionally, the goal is not merely to identify data points with high function values but to find solutions that satisfy a well-defined notion of optimality, such as the Pareto optimality. Prior work in single-objective offline optimization can generally be categorized into two main approaches: forward and inverse methods. Forward approaches attempt to mimic strategies used in online optimization while leveraging offline data [ 42 ] . These methods train a surrogate model of the objective function and optimize it using gradient-based techniques to propose a set of promising inputs for evaluation. They are effective when the search space is well-defined. In contrast, inverse approaches use generative models to learn an inverse mapping from function values to inputs, enabling the generation of new candidates with potentially high objective values [ 25 , 6 , 16 , 24 ] . This distinction is critical when optimal inputs are not known in advance. For example, in chemistry, if the goal is to evaluate a known molecule, surrogate models are effective in predicting its properties. However, if the goal is to discover entirely new molecules with desired properties, inverse methods are essential, as they directly generate novel candidates rather than selecting from a predefined space. Some generative modeling approaches also draw inspiration from online methods by constructing synthetic optimization trajectories from offline data, aiming to generate new optimal points by extrapolating from the learned trajectories [ 23 ] . Multi-objective offline optimization introduces additional challenges beyond those encountered in the single-objective setting. In single-objective optimization, the goal is simply to maximize (or minimize) a function value. However, in the multi-objective case, we seek to achieve best trade-offs among competing objectives, which is formalized by Pareto optimality. Beyond identifying Pareto-optimal solutions, another critical challenge is ensuring diversity on the Pareto front. A well-structured Pareto front should provide solutions spread across different regions of the objective space, representing a broad range of Pareto-optimal designs. Although the problem of ensuring diversity of solutions is addressed in online MOO [ 1 ] , ensuring both optimality and diversity is challenging in the offline setting as the algorithm must infer these solutions solely from existing data. Xue et al. [45] has recently explored benchmarking offline multi-objective optimization (MOO) by building offline datasets for a variety of MOO benchmarks and proposing several potential algorithms. Their work extends some offline single-objective optimization (SOO) approaches to the MOO setting, such as fitting surrogate models to the offline data and optimizing over the surrogates using evolutionary algorithms. However, this work has not explored the possibility of using generative models for offline MOO. Our contributions are: • We formulate offline MOO as an inverse problem, and propose a novel algorithm based on diffusion models with preference-based classifier guidance. This classifier is trained to compare two candidate designs and determine which is more likely to dominate the other in the objective space. • We address the diversity challenge in the offline MOO setting. Rather than just identifying high-value designs, we train the preference model to favor solutions that are not only optimal but also well-distributed across the objective space. • We demonstrate through extensive experiments on both synthetic and real-world settings that our approach performs better than other inverse methods in terms of convergence to the Pareto front and the diversity of obtained solutions. • Among inverse techniques, our approach consistently achieves the strongest results; making it a competitive choice in cases where forward methods are not feasible. Even when forward methods are applicable, our algorithm remains competitive while maintaining the benefits of generative exploration. Figure 1 : A schematic representation of the proposed preference guided diffusion approach for offline multi-objective optimization. 2 Background 2.1 Offline Multi-Objective Optimization Multi-objective optimization (MOO) seeks to find the design {\bm{x}}\in{\mathcal{X}}^{d} that minimizes (or maximizes) a set of m different objectives: \min_{{\bm{x}}\in{\mathcal{X}}^{d}}{\bm{f}}({\bm{x}})\coloneqq\{f_{1}({\bm{x}}),\dots,f_{m}({\bm{x}})\}, (1) where f_{i}:{\mathcal{X}}^{d}\mapsto{\mathbb{R}} is an unknown and expensive-to-evaluate objective. For most practical problems, the objectives are not simultaneously optimizable by a single design. Hence,

Read the complete text