Define latent spaces by example: optimisation over the outputs of generative models
Samuel Willis, Alexandru I. Stere, Dragos D. Margineantu, Henry T. Oldroyd, John A. Fozard, Carl Henrik Ek, Henry Moss, Erik Bodin · 2025
Read Define latent spaces by example: optimisation over the outputs of generative models by Samuel Willis, Alexandru I. Stere, Dragos D. Margineantu, Henry T. Oldroyd, John A. Fozard, Carl Henrik Ek, Henry Moss, Erik Bodin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23800.txt.
Introduction
A generative model can pour out samples, but science often needs the best sample, not a typical one. Samuel Willis and 12 co-authors propose O3, a method for optimizing over the outputs of diffusion and flow-matching models. It works in surrogate latent spaces: low-dimensional Euclidean embeddings pulled from the model with no extra training. Standard optimizers can then search those spaces directly, at a chosen dimension.
On image and protein design tasks, surrogate-space search found far higher-scoring samples than plain sampling or searching the original latent space. The method is agnostic to model and optimizer, adds negligible cost, and needs no retraining of the generator.
Catalog record
| CREATOR | Samuel Willis, Alexandru I. Stere, Dragos D. Margineantu, Henry T. Oldroyd, John A. Fozard, Carl Henrik Ek, Henry Moss, Erik Bodin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23800
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Define latent spaces by example: optimisation over the outputs of generative models Samuel Willis, Alexandru I. Stere, Dragos D. Margineantu, Henry T. Oldroyd, John A. Fozard, Carl Henrik Ek, Henry Moss, Erik Bodin Abstract Modern generative AI models such as diffusion and flow matching can sample from rich data distributions, but many downstream tasks — such as experimental design or creative content generation — require a higher level of control than unconstrained sampling. The challenge is to efficiently identify outputs that are both probable under the model and satisfy task-specific constraints. We address this by introducing surrogate latent spaces : non-parametric, low-dimensional Euclidean embeddings that can be extracted from any generative model without additional training. The axes in the Euclidean space can be defined via examples, providing a simple and interpretable approach to define custom latent spaces that both express intended features and are convenient to use in downstream tasks. The representation is Euclidean and has controllable dimensionality, permitting direct application of standard optimisation algorithms to traverse the outputs of generative models. Our approach is architecture-agnostic, incurs almost no additional computational cost, and generalises across modalities, including images, audio, videos, and structured objects like proteins. 1 University of Cambridge 2 Lancaster University 3 Karolinska Institutet 4 Boeing Commercial Airplanes 5 Boeing AI 1 Introduction In recent years, generative models have undergone a step-change in performance. Whereas a decade ago they were often tailor-made and domain-specific, modern sample-based approaches such as diffusion and flow matching provide a general framework. Their strength lies in avoiding explicit modelling of the data-generating process and instead specifying a transformation between data samples and a latent distribution, then learning the expected path, with sampling proceeding by reversing this path. This paper focuses on fine-grained control of the latent variable (“the noise”) to encourage desired output properties. Throughout, the model is assumed to generate deterministically, e.g. be a flow matching model ( Lipman et al. 2022 ) , DDIM ( Song et al. 2020a ) for discrete-time diffusion, or the probability flow ODE ( Song et al. 2020b ) for continuous-time. A common strategy for controlling outputs in generative models is known as Latent Space Optimisation (LSO) ( Kusner et al. 2017 ; Gómez-Bombarelli et al. 2018 ; Lu et al. 2018 ; Luo et al. 2018 ) . The idea is to apply an optimisation algorithm in a latent representation of a generative model to search over the objects the model can generate. LSO is often simpler and more efficient than directly searching in the data space, as the model may encode structural details of the data, resulting in a latent space with simpler structure than the original data manifold. While LSO has been applied successfully in the context of Variational Autoencoders (VAE) ( Gómez-Bombarelli et al. 2018 ; Kusner et al. 2017 ) , there has been limited progress in applying LSO to sample-based models such as diffusion and normalising flows due to two key challenges. Challenge 1 (C1): The latent variable in sample-based models retain the dimensionality of the generated outputs and so efficient exploration will be severely hampered by the curse of dimensionality. Challenge 2 (C2): It was until recently unclear how to safely manipulate the latent variable in these models without leaving the support of the model, yielding unrealistic generations. Moreover, although alternative optimisation algorithms have been designed for sample-based models ( Krishnamoorthy et al. 2023 ; Li et al. 2025 ) , they are sophisticated, model-specific, and do not operate on the latent variable. In this paper we show how the challenges C1 and C2 can be circumvented, allowing a large range of downstream optimisation applications to benefit from the generative performance of sample-based models. In particular, we will address the challenges by proposing by-example-specified latent spaces which let us construct targeted latent spaces with a dimensionality of our choosing without losing generation fidelity, which we illustrate in Figure 1 with a 2-dimensional space constructed from three examples. Our proposed framework builds upon recent work ( Bodin et al. 2024 ) for building low-dimensional subspaces of the latent distribution and has the following key takeaways: 1. We introduce surrogate latent spaces using a coordinate chart from the Euclidean space to the latent subspace. This allows us to form simple-to-use subspaces which are completely customisable by choosing the “seed latents” which defines the space. 2. We show that our surrogate latent spaces — which via a coordinate chart is designed to be approximately Euclidean — are well-suited for popular optimisation methods, including Bayesian Optimisation (BO) and CMA-ES, opening the door for LSO using diffusion and flow matching models. 3. Our approach is general and can be applied to any latent variable model, such as a pre-trained diffusion or flow matching model, and across a range of data modalities (see Figure 2 ). In a key experiment we take a state-of-the-art protein generation framework and show a significant improvement in the number of successful generations, allowing us to generate proteins of a greater length than was previously feasible. “A photo of two astronauts on the moon playing badminton while drinking tea” Seeds defining \mathcal{U}^{2} \bm{x}_{1} Score: 1.31 \bm{x}_{2} Score: 1.61 \bm{x}_{3} Score: 1.64 Highest scoring image in \mathcal{U}^{2} Score: 1.96 u_{1} u_{2} u_{1} u_{2} Figure 1: Surrogate latent spaces . ( bottom left ) Generations associated with a grid over a 2-dimensional surrogate latent space, formed using the latent vectors corresponding to the examples \bm{x}_{1} , \bm{x}_{2} , and \bm{x}_{3} for the FLUX.1-schnell ( Labs 2024 ) rectified flow model. ( bottom right ) The ImageReward score for a target prompt (the objective function) over a dense grid ( 256\times 256 ) of generations from our surrogate space show rich structure that can be exploited by standard optimisers. The example (‘seed’) images were obtained by sampling the model using the target prompt but they fail to follow it; by navigating our surrogate space, we can find images with better alignment. u_{1} u_{2} u_{1} u_{2} u_{1} u_{2} Figure 2: We can build smooth surrogate spaces for any generative model. ( left ) Waveform generations over a grid of a 2D slice of a 7D surrogate space formed from 8 seed latents and the 8256-dimensional StableAudio2.0 text-to-audio generation model ( Evans et al. 2025 ) . ( middle ) The first frames of a similarly constructed grid of videos from the 4,308,480-dimensional HunyuanVideo text-to-video generation model ( Kong et al. 2024 ) . ( right ) A grid of proteins over a 2D surrogate space formed from 3 seed latents corresponding to 3 proteins using RFDiffusion ( Watson et al. 2023 ) . 2 Background Deterministic generation in diffusion and flow matching models . Both diffusion ( Ho et al. 2020 ; Song et al. 2020b ; Song et al. 2020a ) and flow matching ( Lipman et al. 2022 ) models share a key property: the ability to perform deterministic generation , where a latent variable \bm{z}\sim p — where p is the latent distribution — fully specifies the generated data \bm{x} . Moreover, using reverse generation procedures, they allow for a known data object \bm{x} to be reverted to its corresponding latent representation \bm{z} — a process referred to as inversion . Determinism