Inference-Time Alignment of Diffusion Models with Evolutionary Algorithms
Purvish Jajal*, Nick John Eliopoulos*, Benjamin Shiue-Hal Chou, George K. Thiruvathukal, James C. Davis, Yung-Hsiang Lu · 2025
Read Inference-Time Alignment of Diffusion Models with Evolutionary Algorithms by Purvish Jajal*, Nick John Eliopoulos*, Benjamin Shiue-Hal Chou, George K. Thiruvathukal, James C. Davis, Yung-Hsiang Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00299.txt.
Introduction
Purvish Jajal and co-authors present a way in a 2025 paper to steer image-making models at run time. Diffusion models draw state-of-the-art images, yet their outputs often miss goals such as safety rules or domain fit. Older steering methods need inner access to the model or heavy computing.
The team treats the model as a black box and searches its input space with an evolutionary method, keeping outputs that score best on the goal. In tests it raised a standard image quality score by three to thirty-five percent over rival methods in the same time. It also used far less computing, which widens who can apply it.
Catalog record
| CREATOR | Purvish Jajal*, Nick John Eliopoulos*, Benjamin Shiue-Hal Chou, George K. Thiruvathukal, James C. Davis, Yung-Hsiang Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00299
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inference-Time Alignment of Diffusion Models with Evolutionary Algorithms Purvish Jajal*, Nick John Eliopoulos*, Benjamin Shiue-Hal Chou, George K. Thiruvathukal, James C. Davis, Yung-Hsiang Lu Abstract Diffusion models are state-of-the-art generative models in various domains, yet their samples often fail to satisfy downstream objectives such as safety constraints or domain-specific validity. Existing techniques for alignment require gradients, internal model access, or large computational budgets. We introduce an inference - time alignment framework based on evolutionary algorithms. We treat diffusion models as black-boxes and search their latent space to maximize alignment objectives. Our method enables efficient inference-time alignment for both differentiable and non-differentiable alignment objectives across a range of diffusion models. On the DrawBench and Open Image Preferences benchmark, our EA methods outperform state - of - the - art gradient - based and gradient - free inference - time methods. In terms of memory consumption, we require 55 % 55\% to 76 % 76\% lower GPU memory than gradient-based methods. In terms of running-time, we are 72 % 72\% to 80 % 80\% faster than gradient-based methods. We achieve higher alignment scores over 50 optimization steps on Open Image Preferences than gradient-based and gradient-free methods. † † footnotetext: * Purvish Jajal and Nick John Eliopoulos contributed equally to this work. 1 Introduction Figure 1 : Samples generated by our method on Stable Diffusion-3 across three different prompts under various alignment objectives (brightness, saturation, file size). Each row shows the progression over optimization steps i , with the corresponding metric values displayed in the top-right corner. The noise in each generation is optimized via the CoSyNE algorithm [ 13 ] over 14 optimization steps. Arrows ( \uparrow / \downarrow ) indicate whether the metric is being maximized or minimized. Diffusion models [ 40 , 34 , 9 ] have emerged as state-of-the-art generative models for synthesizing high-quality images, video, and audio [ 3 , 16 , 18 , 25 ] . However, the outputs of diffusion models often fail to meet downstream objectives, such as adhering to user preferences [ 53 , 51 , 37 ] , safety constraints [ 28 ] , or domain-specific validity [ 14 , 49 ] . This problem is referred to as alignment —ensuring that generated samples satisfy external objectives beyond the original maximum-likelihood objective [ 24 , 48 , 2 , 21 , 5 , 31 , 4 ] . Alignment is an important concern for many practical applications such as image generation [ 53 , 51 , 37 ] and drug discovery [ 14 , 49 ] . Methods for diffusion model alignment can be categorized into two types: fine-tuning-based alignment and inference-time alignment. Fine-tuning-based alignment typically relies on supervised fine-tuning or reinforcement learning to adjust model parameters [ 24 ] . In contrast, inference-time alignment modifies the sampling process during generation to maximize a reward function ( i.e. , alignment objective) [ 45 ] . These methods aim to be computationally efficient and support arbitrary rewards while maintaining sample quality. However, existing approaches have several drawbacks: they are often not black-box, requiring access to gradients or internal model states; they cannot efficiently generate many aligned samples; and they are often not sample-efficient. As a result, they are difficult to integrate with a variety of models, sampling strategies, and reward functions; generating many aligned samples becomes cost-prohibitive; and obtaining aligned samples may take too long. In this work, we demonstrate that evolutionary algorithms (EAs)—specifically genetic algorithms (GA) and evolutionary strategies (ES)—can be used to align diffusion models. As a class of algorithms, EAs have desirable properties: they are black-box, gradient-free optimizers, that can be highly parallelized [ 39 , 36 ] . By interpreting the diffusion models as black-boxes and searching over their latent space by using EAs we can optimize the alignment objective. Our approach enables efficient alignment that supports both differentiable and non-differentiable rewards, and is compatible with various diffusion models. Fig. 1 depicts our method applied to various alignment objectives. To quantify alignment quality, GPU - memory footprint, and runtime efficiency, we evaluate our inference-time alignment method on two prompt collections [ 35 , 6 ] , four reward functions [ 53 , 51 , 32 , 4 ] , and three diffusion models [ 34 , 9 , 3 ] . On both prompt collections, our method outperforms existing gradient-free and gradient-based inference-time alignment approaches. In terms of computational costs, we exhibit 55\% to 76\% reduction in GPU memory than gradient-based methods, while only consuming up to 1.5\times more memory than gradient-free methods. When producing the same number of aligned samples, evolutionary methods achieve 3.5\times to 5.5\times lower runtime per optimization step compared to gradient-based methods, and only 1.1\times to 1.6\times higher running time per step than gradient-free methods. Moreover, evolutionary methods reach higher alignment scores than gradient-free and gradient-based methods in 10 optimization steps. In sum, we contribute : • We propose a novel inference-time alignment approach using evolutionary algorithms (EAs). Specifically, we introduce two black-box methods for aligning the outputs of diffusion models: (1) optimizing noise, and (2) optimizing transformations applied to the noise. We elaborate upon the requisite choices for utilizing EAs for inference-time alignment, such as search algorithms, initialization, operators, and computational considerations. • We demonstrate the effectiveness of two widely used classes of evolutionary algorithms on the diffusion model alignment task: genetic algorithms and evolutionary strategies. Our evaluation spans a diverse set of reward functions and diffusion models. 2 Related Work In this section, we review prior work on diffusion model alignment ( Section 2.1 ) and evolutionary algorithms ( Section 2.2 ). An extended treatment of related works can be found in Section B.1 . 2.1 Diffusion Model Alignment Diffusion models use a reverse diffusion process to convert some latent noise distribution into a data distribution, such as images [ 40 , 16 ] . The reverse diffusion process iteratively denoises the initial latent noise z_{T} over some number of steps T to yield a sample, z_{0} . Though diffusion models are capable of modeling complex data distributions, they often fail to produce samples that meet some downstream objective. Diffusion model alignment methods adjust diffusion models such that the resulting samples better meet an objective beyond the model’s original maximum likelihood criterion. These objectives commonly focus on producing images that reflect human aesthetic preferences [ 17 , 51 , 52 , 53 ] , or other metrics such as compressibility [ 2 ] . Alignment methods can generally be grouped into two main categories: fine-tuning–based methods and inference-time methods. Fine-tuning-based alignment methods involve adjusting the diffusion model’s parameters so that generated samples better match alignment objectives [ 21 , 5 , 31 , 4 , 48 , 2 ] . Fine-tuning based methods, although powerful, require retraining and thus all the associated costs. We follow the alternative approach, inference-time methods . Rather than altering model parameters, these techniques adjust sampling or conditioning to ensure samples meet alignment objectives. Formally, they seek the control variable \psi that maximizes the expected reward R(x) of samples x from a pretrained diffusion model p_{\theta} , as shown in Eq. 1 . \psi^{\star}=\underset{\psi\in\Psi}{\arg\max}\;\mathbb{E}_{x\sim p_{\theta}(x\mid\psi)}\bigl[R(x)\bigr]. (1) Examples of control variables, \psi , include optimizing conditional input prompts [ 15 ,