Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review

Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, Tommaso Biancalani · 2025

Read Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review by Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, Tommaso Biancalani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09685.txt.

Introduction

Masatoshi Uehara and co-authors present a tutorial review from 2025 on steering image generators. Diffusion models make pictures or molecules by clearing noise step by step. Many real tasks need outputs that score high on a goal, like a stable protein shape. Retraining the model for each goal is costly.

Guidance at run time offers a cheaper path: steer the clearing steps toward high scores as the sample forms. The review unifies rival steering methods under one view. All of them mimic an ideal process that pairs the base model with a look-ahead score. The authors add new methods to the family and cover search-based steering as well. Code for a protein design case is shared. Links to steering in language models close the survey.

Catalog record

CREATORMasatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, Tommaso Biancalani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE138 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09685

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, Tommaso Biancalani Abstract This tutorial provides an in-depth guide on inference-time guidance and alignment methods for optimizing downstream reward functions in diffusion models. While diffusion models are renowned for their generative modeling capabilities, practical applications in fields such as biology often require sample generation that maximizes specific metrics (e.g., stability, affinity in proteins, closeness to target structures). In these scenarios, diffusion models can be adapted not only to generate realistic samples but also to explicitly maximize desired measures at inference time without fine-tuning. This tutorial explores the foundational aspects of such inference-time algorithms. We review these methods from a unified perspective, demonstrating that current techniques—such as Sequential Monte Carlo (SMC)-based guidance, value-based imoprtance sampling, and classifier guidance—aim to approximate soft optimal denoising processes (a.k.a. policies in RL) that combine pre-trained denoising processes with value functions serving as look-ahead functions that predict from intermediate states to terminal rewards. Within this framework, we present several novel algorithms not yet covered in the literature. Furthermore, we discuss (1) fine-tuning methods combined with inference-time techniques, (2) inference-time algorithms based on search algorithms, which have received limited attention in current research, and (3) connections between inference-time algorithms in language models and diffusion models. The code of this tutorial on protein design is available at https://github.com/masa-ue/AlignInversePro . Keyword: Diffusion Models, Test-Time Alignment, Reinforcement Learning, Classifier Guidance, Sequential Monte Carlo, Model-Based Optimization, Tree Search, Protein Design Introduction Diffusion models ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Song et al., 2020 ) have demonstrated remarkable success in computer vision, particularly as generative models for continuous domains such as images ( Rombach et al., 2022 ) . This success has been further extended to scientific areas such as the generation of protein 3D structures ( Yim et al., 2023 ; Watson et al., 2023 ; Chu et al., 2024 ; Abramson et al., 2024 ) and small molecule 3D structures ( Xu et al., 2022 ; Jing et al., 2022 ; Corso et al., 2022 ) . Furthermore, recent works ( Shi et al., 2024 ; Sahoo et al., 2024 ; Lou et al., 2023 ) have shown promising results with diffusion over traditional autoregressive models in discrete domains. Building on this progress in natural language processing (NLP), the use of diffusion models has also been explored for generating biological sequences (proteins, RNA, and DNA), which are inherently non-causal, because these sequences fold into complex tertiary (3D) structures ( Campbell et al., 2024 ; Sarkar et al., 2024 ; Winnifrith et al., 2024 ; Wang et al., 2024 ) . Controlled generation is a pivotal topic in the study of diffusion models. In the context of “foundational models”, the process typically begins with training conditional diffusion models on large datasets to generate natural designs (e.g., biologically plausible protein sequences) conditioned on fundamental functionalities. Following this pre-training stage, the focus often shifts to optimizing specific downstream reward functions, commonly referred to as “alignment” problems in AI. By guiding generation to maximize a given reward during inference (e.g., binding affinity or stability in protein sequences), diffusion models can be effectively utilized as robust computational design frameworks. Similarly, conditioning on target properties during inference is treated as a reward maximization task, where rewards are frequently defined using classifiers. Figure 1 : The objective of inference-time techniques is to generate natural designs (e.g., natural images or natural-like protein sequences) with high functionality, without any direct fine-tuning of diffusion models. In this tutorial, we aim to explore inference-time techniques for controlled generation in diffusion models, along with their foundational properties. These techniques aim to seamlessly integrate pre-trained generative models trained on large-scale datasets with reward models, as illustrated in Figure 1 . Specifically, at each generation step in pre-trained diffusion models, certain modifications are introduced to optimize downstream reward functions as summarized in Figure 2 . A significant advantage of these methods is that they don’t require post-training of the diffusion models, which can be computationally demanding. The simplest such approach is best-of-N sampling in Figure 2 a, which involves generating multiple designs (N samples) from a pre-trained diffusion model and selecting the best one based on reward functions (e.g., Nakano et al. (2021) ). However, this method can be highly inefficient when the reward functions are difficult to optimize. More efficient sophisticated strategies include classifier guidance in Figure 2 b and its variants ( Dhariwal and Nichol, 2021 ; Song et al., 2021 ) , sequential Monte Carlo-based methods in Figure 2 c ( Wu et al., 2024 ; Dou and Song, 2024 ; Cardoso et al., 2023 ; Phillips et al., 2024 ) , and value-based sampling methods in Figure 2 d ( Li et al., 2024 ) . (a) Best-of-N (b) Classfier Guidance ( Section 4 , 5 ) (c) SMC-Based Guidance ( Section 3.1 ) (d) Value-Based Importance Sampling ( Section 3.2 ) Figure 2 : Summary of representative inference-time algorithms. Here, we aim to optimize downstream reward functions r:\mathcal{X}\to\mathbb{R} given pre-trained masked diffusion models for sequences. The value function v(\cdot) serves as a look-ahead function, mapping intermediate states to expected future rewards r(\cdot) . Best-of-N is a naïve method that selects the best sample among N generated ones. Derivative-based guidance adds gradients of differentiable value function models during inference, making it a powerful method when we can construct the actual value function models. SMC-Based Guidance and Value-Based Importance Sampling (a.k.a. beam search with value functions) are gradient-free methods that sequentially select favorable intermediate states based on value functions. These methods do not require constructing differentiable value function models, which can often be challenging in molecular design. Before delving into the details of inference-time techniques, we provide a brief overview of this tutorial in the introduction. We begin by emphasizing the advantages of inference-time methods over post-training approaches, which can also enable controlled generation. Next, we outline the key components essential for inference-time controlled generation. Finally, we offer a comprehensive overview of the inference-time techniques covered in this work. (a) The Y-axis represents the mean pLDDT across residues, which is a common computational proxy for stability derived from AlphaFold2, as noted in Dauparas et al. (2022) ; Ye et al. (2024) ; Ahdritz et al. (2024) (b) LAION Aesthetic Predictor V2 ( Schuhmann, 2022 ) serves as a typical proxy for aesthetic scores. For instance, in the figure on the right, the scores correspond to images in pink generated by the pre-trained diffusion model (Stable Diffusion in Podell et al. (2023) ), while the scores correspond to images in blue obtained using inference-time techniques. Figure 3 : Scaling inference time compute via value-based beam search ( Li et al., 2024 ) in Section 3.4 , progressively increasing the tree width (see Figure 2 d). These figures demonstrate that as the computational budget (x-axis) increases, rewards (y-axis) can be optimized more effectively. Inference-Time Techniques vs. Post-Training After pre-trainig, there are two main approaches for controlled generation: inference-time techniques (i.e., without fine-tuning diffusion models) and post-training methods such as RL-based fine-tuning ( Black et al., 2023 ; Fan et al., 2023 ; Clark et al.,

Read the complete text