Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control.
Dom Huh, Prasant Mohapatra · 2025
Read Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control. by Dom Huh, Prasant Mohapatra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12198.txt.
Introduction
Dom Huh and Prasant Mohapatra study, in a 2025 paper, how to fine-tune diffusion-based controllers so they chase reward more directly. Diffusion planners are expressive decision makers, yet how best to squeeze reward from them inside the decision loop stayed scattered across rival recipes. The authors line up four fine-tuning families side by side.
The four are reinforcement learning alignment, direct preference optimization, supervised fine-tuning, and cascading diffusion. By probing design choices in each, they merge the separate tricks into one unified alignment paradigm. In offline reinforcement learning tests, the unified approach improved results across a rich spread of control tasks. The paper reads as a field guide for practitioners who must pick, and tune, a reward-maximizing diffusion controller.
Catalog record
| CREATOR | Dom Huh, Prasant Mohapatra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12198
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control. Dom Huh, Prasant Mohapatra Abstract Diffusion-based planning, learning, and control methods present a promising branch of powerful and expressive decision-making solutions. Given the growing interest, such methods have undergone numerous refinements over the past years. However, despite these advancements, existing methods are limited in their investigations regarding general methods for reward maximization within the decision-making process. In this work, we study extensions of fine-tuning approaches for control applications. Specifically, we explore extensions and various design choices for four fine-tuning approaches: reward alignment through reinforcement learning, direct preference optimization, supervised fine-tuning, and cascading diffusion. We optimize their usage to merge these independent efforts into one unified paradigm. We show the utility of such propositions in offline RL settings and demonstrate empirical improvements over a rich array of control tasks. 1 Introduction Denoising diffusion probabilistic models (DDPM) have demonstrated incredible successes in numerous domains as one of the de facto standards for generative modeling in continuous domains ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Yang et al., 2024 ) , garnering heightened attention in research communities with the model’s exceptional ability to capture highly complex data distributions and flexibility in constraint-based optimization that can be applied to diverse modalities. A notable direction is their adoption and realizations in decision-making applications ( Zhu et al., 2024 ) , contributing to an evolving class of model-based planning as well as model-free algorithms we will refer to as diffusion-based planning, learning, and control. Despite the appealing promises of DDPMs, the algorithm’s sample generation inefficiencies loom as a notorious and significant concern ( Cao et al., 2024 ) , causing a severe bottleneck as an efficient solution for closed-loop control. Figure 1 : A illustration of the reward maximization of diffusion model for control, visualizing the diffusion process at \tau\shortrightarrow\tau-1 such that z^{\tau-1}_{t} is denoised towards the target distribution x^{0}_{t} but also aligned with maximize the return distribution G_{t} , shown in red. The focus of this work addresses this limitation, proposing a general investigation into reward maximization, or more aptly called reward alignment , directly on the diffusion process for control. In turn, our objective is to reduce the number of samples required to generate to maximize the reward signal. We adopt modern optimization procedures to facilitate a guided and constrained distributional shift biased towards maximal returns. Specifically, we investigate four reward alignment approaches — reward maximization with reinforcement learning, direct preference optimization, supervised fine-tuning, and cascading diffusion — and compare their performance in various control settings, more so in offline RL settings. We evaluate various design choices as well as an unification of the studied alignment approaches on a diverse testbed of challenging control tasks consisting of a variety of complex tasks from benchmarks such as D4RL ( Fu et al., 2020 ) , MetaWorld ( Yu et al., 2021 ) , PushT ( Florence et al., 2022 ) , Relay-Kitchen ( Gupta et al., 2019 ) , and RoboMimic ( Mandlekar et al., 2021 ) as well as a novel evaluation task that enables in-depth analysis and visualization of the various features of the environment, the controller and their interactions. 2 Background Diffusion Models DDPMs ( Ho et al., 2020 ) are commonly known as a class of generative models, which aim to estimate and sample from unknown data distributions X . DDPMs define a Markov chain of forward Gaussian kernels q(x_{t}|x_{t-1}) with a scheduled ( Nichol & Dhariwal, 2021 ) sequence of variance coefficients \{\beta_{t}\in(0,1)\}_{t=1}^{T} that gradually add noise to input data q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathrm{I}) (1) and then learn to reverse the diffusion process by denoising with learnable reverse Gaussian kernels p_{\theta}(z_{t-1}|z_{t}) , parameterized by \theta , to generate data samples z_{0} from a standard Gaussian noise prior z_{T}\sim\mathcal{N}(0,I) . p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\mu_{\theta}(z_{t},t),\Sigma(z_{t},t)) (2) A common practice is to parameterize {\mu}_{\theta}({z}_{t},t) to perform \epsilon- prediction, motivated by its resemblance to the Langevin dynamics, which enables a simplification of the training objective. {\mu}_{\theta}({z}_{t},t)\coloneqq{\frac{1}{\sqrt{\alpha_{t}}}\Big({z}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\epsilon}_{\theta}({z}_{t},t)\Big)} (3) The training objective of DDPM uses the variational lower bounds of the marginal likelihood to minimize the negative log-likelihood ( Kingma et al., 2021 ) , as shown in Equation 4 , where in practice, a simplified yet equivalent objective \mathcal{L}_{\text{DDPM}}(\theta) is commonly used ( Ho et al., 2020 ) . \mathcal{L}_{\text{NLL}}(\theta)=\mathbb{E}[-\log{p_{\theta}(z_{0})}]\leq-\mathbb{E}_{q}[\log\frac{p_{\theta}({z}_{0:T})}{q({x}_{1:T}|{x}_{0})}] (4) \displaystyle\mathcal{L}_{\text{DDPM}}(\theta)=\mathbb{E}_{x_{0}}\Big[\|{\epsilon}_{t}-{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_{t}}{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\epsilon}_{t},t)\|^{2}\Big] (5) where t\sim[0,T] with T is the total number of diffusion steps, \beta_{t}=1-\alpha_{t} , \bar{\alpha}_{t}=\mathop{\Pi}_{i=1}^{t}\alpha_{i} , and \epsilon_{t}\sim\mathcal{N}(0,\mathrm{I}) . A common and useful intuition is that DDPMs learn information regarding a time-dependent vector field u_{t}(z_{t},\theta) that generates a probability density/diffusion path p_{t}(z_{t-1}|z_{t}) from a simple and known distribution (e.g., standard Gaussian) to the unknown target distribution (e.g., data distribution). Several recent improvements and reformulations of this DDPM paradigm have been explored ( Cao et al., 2024 ) , most notably and relevantly, towards efficient sample generation with implicit probabilistic models (DDIM ( Song et al., 2020a ) ), v- prediction ( Salimans & Ho, 2022 ) and rectification ( Liu et al., 2022 ; Wang et al., 2024 ) , conditional guidance ( Dhariwal & Nichol, 2021 ; Ho & Salimans, 2022 ) , model selection ( Rombach et al., 2022 ; Ho et al., 2022 ; Peebles & Xie, 2023 ) , novel DDPM training paradigms ( Fan & Lee, 2023 ; Black et al., 2023 ) and related classes of generative modeling such as score/flow-matching ( Song et al., 2020b ; Lipman et al., 2023 ) . Alignment of Diffusion Paths Training deep learning models, such as a DDPM, is often done in two stages. In the first stage, the foundation model is optimized, where a general underlying data distribution is learned by training on large-scale datasets. In the second stage, this foundation model is fine-tuned to align to a reference distribution (e.g., towards downstream tasks, human preference, or stitching with multiple distributions) through supervised fine-tuning (SFT) ( Ziegler et al., 2019 ) , RL with AI/human feedback (RLAIF/RLHF) ( Bai et al., 2022 ; Ouyang et al., 2022a ) , preference optimization (DPO) ( Rafailov et al., 2024 ; Wallace et al., 2024 ; Xiao et al., 2024 ) , or reward gradients ( Prabhudesai et al., 2023 ; Clark et al., 2023 ) . In this work, we apply and modify these techniques toward DMCs to better align using a scalar feedback signal that defines the reward distribution for sequential decision-making tasks. Diffusion Models for Control (DMC) Like many generative modeling algorithms, the role of DDPMs in decision-making applications has spanned from diffusion-based planning ( Janner et al., 2022 ) to policy learning ( Ren et al., 2024 ) , with significant successes and advancements in online and more prevalently in offline RL settings. Specifically, its utilization has been quite diverse, taking various forms such as diffusion-based policies, value functions, and world models, and these models can either indirectly support the training of RL policies ( Rigter et al., 2023 ) or be more of an active component of the decision-making loop ( Wang et al., 2022 ; Chen et al., 2022 ) . In this work, we extend these prior efforts, specifically Diffuser ( Janner et al., 2022 ) , Diffusion Policy (DP) ( Chi et al.,