Efficient Adjoint Matching for Fine-tuning Diffusion Models
Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu, Wei Guo, Yongxin Chen, Joonseok Lee, Jaewoong Choi, Jaemoo Choi · 2026
Read Efficient Adjoint Matching for Fine-tuning Diffusion Models by Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu, Wei Guo, Yongxin Chen, Joonseok Lee, Jaewoong Choi, Jaemoo Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11480.txt.
Introduction
Shin and seven colleagues propose Efficient Adjoint Matching, a faster way to fine-tune diffusion models with rewards. Adjoint Matching casts reward fine-tuning as stochastic optimal control, but costs heavily in trajectory simulation and backward adjoint simulation. Both costs trace to base drift inherited from the pretrained model. A linear base drift and modified terminal cost enable few-step deterministic sampling and a closed-form adjoint solution. On image benchmarks it converges up to four times faster and matches or beats Adjoint Matching on five metrics.
Catalog record
| CREATOR | Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu, Wei Guo, Yongxin Chen, Joonseok Lee, Jaewoong Choi, Jaemoo Choi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44179 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11480
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Adjoint Matching for Fine-tuning Diffusion Models Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu, Wei Guo, Yongxin Chen, Joonseok Lee, Jaewoong Choi, Jaemoo Choi Abstract Reward fine-tuning has become a common approach for aligning pretrained diffusion and flow models with human preferences in text-to-image generation. Among reward-gradient-based methods, Adjoint Matching (AM) provides a principled formulation by casting reward fine-tuning as a stochastic optimal control (SOC) problem. However, AM inevitably requires a substantial computational cost: it requires (i) stochastic simulation of full generative trajectories under memoryless dynamics, resulting in a large number of function evaluations, and (ii) backward ODE simulation of the adjoint state along each sampled trajectory. In this work, we observe that both bottlenecks are closely tied to the non-trivial base drift inherited from the pretrained model. Motivated by this observation, we propose Efficient Adjoint Matching (EAM) , which substantially improves training efficiency by reformulating the SOC problem with a linear base drift and a correspondingly modified terminal cost . This reformulation removes both sources of inefficiency; it enables training-time sampling with a few-step deterministic ODE solver and yields a closed-form adjoint solution that eliminates backward adjoint simulation. On standard text-to-image reward fine-tuning benchmarks, EAM converges up to 4× faster than AM and matches or surpasses it across various metrics including PickScore, ImageReward, HPSv2.1, CLIPScore and Aesthetics. 1 Introduction Diffusion 16 ; 32 and flow 20 ; 23 ; 1 models have become the standard backbone for large-scale text-to-image (T2I) generation 27 ; 28 ; 8 . While their pretraining objective produces photorealistic samples, it is often poorly aligned with the qualities that actually matter in deployment such as aesthetic quality, prompt fidelity, and adherence to human preferences 37 ; 36 . To close this gap, reward-based fine-tuning has emerged as a standard recipe for aligning text-to-image models, where pretrained diffusion models are post-trained to maximize an external reward function that reflects these preferences 22 ; 6 ; 42 ; 4 ; 5 ; 37 ; 26 . Among existing reward fine-tuning methods, reward-gradient-based methods provide direct supervision by leveraging the gradient of the reward function with respect to its input 5 ; 37 ; 26 ; 6 . This gradient is readily available when the reward is parameterized by a differentiable network, as is typical for learned preference models such as CLIPScore, HPS, Aesthetic score, ImageReward, and PickScore 15 ; 36 ; 29 ; 37 ; 19 . While many reward fine-tuning methods use only scalar reward values 22 ; 42 ; 4 , reward-gradient-based methods additionally use the local gradient direction of the reward model. This provides each generated sample with direct supervision toward higher reward, offering an effective and sample-efficient route when reliable reward gradients are available. Adjoint Matching (AM) 6 introduces a theoretically principled framework for reward-gradient-based fine-tuning. Specifically, AM casts reward fine-tuning as a stochastic optimal control (SOC) problem and learns an additional control that transforms the pretrained model toward the reward-tilted target distribution p^{\star}_{1} , defined as \displaystyle p^{\star}_{1}(x)\propto e^{\beta r(x)}p_{\text{data}}(x), (1) where r(x) is a given reward function, \beta>0 is the constant that controls the strength of reward guidance, and p_{\text{data}} denotes the data distribution. However, AM requires heavy computational cost for optimization ( Section 3.1 ). First, AM relies on forward SDE simulation of full generative trajectories under memoryless dynamics. This SDE simulation requires a large number of function evaluations (NFEs), substantially increasing sampling time during training. Second, AM requires backward ODE simulation for adjoint state along sampled trajectory to propagate reward-gradient signal ( Section 2.3 ). Together, these two simulation procedures dominate the training cost. Our key observation is that these bottlenecks arise from the choice of the drift term of the base dynamics ( Section 3.2 ). Building on this observation, we redesign the base dynamics using a simple linear drift that satisfies the desired conditions. We accordingly modify the terminal cost in the SOC problem so that the optimal terminal distribution matches with p^{\star}_{1} in Equation 1 . This reformulation leads to our algorithm, Efficient Adjoint Matching (EAM) , which removes both costly simulations in AM in the following way. First, forward trajectory simulation becomes solver-agnostic: the endpoint image X_{1} can be generated using any efficient few-step deterministic ODE solver, and intermediate states X_{t} can be sampled from the original noising kernel q_{t}(\cdot\mid X_{1}) . Second, the backward ODE simulation for computing the adjoint state could be replaced by closed-form solution, eliminating backward ODE simulation entirely. Empirically, EAM reduces the per-iteration training cost by up to 4× compared to AM, while matching or surpassing its performance across various human preference metrics. Our contributions can be summarized as follows: Figure 1: Comparison of Efficient Adjoint Matching (EAM) with Adjoint Matching (AM). ( Left ) AM relies on a stochastic SDE solver to construct each training trajectory and a sequential backward simulation to obtain the adjoint state along that trajectory. ( Right ) EAM eliminates both: intermediate states X_{t} are obtained by first simulating the endpoint X_{1} with a few-step ODE and then sampling X_{t} from the original noising kernel q_{t}(\cdot|X_{1}) , while the adjoint state is given by a single closed-form evaluation, removing the backward simulation entirely. • We propose Efficient Adjoint Matching (EAM) , an efficient reward-gradient-based fine-tuning algorithm derived by redesigning the base drift and a terminal cost of SOC problem. • We characterize the linear base drifts satisfying our design requirements, enabling efficient ODE-based trajectory construction and a closed-form adjoint state. • We show that EAM achieves comparable or better performance on standard text-to-image reward fine-tuning benchmarks while converging up to 4× faster than AM. 2 Preliminaries 2.1 Diffusion and Flow Models Diffusion 16 ; 32 and flow matching 20 ; 1 ; 23 models share a common formulation through a pretrained velocity field v^{\mathrm{pt}}:\mathbb{R}^{d}\times[0,1]\rightarrow\mathbb{R}^{d} , learned by velocity matching along a conditional probability path. Throughout this paper, we focus on the linear path 20 ; 23 . Equivalently, given an endpoint X_{1} , intermediate states are sampled from the following noising kernel: \displaystyle q_{t}(X_{t}\mid X_{1}):=\mathcal{N}\!\left(tX_{1},(1-t)^{2}I\right),\qquad\emph{i.e.,}\quad X_{t}=tX_{1}+(1-t)\epsilon,\;\;\epsilon\sim\mathcal{N}(0,I), (2) where X_{1}\sim p_{\mathrm{data}} . We denote the endpoint distributions by p_{0}=\mathcal{N}(0,I) and p_{1}=p_{\mathrm{data}} . The pretrained velocity field v^{\mathrm{pt}} induces the following forward generative dynamic: \displaystyle\mathrm{d}X_{t}=\left(-\frac{1}{t}X_{t}+2v^{\mathrm{pt}}(X_{t},t)\right)\mathrm{d}t+\sigma(t)\,\mathrm{d}W_{t},\qquad X_{0}\sim p_{0},\qquad\sigma(t)=\sqrt{\frac{2(1-t)}{t}}. (3) We write p^{\mathrm{pt}} for the path measure induced by Equation 3 , and throughout the paper we fix the diffusion coefficient \sigma(t) as in Equation 3 . Since the pretrained model is trained to approximate the data distribution, we use the standard idealization p^{\mathrm{pt}}_{1}=p_{\mathrm{data}} . 2.2 Stochastic Optimal Control for Fine-tuning Diffusion Models Problem Setting and Notations. We consider the following SOC problem: \displaystyle\min_{u}\; \displaystyle\mathbb{E}_{p^{u}}\!\left[\int_{0}^{1}\frac{1}{2}\,\|u(X_{t},t)\|^{2}\,\mathrm{d}t+g(X_{1})\right], (4) \displaystyle\text{s.t.}\quad\mathrm{d}X_{t} \displaystyle=\bigl(b(X_{t},t)+\sigma(t)\,u(X_{t},t)\bigr)\,\mathrm{d}t+\sigma(t)\,\mathrm{d}W_{t},\qquad X_{0}\sim p_{0}, (5) where X_{t}\in\mathbb{R}^{d} is the state of the controlled dynamic ( 5 ), u:\mathbb{R}^{d}\times[0,1]\rightarrow\mathbb{R}^{d} is the control vector field, b:\mathbb{R}^{d}\times[0,1]\rightarrow\mathbb{R}^{d} is the base drift, and g:\mathbb{R}^{d}\rightarrow\mathbb{R} is the terminal cost. We denote by p^{u} the path measure induced by the controlled dynamic ( 5 ), and by p^{\text{base}} the path measure induced by the uncontrolled base dynamic, i.e. , Equation 5 with u\equiv 0 . With a properly designed terminal cost g , the controlled dynamic ( 5 ) under the optimal control u^{\star} reaches the target distribution p^{\star}_{1} at t=1 . Standard setup for SOC-based reward fine-tuning. Suppose the