Reinforce Adjoint Matching

Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken, Carles Domingo-Enrich, Jakiw Pidstrigach · 2026

Read Reinforce Adjoint Matching by Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken, Carles Domingo-Enrich, Jakiw Pidstrigach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10759.txt.

Introduction

Andreas Bergmeister and colleagues derive Reinforce Adjoint Matching, a method scaling reinforcement learning post-training of diffusion and flow-matching models while keeping pretraining's regression build. Diffusion and flow-matching models grow since pretraining is watched regression: a clean sample is noised by rule and the model fits a closed-form goal.

Under KL-ruled reward growth, the best making process tilts the clean-end spread toward higher-reward samples and leaves the noising law as is. Joining this with the adjoint-matching best rule and a REINFORCE fact gives a fit loss that fixes the pretraining goal with the reward. No rollouts, backward adjoint sweeps, or reward gradients are needed. On Stable Diffusion 3.5M, the method hits the top reward on composability, text rendering, and human preference, reaching a rival's peak reward in up to fifty times fewer training steps.

Catalog record

CREATORAndreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken, Carles Domingo-Enrich, Jakiw Pidstrigach
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73443
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10759

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforce Adjoint Matching Andreas Bergmeister, Stefanie Jegelka, Nikolas Nüsken, Carles Domingo-Enrich, Jakiw Pidstrigach Abstract Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses against a closed-form target. RL post-training aligns the model with a reward. In image generation, this makes samples compose objects correctly, render text legibly, and match human preferences. Existing methods rely on costly SDE rollouts, reward gradients, or surrogate losses, sacrificing pretraining’s regression structure. We show that the structure extends to RL post-training. Under KL-regularized reward maximization, the optimal generative process tilts the clean-endpoint distribution towards samples with higher reward and leaves the noising law unchanged. Combining this with the adjoint-matching optimality condition and a REINFORCE identity, we derive Reinforce Adjoint Matching (RAM): a consistency loss that corrects the pretraining target with the reward. At each step, we draw a clean endpoint from the current model, evaluate its reward, noise it as in pretraining, and regress. No SDE rollouts, backward adjoint sweeps, or reward gradients are required. Like the pretraining objective, RAM is simple and scales. On Stable Diffusion 3.5M, RAM achieves the highest reward on composability, text rendering, and human preference, reaching Flow-GRPO’s peak reward in up to 50 × 50\times fewer training steps. \authorbreak 1]TU Munich, MCML 2]MIT CSAIL 3]King’s College London 4]Microsoft Research New England 5] University of Oxford \contribution † Joint last authors \metadata [Code] https://github.com/AndreasBergmeister/ram Scaling RL Post-Training of Diffusion and Flow-Matching Models 1 Introduction Diffusion and flow-matching models dominate high-fidelity generation in continuous domains ( 33 ; 16 ; 34 ; 22 ; 24 ) . They scale because pretraining is supervised regression. A clean sample from the dataset is corrupted by Gaussian noise, and a model regresses against a closed-form target. That is the entire training procedure. We often want to optimize generation towards a reward function, rather than just matching the data distribution. In image generation, this is what makes a model follow complex prompts ( 14 ) , render text legibly, or align with human preferences ( 40 ) . In language modeling, reinforcement learning (RL) post-training is what produces today’s reasoning systems. Given a pretrained model and a reward function, the canonical post-training objective is to maximize expected reward while staying close to the pretrained model in a KL sense. For models with tractable likelihoods, such as language models, policy gradients optimize this objective using the log-likelihood of sampled outputs. In diffusion and flow-matching models, sample likelihoods are intractable, so post-training is more challenging. Existing methods pay a different price to work around this. Policy-gradient methods make the denoising process stochastic and use Gaussian transition densities to optimize a variational lower bound ( 4 ; 13 ; 23 ) . Stochastic optimal control methods target the KL-regularized objective exactly, but current estimators require reward gradients backpropagated through the sampling process via an adjoint ODE ( 36 ; 11 ) . Both approaches sample by SDE rollout, which requires many steps, and for flow-matching models the noise schedule diverges near the noisy end. To sidestep this, recent methods rely on surrogate objectives: 41 replace the intractable likelihood with a flow-matching ELBO, and 46 distill an improvement direction by contrasting positive and negative endpoints. We show that the regression structure of pretraining extends to RL post-training. Casting post-training as a stochastic optimal-control problem gives a fixed-point condition for the optimal control ( 11 ) . From this condition and a REINFORCE identity, we derive a reward-corrected regression target for the model. To scale this target, we use a key structural property of the optimum: it changes which clean samples are likely, but not how a fixed clean sample is noised. The rule that generated training pairs at pretraining is the same at the optimum. This lets us sample an endpoint from the current model using any off-the-shelf sampler, evaluate its reward, and noise it analytically as in pretraining. We reuse each endpoint for multiple noisy samples, amortizing the cost of sampling and reward evaluation. Our method, Reinforce Adjoint Matching (RAM), uses no SDE rollouts, no backward adjoint sweeps, and no reward gradients. We post-train Stable Diffusion 3.5M on compositional generation (GenEval), visual text rendering (OCR), and human-preference alignment (PickScore). RAM achieves the highest reward on each task without reward hacking or visible quality degradation. It matches Flow-GRPO’s peak reward in 50\times , 48\times , and 34\times fewer training steps, at slightly lower per-step compute cost. The simple regression that scales pretraining now scales post-training too. 2 Diffusion and Flow-Matching Models Diffusion and flow-matching models share a common structure: a forward process that corrupts data to noise from time t{=}0 to t{=}1 , and a backward process that reverses it. Diffusion models construct both processes explicitly. Flow-matching models specify only the noising kernel and generate by integrating the velocity field (the probability-flow ODE). The associated forward and backward SDEs are nonetheless well-defined. Forward noising. We linearly interpolate between a data distribution p_{0} on \mathbb{R}^{d} at t{=}0 and the Gaussian prior \mathcal{N}(0,\,I) at t{=}1 ( 24 ; 22 ; 12 ) . Non-linear interpolations recover other classical schedules such as DDPM ( 16 ; 33 ; 34 ) , and all results in this paper generalize. The interpolation defines a noising kernel that sends a clean sample X_{0}\sim p_{0} to a noisy version at time t , X_{t}\mid X_{0}\;\sim\;\mathcal{N}\!\bigl((1-t)\,X_{0},\,t^{2}I\bigr). (1) The kernel is realized by the linear forward SDE \mathrm{d}X_{t}=\kappa_{t}\,X_{t}\,\mathrm{d}t+\sigma_{t}\,\mathrm{d}B_{t},\qquad X_{0}\sim p_{0}, (2) with \kappa_{t}=-1/(1-t) , \sigma_{t}^{2}=2t/(1-t) , and B_{t} a standard Brownian motion on \mathbb{R}^{d} . We write p_{t} for the marginal density of X_{t} . Velocity matching. The standard choice in large-scale generative modeling ( 24 ; 12 ) is to learn a model v^{\theta} of the velocity field v_{t}(x)=\mathbb{E}\!\left[\epsilon-X_{0}\;\middle|\;X_{t}=x\right],\qquad X_{0}\sim p_{0},\quad\epsilon\sim\mathcal{N}(0,\,I). (3) By the tower property, regressing v_{t}^{\theta} against the random target inside ( 3 ) recovers the conditional expectation. Because X_{t} is an affine function of X_{0} and \epsilon , each training pair is cheap to construct from a data sample and independent noise. The flow-matching loss is {graybox} \begin{gathered}\mathcal{L}_{\mathrm{FM}}(\theta)=\mathbb{E}_{t}\!\left[\bigl\|v_{t}^{\theta}(X_{t})-(\epsilon-X_{0})\bigr\|^{2}\right],\\ \text{where }X_{0}\sim p_{0},\quad\epsilon\sim\mathcal{N}(0,\,I),\quad X_{t}=(1-t)\,X_{0}+t\epsilon.\end{gathered} (4) Alternative parametrizations that predict the clean data or the noise instead are equivalent: given X_{t} , any two of the velocity, clean data, and noise determine the third. Backward generation. The time reversal of the forward SDE ( 2 ) ( 2 ) is \mathrm{d}X_{t}=\bigl(\kappa_{t}X_{t}-\sigma_{t}^{2}\nabla_{x}\!\log p_{t}(X_{t})\bigr)\mathrm{d}t+\sigma_{t}\,\mathrm{d}B_{t},\qquad X_{1}\sim\mathcal{N}(0,\,I), (5) which we integrate from t{=}1 to t{=}0 to sample from p_{0} . The score is related to the velocity by \nabla_{x}\!\log p_{t}(x)=\frac{2}{\sigma_{t}^{2}}\bigl(\kappa_{t}x-v_{t}(x)\bigr), (6) so a learned velocity model v^{\theta} suffices to sample. More generally, replacing the noise coefficient in ( 5 ) by any other schedule \tilde{\sigma}_{t}\geq 0 (with correspondingly adjusted drift) preserves the marginals p_{t} ( 1 ; 34 ) . The null choice \tilde{\sigma}{\equiv}0 recovers the probability-flow ODE \mathrm{d}X_{t}=v_{t}(X_{t})\,\mathrm{d}t , which is the default sampler for flow-matching models. 3 RL Post-Training as Optimal Control Given a pretrained generative model with endpoint distribution p_{0}^{\mathrm{ref}} and a scalar reward function r:\mathbb{R}^{d}\to\mathbb{R} , the canonical KL-regularized target is the tilted distribution p_{0}^{\ast}=\argmax_{p}\bigl\{\mathbb{E}_{x\sim p}[r(x)]-D_{\mathrm{KL}}\!\bigl(p\|p_{0}^{\mathrm{ref}}\bigr)\bigr\}\;\propto\;p_{0}^{\mathrm{ref}}(x)\exp\bigl(r(x)\bigr). (7) The KL term keeps the post-trained model close to the pretrained reference and within its support. The reward shifts mass toward desirable samples. The regularization strength is controlled by scaling

Read the complete text