Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang · 2025

Read Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning by Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01819.txt.

Introduction

Hanyang Zhao and four co-authors improve how image-generating models are fine-tuned, in a 2025 paper. Such tuning often uses feedback methods that work in discrete steps. Step-based methods can add errors and may not suit some solvers. Their approach treats the tuning problem in continuous time instead.

They frame the task as steering a random process toward a reward tied to the prompt. The key move is to treat the model's score signal as the control action itself. That link opens a new way to design the value parts of the system. Tests on a large text-to-image model showed gains on downstream tasks, backing the approach.

Catalog record

CREATORHanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-01819

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang Abstract Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area use a discrete-time formulation, which is prone to induced discretization errors, and often not applicable to models with higher-order/black-box solvers. The objective of this study is to develop a disciplined approach to fine-tune diffusion models using continuous-time RL, formulated as a stochastic control problem with a reward function that aligns the end result (terminal state) with input prompt. The key idea is to treat score matching as controls or actions, and thereby making connections to policy optimization and regularization in continuous-time RL. To carry out this idea, we lay out a new policy optimization framework for continuous-time RL, and illustrate its potential in enhancing the value networks design space via leveraging the structural property of diffusion models. We validate the advantages of our method by experiments in downstream tasks of fine-tuning large-scale Text2Image models of Stable Diffusion v1.5. 1 Introduction Diffusion models ( Sohl-Dickstein et al., 2015 ) , with the capacity to turn a noisy/non-informative initial distribution into a desired target distribution through a well-designed denoising process ( Ho et al., 2020 ; Song et al., 2020 ; Song et al., 2021b ) , have recently found applications in diverse areas such as high-quality and creative image generation ( Ramesh et al., 2022 ; Shi et al., 2020 ; Saharia et al., 2022 ; Rombach et al., 2022 ) , video synthesis ( Ho et al., 2022 ) , and drug design ( Xu et al., 2022 ) . And, the emergence of human-interactive platforms like ChatGPT ( Ouyang et al., 2022 ) and Stable Diffusion ( Rombach et al., 2022 ) has further increased the demand for diffusion models to align with human preference or feedback. To meet such demands, ( Hao et al., 2022 ) proposed a natural way to fine-tune diffusion models using reinforcement learning (RL, ( Sutton & Barto, 2018 ) ). Indeed, RL has already demonstrated empirical successes in enhancing the performance of LLM (large language models) using human feedback ( Christiano et al., 2017 ; Ouyang et al., 2022 ; Bubeck et al., 2023 ) , and ( Fan & Lee, 2023 ) is among the first to utilize RL-like methods to train diffusion models for better image synthesis. Moreover, ( Lee et al., 2023 ; Fan et al., 2023 ; Black et al., 2023 ) have improved the text-to-image (T2I) diffusion model performance by incorporating reward models to align with human preference (e.g., CLIP ( Radford et al., 2021 ) , BLIP ( Li et al., 2022 ) , ImageReward ( Xu et al., 2024 ) ). Notably, all studies referenced above that combine diffusion models with RL are formulated as discrete-time sequential optimization problems, such as Markov decision processes (MDPs, ( Puterman, 2014 ) ), and solved by discrete-time RL algorithms such as REINFORCE ( Sutton et al., 1999 ) or PPO ( Schulman et al., 2017 ) . Yet, diffusion models are intrinsically continuous-time as they were originally created to model the evolution of thermodynamics ( Sohl-Dickstein et al., 2015 ) . Notably, the continuous-time formalism of diffusion models provides a unified framework for various existing discrete-time algorithms as shown in ( Song et al., 2021b ) : the denoising steps in DDPM ( Ho et al., 2020 ) can be viewed as a discrete approximation of a stochastic differential equation (SDE) and are implicitly score-based under a specific variance-preserving SDE ( Song et al., 2021b ) ; and DDIM ( Song et al., 2020 ) , which underlies the success of Stable Diffusion ( Rombach et al., 2022 ) , can also be seen as a numerical integrator of an ODE (ordinary differential equation) sampler ( Salimans & Ho, 2022 ) . Awareness of the continuous-time nature informs the design structure of the discrete-time SOTA large-scale T2I generative models (e.g., ( Dhariwal & Nichol, 2021 ; Rombach et al., 2022 ; Esser et al., 2024 ) ), and enables simple controllable generations by classifier guidance to solve inverse problems ( Song et al., 2021b ; Song et al., 2021a ) . It also motivates more efficient diffusion models with continuous-time samplers, including the ODE-governed probability (normalizing) flows ( Papamakarios et al., 2021 ; Song et al., 2021b ) and rectified flows ( Liu et al., 2022 ; Liu et al., 2023 ) underpinning Stable Diffusion v3 ( Esser et al., 2024 ) . A discrete-time formulation of RL algorithms for fine-tuning diffusion models, if/when directly applied to continuous-time diffusion models via discretization, can nullify the models’ continuous nature and fail to capture or utilize their structural properties. For fine-tuning diffusion models, discrete-time RL algorithms (such as DDPO) require a prior chosen time discretization in sampling. We thus examine the robustness of a fine-tuned model to the inference time discretization, and observe an “overfitting” phenomenon as illustrated in Figure 1 . Specifically, improvements observed during inference at alternative discretization timesteps (25 and 100) are significantly smaller than that of sampling timestep (50) in RL. Figure 1: Reward curve of model checkpoints sampling under different discretization steps (25, 50, 100): After training Stable Diffusion v1.4 for a fixed prompt with 60 training steps by DDPO ( Black et al., 2023 ) with 50 discretization steps, the average reward of images generated by the checkpoints obtained (under 50 discretization steps) evaluated by ImageReward ( Xu et al., 2024 ) increases by 0.046, while the average reward of images generated with 100 discretization steps only increases by less than 0.016. In addition, for high-order solvers (such as 2 {}^{\text{nd}} order Heun used in EDM ( Karras et al., 2022 ) ), discrete-time RL methods will require solving a high-dimension root-finding problem for each inference step, which is inefficient in practice. Main contributions. To address the above issues, we develop a unified continuous-time RL framework to fine-tune score-based diffusion models. Our first contribution is a continuous-time RL framework for fine-tuning diffusion models by treating score functions as actions . This framework naturally accommodates discrete-time diffusion models with any solver as well as continuous-time diffusion models, and overcomes the afore-mentioned limitations of discrete-time RL methods. (See Section 3 .) Second, we illustrate the promise of leveraging the structural property of diffusion models to generate tractable optimization problems and to enhance the design space of value networks. This includes transforming the KL regularization to a tractable running reward over time, and a novel design of value networks that involves “sample prediction” (also known as x -prediction in diffusion literature) by sharing parameters with policy networks and fine-tuned diffusion models. Through experiments, we demonstrate the drastic improvements over naive value network designs. Third, we provide a new theory for RL in continuous-time and space, which leads to the first scalable policy optimization algorithm (for continuous-time RL), along with estimates of policy gradients via generated samples. Compared with existing works in continuous-time PPO, we consider a special case of state-independent diffusion coefficients cater for the

Read the complete text