Stepwise Credit Assignment for GRPO on Flow-Matching Models
Yash Savani, Branislav Kveton, Yuchen Liu, Yilin Wang, Jing Shi, Subhojyoti Mukherjee, Nikos Vlassis, Krishna Kumar Singh, [2pt] Work done while intern at Adobe. Corresponding author: [email protected] . https://stepwiseflowgrpo.com · 2026
Read Stepwise Credit Assignment for GRPO on Flow-Matching Models by Yash Savani, Branislav Kveton, Yuchen Liu, Yilin Wang, Jing Shi, Subhojyoti Mukherjee, Nikos Vlassis, Krishna Kumar Singh, [2pt] Work done while intern at Adobe. Corresponding author: [email protected] . https://stepwiseflowgrpo.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28718.txt.
Introduction
Savani and co-authors improve credit assignment for GRPO on flow-matching models. Flow-GRPO gives every step the same credit, based only on the final image. Early steps set composition, while late steps add detail, so uniform credit can reward bad early steps. Their Stepwise-Flow-GRPO credits each step by its reward improvement. It uses Tweedie's formula for intermediate reward estimates and gain-based advantages. A DDIM-inspired stochastic equation improves reward quality while keeping randomness for policy gradients.
Catalog record
| CREATOR | Yash Savani, Branislav Kveton, Yuchen Liu, Yilin Wang, Jing Shi, Subhojyoti Mukherjee, Nikos Vlassis, Krishna Kumar Singh, [2pt] Work done while intern at Adobe. Corresponding author: [email protected] . https://stepwiseflowgrpo.com |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60821 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28718
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stepwise Credit Assignment for GRPO on Flow-Matching Models Yash Savani, Branislav Kveton, Yuchen Liu, Yilin Wang, Jing Shi, Subhojyoti Mukherjee, Nikos Vlassis, Krishna Kumar Singh, [2pt] Work done while intern at Adobe. Corresponding author: [email protected] . https://stepwiseflowgrpo.com Abstract Flow-GRPO successfully applies reinforcement learning to flow models, but uses uniform credit assignment across all steps. This ignores the temporal structure of diffusion generation: early steps determine composition and content (low-frequency structure), while late steps resolve details and textures (high-frequency details). Moreover, assigning uniform credit based solely on the final image can inadvertently reward suboptimal intermediate steps , especially when errors are corrected later in the diffusion trajectory. We propose Stepwise-Flow-GRPO , which assigns credit based on each step’s reward improvement. By leveraging Tweedie’s formula to obtain intermediate reward estimates and introducing gain-based advantages, our method achieves superior sample efficiency and faster convergence. We also introduce a DDIM-inspired SDE that improves reward quality while preserving stochasticity for policy gradients. Figure 1 : Stepwise credit assignment from temporal reward structure. (Left) Two trajectories for the same prompt, showing Tweedie estimates \hat{x}_{0}^{i}(t) and their PickScore rewards r_{t}^{i} at each denoising step. The reward curves are non-monotonic and frequently cross—trajectory 0 (blue) dips at t{=}0.86 before recovering, while trajectory 1 (orange) drops sharply at t{=}0.71 —yet both reach similar final rewards ( {\sim}0.90 ). Uniform credit assignment would treat these trajectories nearly identically, reinforcing the poor intermediate steps along with the good ones. Stepwise-Flow-GRPO instead uses gains g_{t}^{i}=r_{t-1}^{i}-r_{t}^{i} to penalize steps that hurt reward and credit steps that improve it, regardless of final outcome. (Right) This finer credit assignment yields faster convergence and higher final reward compared to Flow-GRPO. 1 Introduction Reinforcement learning (RL) has dramatically improved reasoning in large language models (LLMs) 41 , but adapting it to flow-matching models for text-to-image generation remains challenging. Flow-GRPO 26 and Dance-GRPO 56 showed that policy gradients can be applied to flow-matching models like SD-3.5-M by converting the deterministic flow ordinary differential equation (ODE) into a stochastic differential equation (SDE) with matched marginals. These approaches assign the same advantage—calculated from the rewards on the final images—to all steps in a trajectory, effectively rewarding or penalizing every denoising step based solely on the final generated image. This uniform credit assignment ignores the temporal structure of all diffusion generation processes, where different steps contribute to distinct image properties. Early denoising steps contain primarily low-frequency information, which determines composition and layout, while late steps resolve high-frequency details and textures. Rewarding an entire trajectory based on final image quality conflates these phases, potentially reinforcing poor early decisions that are compensated for later. As shown in Fig. 1 , intermediate reward estimates reveal substantial temporal structure across steps that uniform credit assignment fails to exploit. To capitalize on this opportunity, we propose Stepwise-Flow-GRPO , a stepwise credit assignment approach that directly rewards the contribution of each denoising step. Using Tweedie’s formula 9 , we predict \hat{x}_{0}(t):=\E[x_{0}|x_{t}] from intermediate noisy states x_{t} and reward these estimates as r_{t}:=R(\hat{x}_{0}(t),\text{prompt}) . We then use the stepwise reward gain g_{t}=r_{t-1}-r_{t} —measuring the amount each step improves the reward—to calculate the advantages that are used to optimize the policy with GRPO 41 . This rewards each step based on its actual contribution rather than final image quality alone. This design offers better credit assignment and improves sample efficiency. Fig. 4 shows that Stepwise-Flow-GRPO achieves better sample efficiency during training and converges faster. We also notice that while the Flow-GRPO SDE provably matches marginals of the flow ODE, the images generated with the SDE are noisy. To account for this, we introduce a new SDE, inspired by DDIM 43 , that produces high-quality images while matching the Flow-ODE Fokker-Planck marginal in the Taylor limit. Since reward models are trained on clean images, the noisy samples from Flow-GRPO’s SDE degrade reward signal; our formulation mitigates this, providing a complementary way to accelerate Flow-GRPO. We make the following contributions: • We introduce Stepwise-Flow-GRPO, which rewards relative gain for each diffusion step instead of just rewarding final image. • We further optimize Flow-GRPO using an improved SDE inspired by DDIM to produce less noisy samples. • Our results show that Stepwise-Flow-GRPO achieves significantly better sample efficiency and convergence rates than standard Flow-GRPO. 2 Related Work Reinforcement learning (RL). RL is an area of machine learning where the goal is to learn a policy that maximizes a long-term reward in an uncertain environment ( 45 ) . The beginnings of RL can be traced to model-based approaches for acting under uncertainty, such as Markov decision processes (MDPs) ( 4 ; 33 ) and partially-observable MDPs (POMDPs) ( 42 ) . Because of the generality of the framework, many RL algorithms have been proposed, such as temporal-difference learning ( 47 ) , Q-learning ( 51 ) , policy gradients ( 52 ) , and actor-critic methods ( 46 ) . The main challenge with applying classic RL algorithms, which rely on value functions 4 and Q functions ( 51 ) , to modern generative AI models is their scale. Therefore, most modern approaches to RL are based on policy gradients with KL regularization ( 48 ; 38 ) . In proximal policy optimization (PPO) ( 40 ) , a KL-regularized policy is optimized with respect to a reward model. This approach has been successfully applied to reinforcement learning from human feedback ( 30 ) . The main challenge with applying PPO in practice is that it requires a fine-grained critic model, which is yet another challenging learning problem to solve. Group relative policy optimization (GRPO) ( 41 ) overcomes this problem by estimating and standardizing the rewards using simulation against an environment. Our work replaces the total trajectory reward in GRPO with per-step gains, yielding fine-grained credit assignment analogous to a learned critic in PPO—but without requiring a separate critic model. This design is motivated by 23 , who showed that KL-regularized policy gradients over stepwise gains can learn near-optimal greedy policies when the reward is monotone and submodular in the state, extending classic guarantees from submodular optimization ( 29 ; 13 ) . We are the first to apply this gain-based transformation to GRPO and flow models, generalizing beyond the on-policy setting of 23 to off-policy optimization with group-relative advantages. RL for image generation. Recent progress in diffusion models 8 ; 35 ; 32 ; 7 ; 37 has led to substantial improvements in text-to-image generation, outperforming earlier GAN 14 ; 55 ; 21 and VAE-based 34 methods in both image quality and semantic alignment. Among these, flow-matching techniques 10 ; 24 have shown strong performance but still face challenges in achieving precise image-text alignment and maintaining aesthetic fidelity 19 ; 53 ; 5 . To address these issues, researchers have explored methods such as attention manipulation 6 ; 1 ; 3 , contrastive learning 25 , and asynchronous diffusion 18 . While these approaches offer partial improvements, they often lack robustness and generalization. More recently, RL has emerged as a promising direction for enhancing alignment and image quality 20 ; 31 ; 56 ; 26 , which leverages pre-trained VLM models 22 ; 54 ; 50 ; 28 to measure various aspects related to image quality and text alignment. Flow-GRPO 26