TempFlow-GRPO: When Timing Matters for GRPO in Flow Models

Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, Bo Zhang, [1ex] ZheJiang University · 2025

Read TempFlow-GRPO: When Timing Matters for GRPO in Flow Models by Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, Bo Zhang, [1ex] ZheJiang University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04324.txt.

Introduction

Xiaoxuan He and co-authors present a 2025 method to train image models with human likes and dislikes. Flow models can make strong text-to-image art, yet past training gave each step in the build the same weight. The team argues that early steps shape the image far more than late steps. Flat credit for all steps thus wastes effort and slows gains.

Their method, named TempFlow-GRPO, adds three fixes for that flaw. First, it branches paths at set points so rewards can guide single steps. Second, it weights each step by how much new ground that step can still open. Third, it groups runs by seed to strip out luck from the start. The authors report top scores on tests of taste and image quality. The work points to training that respects how images form over time.

Catalog record

CREATORXiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, Bo Zhang, [1ex] ZheJiang University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04324

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TempFlow-GRPO: When Timing Matters for GRPO in Flow Models Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, Bo Zhang, [1ex] ZheJiang University Abstract Recent flow matching models for text-to-image generation have achieved remarkable quality, yet their integration with reinforcement learning for human preference alignment remains suboptimal, hindering fine-grained reward-based optimization. We observe that the key impediment to effective GRPO training of flow models is the temporal uniformity assumption in existing approaches: sparse terminal rewards with uniform credit assignment fail to capture the varying criticality of decisions across generation timesteps, resulting in inefficient exploration and suboptimal convergence. To remedy this shortcoming, we introduce TempFlow-GRPO (Temporal Flow GRPO), a principled GRPO framework that captures and exploits the temporal structure inherent in flow-based generation. TempFlow-GRPO introduces two key innovations: (i) a trajectory branching mechanism that provides process rewards by concentrating stochasticity at designated branching points, enabling precise credit assignment without requiring specialized intermediate reward models; and (ii) a noise-aware weighting scheme that modulates policy optimization according to the intrinsic exploration potential of each timestep, prioritizing learning during high-impact early stages while ensuring stable refinement in later phases. These innovations endow the model with temporally-aware optimization that respects the underlying generative dynamics, leading to state-of-the-art performance in human preference alignment and standard text-to-image benchmarks. 1 Introduction While text-to-image diffusion models have achieved unprecedented visual quality and semantic control Esser et al. 2024 ; Xie et al. 2024a ; Labs et al. 2025 , aligning their outputs with human preference remains a formidable challenge. Reinforcement learning has emerged as a promising solution, giving rise to the field of Diffusion RL Wallace et al. 2024 ; Black et al. 2023 ; Fan et al. 2023 . However, the performance of these methods remains suboptimal, hindered by two fundamental limitations that have been largely overlooked: ignoring the temporal dynamics of generation and lacking intermediate feedback signals . These approaches apply uniform optimization across all timesteps and provide rewards only at completion, missing the varying importance of decisions throughout the generation process. The majority of existing approaches Gu et al. 2024 ; Hong et al. 2024 , including recent works like Flow-GRPO Liu et al. 2025 and DanceGRPO Xue et al. 2025 , treat the multi-step generation process as a ”black box” with temporally agnostic optimization. They apply uniform updates across all timesteps despite the fact that each timestep operates under different noise conditions and contributes differently to final image quality. Specifically, we plot the Fig. 1 (left) with applying SDE at only one timestep in the entire ODE trajectory, which ensures that any deviation in the final reward can be attributed to the stochastic exploration introduced at that specific step. As shown in Fig. 1 (left), the std of the reward varies dramatically between timesteps, reaching a peak during early structural decisions (steps 0-2) and approaching zero during final refinements (steps 6-8). Yet Flow-GRPO maintains uniform treatment throughout, squandering high-impact exploration opportunities while focusing on later steps. Alternative approaches like SPO Liang et al. 2025 attempt to address temporal dynamics through process reward models, but training such models on semantically ambiguous intermediate states is notoriously difficult. This raises a fundamental question: how can we effectively achieve precise credit assignment for intermediate actions while adapting optimization intensity to each timestep’s exploration capacity? Figure 1: (Left) Reward Variance Analysis: We plot the standard deviation of PickScore at each denoising step for 200 prompts, per prompt group size is 24. The results, obtained via applying SDE at only one step, reveal that reward variance is highest in the initial steps, indicating that early-stage interventions are most impactful for exploration. (Right) Method Illustration: By branching a stochastic (SDE) exploration from a specific, known state on a deterministic (ODE) trajectory, we create a controlled experiment. The resulting difference in the final reward can be unambiguously attributed to the exploration action taken at that precise branching point. We address these limitations with TempFlow-GRPO , a temporally-aware RL framework built on two key insights. First , we define the visualization method in the left of Fig. 1 as trajectory branching, which enables precise credit assignment by strategically introducing stochasticity at individual timesteps while maintaining deterministic evolution elsewhere (Fig. 1 , Right). This provides provable guarantees: (1) reward variance localizes to the branching point, (2) improvements are directly attributable to specific exploration outcomes, and (3) existing reward models require no modification. Second , noise-aware policy weighting modulates optimization intensity based on each timestep’s intrinsic noise level. Early high-noise stages receive larger weight updates to encourage structural exploration, while late low-noise stages receive gentler updates to preserve learned features. Together, these mechanisms create a framework that is conceptually simple, computationally efficient, and seamlessly integrates into existing flow matching architectures—all while respecting the temporal dynamics that uniform approaches ignore. Our main contributions are threefold: • We pinpoint temporal uniformity—the equal treatment of all timesteps—as the primary limitation of flow-based GRPO. Our proposed TempFlow-GRPO overcomes this by introducing two key innovations: precise credit assignment to intermediate actions and noise-aware adaptation of optimization intensity. • We introduce trajectory branching and noise-aware weighting to learn temporally-structured policies that respect generative dynamics. • We demonstrate state-of-the-art performance on standard text-to-image benchmarks, achieving superior sample quality, human preference alignment, and compositional image generation compared to existing flow-based RL methods. 2 Related Work Alignment for Diffusion Models. Alignment for diffusion models has become a rapidly emerging topic in generative modeling research. D3PO Yang et al. 2024 introduce the Direct Preference for Denoising Diffusion Policy Optimization method to directly fine-tune diffusion models. Diffusion-DPO Wallace et al. 2024 is adapted from the Direct Preference Optimization (DPO Rafailov et al. 2023 ), a simpler alternative to RLHF which directly optimizes a policy that best satisfies human preferences under a classification objective. DyMO Xie & Gong 2025 propose a plug-and-play training-free alignment method for aligning the generated images and human preferences during inference. Recently, Flow-GRPO Liu et al. 2025 , the first method to integrate online reinforcement learning (RL) into flow matching models. However, Flow-GRPO applies uniform optimization pressure across all timesteps and suffers from sparse terminal reward problems, failing to account for the time-varying exploration potential inherent in the stochastic diffusion process. Our TempFlow-GRPO addresses these limitations through trajectory branching for precise credit assignment and noise-aware policy weighting that aligns optimization pressure with the natural exploration capacity at each timestep. Figure 2: (Left) Performance comparison on the PickScore benchmark. To ensure a fair and robust evaluation, we first present Flow-GRPO (Fixed), an improved baseline where we replace the original global std stabilization with a more appropriate group-wise std . On top of this stronger baseline, our core innovation, trajectory branching (+ Traj. Branch), delivers a significant performance leap. Our full model, TempFlow-GRPO, integrates all components to achieve the highest performance. (Right) TempFlow-GRPO’s dominance is further confirmed on the Geneval benchmark, where it substantially surpasses not only the Flow-GRPO baseline but also leading state-of-the-art models, including GPT-4o, SD3.5-M, and FLUX. Process Reward. Recent studies have demonstrated that shaping the reward process, rather than solely relying on sparse terminal rewards, can significantly accelerate learning and improve policy performance. Zhang et al. 2025 introduces a more comprehensive evaluation framework, which combines response-level and

Read the complete text