Sample by step, Optimize by Chunk: Chunk-Level GRPO for Text-to-Image Generation
Yifu Luo, Penghui Du, Bo Li, Sinan Du, Tiantian Zhang, Yongzhe Chang, Kai Wu, Kun Gai, Xueqian Wang · 2025
Read Sample by step, Optimize by Chunk: Chunk-Level GRPO for Text-to-Image Generation by Yifu Luo, Penghui Du, Bo Li, Sinan Du, Tiantian Zhang, Yongzhe Chang, Kai Wu, Kun Gai, Xueqian Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21583.txt.
Introduction
Yifu Luo and fourteen co-authors retrain image-generating flow models with reinforcement learning at the level of chunks. Group-based policy methods have shown promise for polishing text-to-image models. But they misjudge which generation steps deserve credit or blame for the final picture. That blurs the learning signal.
Their method groups consecutive steps into coherent chunks. It optimizes the policy over those chunks instead of single steps. Credit then lands on meaningful stretches of the generation process. The authors present this as the first chunk-level reinforcement learning scheme for flow matching post-training. Across standard image benchmarks, the method beats the step-level baseline clearly. It also wins on preference alignment tests. Relative gains reach about forty-three percent.
Catalog record
| CREATOR | Yifu Luo, Penghui Du, Bo Li, Sinan Du, Tiantian Zhang, Yongzhe Chang, Kai Wu, Kun Gai, Xueqian Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21583
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sample by step, Optimize by Chunk: Chunk-Level GRPO for Text-to-Image Generation Yifu Luo, Penghui Du, Bo Li, Sinan Du, Tiantian Zhang, Yongzhe Chang, Kai Wu, Kun Gai, Xueqian Wang Abstract Group Relative Policy Optimization (GRPO) has shown strong potential for flow-matching-based text-to-image (T2I) generation, but it faces two key limitations: inaccurate advantage attribution, and the neglect of temporal dynamics of generation. In this work, we argue that shifting the optimization paradigm from the step level to the chunk level can effectively alleviate these issues. Building on this idea, we propose Chunk-GRPO, the first chunk-level GRPO-based approach for T2I generation. The insight is to group consecutive steps into coherent ‘chunk’s that capture the intrinsic temporal dynamics of flow matching, and to optimize policies at the chunk level. In addition, we introduce an optional weighted sampling strategy to further enhance performance. Extensive experiments show that Chunk-GRPO achieves superior results in both preference alignment and image quality, highlighting the promise of chunk-level optimization for GRPO-based methods. † † footnotetext: * Equal Contibution. \dagger Project Lead. 🖂Corresponding Authors. † † footnotetext: \ddagger Work done during internship in Kolors Team, Kuaishou Technology. Figure 1: Chunk-GRPO significantly improves image quality, particularly in structure, lighting, and fine-grained details, demonstrating the superiority of chunk-level optimization. 1 Introduction Reinforcement learning (RL) ( Sutton et al. 1998 ; Schulman et al. 2017 ) has recently found success beyond traditional domains, particularly in the reasoning of Large Language Models (LLMs) ( Jaech et al. 2024 ; Guo et al. 2025 ) . Inspired by these advances, recent works ( Xue et al. 2025 ; Liu et al. 2025b ; Wang & Yu 2025 ) have explored applying RL to text-to-image (T2I) generation for aligning specific preferences. In this context, Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ; Guo et al. 2025 ) has emerged as a promising approach for flow-matching-based T2I generation ( Lipman et al. 2022 ; Liu et al. 2023 ; Esser et al. 2024 ) . GRPO-based methods typically sample a group of images from the same prompt, evaluate them using reward models, convert the rewards into group relative advantages, and then assign these advantages equally across all timesteps during optimization. Figure 2: While Trajectory1 has the greater final reward (advantage), its t=1 timestep is worse than that in Trajectory2. However, GRPO assigns the final advantages equally across all timesteps. While effective, this uniform assignment introduces two key limitations: (1) inaccurate advantage attribution, and (2) disregard for the temporal dynamics of generation. We first illustrate the former in Figure 2 , and discuss temporal dynamics later. Consider two generation trajectories from the same prompt in Figure 2 , each consisting of three timesteps. Although the final advantage correctly favors the better trajectory (Trajectory1), assigning this same advantage uniformly across all timesteps incorrectly assumes that every step in Trajectory1 is superior to its counterpart in Trajectory2. However, at timestep t=1 Trajectory2 is clearly better than Trajectory1, despite Trajectory1 achieving the higher overall reward. To address this, we draw inspiration from action chunking ( Zhao et al. 2023 ; Li et al. 2025b ) in robotics, which predicts sequences of consecutive actions jointly rather than treating each step independently. In a similar spirit, we propose to group consecutive timesteps into ‘chunk’s, and optimize at the chunk level rather than the step level. This alleviates the issue of inaccurate advantage attribution, as we analyze in detail in Section 4.1 . Related ideas have been explored in LLMs as Group Sequence Policy Optimization (GSPO) ( Zheng et al. 2025 ) , where an entire token sequence is treated as a single unit (analogous to viewing the whole trajectory as one chunk). However, our preliminary studies reveal that different chunk settings (e.g. how many consecutive timesteps for a chunk) have a substantial impact on performance. We argue that this is due to the overlooking of temporal dynamics of flow matching generation, which we proposed before. Different from LLMs, flow matching exhibits distinct temporal dynamics: each timestep operates under different noise conditions and contributes differently to the final image. Specifically, following ( Wimbauer et al. 2024 ; Liu et al. 2025a ) , we analyze the relative L1 distance of intermediate latents. As shown in Figure 3 , the results reveal clear, prompt-invariant dynamic patterns that naturally segment the trajectory into meaningful chunks. These observations suggest that chunks should not be arbitrary but guided by the inherent temporal dynamics, with timesteps that are dynamically correlated optimized together. Based on these, we propose Chunk-GRPO, a novel chunk-level RL approach for flow-matching-based T2I generation. As demonstrated in Figure 4 , our key innovation is grouping timesteps into chunks that reflect temporal dynamics, and optimizing them as units with a principled chunk-level importance ratio. Furthermore, motivated by the varying contributions of different chunks, we design an optional weighted sampling strategy to further boost Chunk-GRPO’s performance. Our contributions can be summarized as follows: • We are the first to introduce the chunk-level RL optimization for T2I generation. We pinpoint that chunk-level optimization alleviates the inaccurate advantage attribution and mitigates the neglect of temporal dynamics from GRPO-based approaches. • We propose Chunk-GRPO, a novel chunk-level approach for flow-matching-based T2I generation, which integrates chunk-level optimization with temporal-dynamic-guided chunking. An optional weighted sampling strategy is introduced to push Chunk-GRPO further. • Extensive experiments demonstrate that Chunk-GRPO achieves superior performance on preference alignment and standard T2I benchmarks, highlighting the effectiveness of chunk-level optimization. 2 Related Work 2.1 Action Chunk Action chunk ( Zhao et al. 2023 ; Lai et al. 2022 ) has been widely applied to robotics Chi et al. 2023 . This approach mitigates compounding error and non-Markovian noise in human demonstrations by jointly predicting a sequence of future actions rather than a single step. By shortening the effective control horizon, action chunking enables smoother and more stable rollouts. Recently, it has also proven effective in vision-language-action (VLA) models ( Black et al. 2024a ; Intelligence et al. ) and in RL ( Li et al. 2025b ) . These successes suggest that chunking stabilizes long-horizon prediction, accelerates value propagation, and more effectively leverages non-Markovian behavior. 2.2 Reinforcement Learning for Diffusion-based Image Generation Diffusion models ( Ho et al. 2020 ; Rombach et al. 2022 ; Podell et al. 2023 ; Labs et al. 2025 ; Wu et al. 2025 ) have become one of the dominant paradigms for T2I generation. Early works ( Xu et al. 2023 ; Black et al. 2024b ; Fan et al. 2023 ) introduced RL into diffusion models through policy gradient optimization. Preference-based methods ( Wallace et al. 2024 ; Sun et al. 2025a ; Sun et al. 2025c ; Sun et al. 2025d ; Sun et al. 2025e ) were later developed, achieving competitive alignment without explicit reward modeling. More recently, GRPO ( Shao et al. 2024 ; Sun et al. 2025b ) has attracted attention as an efficient alternative. Dance-GRPO ( Xue et al. 2025 ) and Flow-GRPO ( Liu et al. 2025b ) pioneered the use of GRPO for T2I generation, unifying diffusion and flow matching through an SDE-based reformulation. MixGRPO ( Li et al. 2025a ) further improved efficiency via a mixed ODE–SDE paradigm. TempFlow-GRPO ( He