SuperFlow : Training Flow Matching Models with RL on the Fly
Kaijie Chen, Zhiyang Xu, Ying Shen, Zihao Lin, [4pt] Yuguang Yao, Lifu Huang · 2025
Read SuperFlow : Training Flow Matching Models with RL on the Fly by Kaijie Chen, Zhiyang Xu, Ying Shen, Zihao Lin, [4pt] Yuguang Yao, Lifu Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17951.txt.
Introduction
Kaijie Chen, Zhiyang Xu, Ying Shen, Zihao Lin, Yuguang Yao, and Lifu Huang presented SuperFlow, a faster way to train flow based image generators with reinforcement learning. The framework varied how many trial images each prompt received and scored credit step by step, matching the flow of the model. Reported tests used far fewer training steps and beat a strong base model by wide margins on text and layout tasks.
Readers should know that this paper has been withdrawn from arXiv. The withdrawal note states it was submitted without the consent of all listed authors. Because the submission itself lacked full author approval, the reported gains should not be relied on. The entry is kept as the record of a withdrawn submission.
Catalog record
| CREATOR | Kaijie Chen, Zhiyang Xu, Ying Shen, Zihao Lin, [4pt] Yuguang Yao, Lifu Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17951
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SuperFlow : Training Flow Matching Models with RL on the Fly Kaijie Chen, Zhiyang Xu, Ying Shen, Zihao Lin, [4pt] Yuguang Yao, Lifu Huang Abstract Recent progress in flow-based generative models and reinforcement learning (RL) has improved text–image alignment and visual quality. However, current RL training for flow models still has two main problems: (i) GRPO-style fixed per-prompt group sizes ignore variation in sampling importance across prompts, which leads to inefficient sampling and slower training; and (ii) trajectory-level advantages are reused as per-step estimates, which biases credit assignment along the flow. We propose SuperFlow , an RL training framework for flow-based models that adjusts group sizes with variance-aware sampling and computes step-level advantages in a way that is consistent with continuous-time flow dynamics. Empirically, SuperFlow reaches promising performance while using only 5.4 % ∼ 56.3 % 5.4\%\sim 56.3\% of the original training steps and reduces training time by 5.2 % ∼ 16.7 % 5.2\%\sim 16.7\% without any architectural changes. On standard text-to-image (T2I) tasks, including text rendering, compositional image generation, and human preference alignment, SuperFlow improves over SD3.5 - M by 4.6 % ∼ 47.2 % 4.6\%\sim 47.2\% , and over Flow - GRPO by 1.7 % ∼ 16.0 % 1.7\%\sim 16.0\% . The code will be available at https://github.com/jkkjjj/superflow . † † footnotetext: ∗ Equal Contribution ϵ Tongji University θ Virginia Tech η University of Illinois Urbana-Champaign ζ University of California, Davis ν Intuit. 1 Introduction First, GRPO 12 constructs per-prompt baselines on the fly by generating multiple images for each prompt within a fixed group size, which leads to three key challenges that directly motivate more sample-efficient, stable, and adaptive RL training for flow models: (1) Sampling Inefficiency. In practice, each prompt may require the policy to generate dozens of images (e.g., up to 64 ), causing training time and computational cost to grow rapidly. This uniform multi-sample requirement motivates approaches that can reduce redundant sampling while preserving learning signal. (2) Advantage Collapse and Training Instability. When all responses in a group receive similar rewards (e.g., all zero), the relative advantages collapse toward zero, which weakens the learning signal and accelerates entropy collapse 11 . This issue becomes particularly pronounced in later training stages, where the diversity of the policy outputs tends to diminish quickly 1 . This motivates advantage estimation and optimization mechanisms that remain informative even when reward variance is low. (3) Inflexible Sampling Allocation. The fixed group size ignores variation in sampling importance across prompts 51 . At a given step, some prompts exhibit high reward variance while others yield nearly constant rewards. From the GRPO objective, the expected gradient magnitude increases with the variance of returns, so high-variance prompts contribute more to policy improvement. Allocating the same number of samples to both low-variance and high-variance prompts therefore wastes sampling budget on uninformative prompts and under-samples the prompts that matter most for learning, motivating variance-aware and adaptive sampling strategies. Second, current RL training pipelines for flow models suffers from inaccurate advantage estimation 24 ; 49 . In GRPO, trajectory advantages are computed by z-score normalization of rewards within each group. DeepSeekMath 46 alleviates this issue by incorporating process-level rewards under GRPO for language generation tasks 59 . However, for flow-based image generation, process-level rewards are difficult to obtain because intermediate states along the denoising trajectory are visually noisy and semantically ambiguous 60 . Recent methods attempt to estimate the contribution of intermediate states via metrics such as cosine similarity 61 or auxiliary networks 63 , but these approaches yield only marginal improvements and do not fundamentally resolve the bias in advantage estimation. Figure 1 : SuperFlow . Left: Variance-aware group sampling allocates more samples to prompts with higher reward variance and uses a value tracker instead of a per-group mean, reducing wasted samples and equal-advantage cases in Flow-GRPO. Right: Step-level advantage re-estimation along the denoising path. As reward variance decreases over time steps, recomputing advantages at each step yields more accurate credit assignment for flow training. To address these challenges, we propose SuperFlow . We name it SuperFlow since it turns streaming rollouts into effective group-wise comparisons that are well-suited for flow models. SuperFlow is a Streaming-to-Group Sampling framework that improves Single-Stream Policy Optimization (SPO) 57 for flow matching models in text-to-image (T2I) generation. As illustrated in Figure 1 , compared to standard GRPO training of a flow matching model denoted Flow-GRPO , SuperFlow introduces two strategies that directly resolve the above limitations. First, we propose Streaming-to-Group Sampling . Instead of using a fixed group size, we dynamically adjust the number of samples per prompt based on reward variance. This allocates more samples to high-variance prompts and fewer samples to low-variance prompts. As a result, it reduces redundant sampling and improves efficiency, and it removes the fixed, prompt-agnostic sampling rule that causes sampling rigidity. The resulting variance-aware groups also provide more reliable normalization, so relative advantages are less likely to collapse to near-zero. This helps mitigate training instability and entropy collapse in later stages. Second, we propose Step-level Advantage Re-estimation . The second challenge is the bias from trajectory-level advantages in flow training, where one trajectory-level advantage is reused across all denoising steps even though reward statistics change across steps. To address this, we re-estimate advantages at each denoising step using step-dependent statistics, including reward standard deviation. This yields more accurate and consistent credit assignment to intermediate states and improves both training stability and reward alignment. Extensive experiments demonstrate that SuperFlow attains comparable or better performance than prior RL methods while requiring fewer training steps and reduced compute time. We evaluate SuperFlow on multiple T2I tasks with diverse reward types. On standard benchmarks, SuperFlow improves GenEval by 38.4% over SD3.5 - M and 2.8% over Flow - GRPO , improves OCR by 47.2% over SD3.5 - M and 16.0% over Flow - GRPO , and improves PickScore by 4.6% over SD3.5 - M and 1.7% over Flow - GRPO . These results indicate that SuperFlow generalizes across reward types and maintains stable performance under different training objectives. • Streaming-to-Group Sampling for Flow Matching Models. We apply Single-Stream Policy Optimization to flow matching with a Streaming-to-Group strategy to dynamically adjust the number of samples per prompt, demonstrating that online RL can effectively improve flow-based T2I generation. • Advantage Re-estimation. We introduce step-level advantage re-estimation along the denoising trajectory, using reward variability to refine credit assignment in continuous-time flows and accelerate convergence. • Efficiency and Stability. Across three T2I tasks, SuperFlow achieves strong performance with substantially fewer training steps and lower compute cost than standard RL baselines. In particular, SuperFlow improves over SD3.5 - M by 4.6\%\sim 47.2\% and over Flow - GRPO by 1.7\%\sim 16.0\% , while using only 5.4\%\sim 56.3\% of the original training steps and reducing training time by 5.2\%\sim 16.7\% . 2 Related Work Reinforcement Learning for Large Language Models Reinforcement learning has become a standard tool for improving large language models under preference-based or verifiable rewards. Proximal Policy Optimization (PPO) 45 and its variants are widely used for RLHF-style training, but often require a learned critic and careful tuning to maintain stability. Group Relative Policy Optimization (GRPO) 46 offers a critic-free alternative by constructing per-prompt baselines from multiple responses and normalizing rewards