FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification

Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lei Wang, Chang Liu, Siming Fu, [1ex] Beihang University, Joy Future Academy, Zhongguancun Academy, Zhejiang University · 2026

Read FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification by Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lei Wang, Chang Liu, Siming Fu, [1ex] Beihang University, Joy Future Academy, Zhongguancun Academy, Zhejiang University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30376.txt.

Introduction

Zheming Fu and six colleagues speed up reinforcement learning for flow-based image generators. Flow models generate along continuous trajectories, not discrete tokens. Their step-by-step likelihoods cannot be computed exactly. Existing fixes use stochastic samplers that clash with training. Others correct the velocity field by a fixed heuristic amount. FlowAWR replaces heuristics with a derived optimum.

The method starts from the best policy under a divergence penalty. It derives the optimal velocity field for that policy. The field takes a rectification form weighted by advantage. Better samples get stronger corrections, scaled by their quality. Optimization needs no stochastic differential equations at all. Generation needs no classifier-free guidance either. On a mid-size image model, alignment scores improve clearly. Convergence runs two to five times faster than the main rival. Multi-reward tests keep quality while obeying structural rules.

Catalog record

CREATORZheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lei Wang, Chang Liu, Siming Fu, [1ex] Beihang University, Joy Future Academy, Zhongguancun Academy, Zhejiang University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lei Wang, Chang Liu, Siming Fu, [1ex] Beihang University, Joy Future Academy, Zhongguancun Academy, Zhejiang University Abstract Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG). While implicit frameworks such as DiffusionNFT directly optimize forward-process velocity fields, its heuristic fixed-magnitude corrections prevent optimization strength from relative intra-group quality. We propose Flow Advantage-Weighted Rectification ( FlowAWR ), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field. Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization and CFG-free generation. In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2 × \times to 5 × \times convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in > > 4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains generation quality, satisfying structural rules while maintaining stable out-of-domain performance. 1 Introduction The empirical success of Reinforcement Learning (RL) in the post-training of Large Language Models (LLMs) ( 4 ; 50 ; 1 ; 6 ; 46 ) has motivated its extension to continuous generative spaces. Within this domain, Flow Matching ( 24 ; 26 ; 39 ) has emerged as a prevalent paradigm, offering a deterministic trajectory framework for modern diffusion models ( 9 ; 20 ; 37 ; 42 ) . However, adapting RL to this continuous space introduces a structural constraint: the approximation of trajectory likelihood. Unlike AR models that compute exact discrete token probabilities, continuous flow models inherently lack tractable step-wise transition densities ( 40 ) . Ranging from offline applications of Direct Preference Optimization (DPO) ( 41 ; 47 ) to recent online policy gradient approaches ( 25 ; 45 ) , this likelihood bottleneck consistently compromises the computational efficiency and theoretical guarantees of RL on continuous spaces. To circumvent this likelihood barrier, a prevalent strategy discretizes the continuous generative process into a multi-step Markov Decision Process (MDP) ( 3 ; 10 ) . Building upon this, recent GRPO-based methods ( 25 ; 45 ) inject stochasticity via SDE samplers to construct tractable Gaussian transition kernels, enabling the evaluation of policy gradients. Although subsequent improvements attempt to interleave ODE and SDE steps ( 23 ) , this paradigm fails to resolve the forward inconsistency between stochastic training trajectories and deterministic inference rollouts. Furthermore, by failing to implicitly distill the guidance signal into the vector field, these methods retain a structural reliance on CFG ( 16 ) to maintain generation quality, thereby imposing computational overhead during inference. An alternative paradigm leverages implicit optimization and velocity field guidance to bypass explicit density estimation. Early reward-weighted methods ( 21 ; 7 ) are typically restricted to offline settings and lack explicit penalization mechanisms for low-quality samples. Recent online RL frameworks, such as DiffusionNFT ( 49 ) , have reformulated policy optimization as a supervised velocity regression task, achieving notable improvements in both training efficiency and generation performance over GRPO-style methods. However, it relies on a heuristic definition of velocity correction direction and applies modifications with a fixed magnitude. Furthermore, its utilization of intra-group samples is restricted to computing a scalar normalized reward, which merely balances the weights between preset positive and negative objectives. Consequently, this simultaneous exploration operates strictly at the individual sample level, thereby failing to leverage relative intra-group quality to directly determine the magnitude of velocity updates. Herein, we propose Flow Advantage-Weighted Rectification ( FlowAWR ), an online RL paradigm for continuous generative spaces that operates without stochastic trajectory formulation or heuristic field modifications. Our framework originates from the exact closed-form optimal policy derived from reward maximization under a temporal KL divergence constraint. By mapping this optimal terminal distribution onto the intermediate probability paths, we analytically derive the corresponding optimal velocity field. We demonstrate that this optimal field is equivalent to rectifying the reference velocity field by an advantage-weighted expectation. Consequently, FlowAWR recasts flow RL as a supervised regression task toward this theoretical optimum. By incorporating a group-based advantage function, our framework quantifies the continuous relative quality of intra-group samples, replacing rigid binary push-pull operations with magnitude-aware rectifications. Our method preserves the architectural benefits of likelihood-free approaches by eliminating the forward inconsistency induced by SDE sampling, while its advantage-weighted updates implicitly distill alignment trajectories into the network parameters to enable CFG-free generation. To evaluate the efficacy of FlowAWR, we conduct systematic experiments on the SD3.5-Medium (2.5B parameters) model ( 9 ) . Quantitative results indicate that, in the single-reward setting, FlowAWR achieves a 2\times to 5\times convergence acceleration alongside improved performance compared to DiffusionNFT. For instance, when optimizing for PickScore, FlowAWR elevates the metric to 24.12 within 1.2k steps. In contrast, DiffusionNFT requires 2.0k steps to reach 23.82, and FlowGRPO necessitates over 4k steps while relying on CFG to achieve 23.50. Furthermore, under the complex distribution shifts induced by multi-reward optimization, our decoupled expert-branching strategy maintains optimization stability and mitigates out-of-domain aesthetic degradation. Our primary contributions are summarized as follows: • We introduce FlowAWR , an online flow RL framework that executes exact policy updates without SDE simulation. Our derivation validates the heuristic correction direction of DiffusionNFT while demonstrating that it constitutes a binary quantized special case . • We characterize the optimal velocity field by propagating the closed-form optimal policy to intermediate optimal distributions along the probability path, ultimately instantiating this theoretical optimum via a group-based approximated advantage function . • We conduct quantitative evaluations across both single- and multi-objective optimization settings, indicating that FlowAWR accelerates convergence rates and improves empirical alignment performance compared to baseline methodologies. 2 Background 2.1 Flow Matching and Rectified Flow Given a prior noise distribution p_{0} and a target data distribution p_{1} , Flow Matching ( 24 ) constructs a continuous probability path p_{t}(x) for t\in[0,1] generated by a marginal vector field u_{t}(x) . Flow Matching constructs the target field via conditional probability paths p_{t}(x|x_{1}) and corresponding conditional vector fields u_{t}(x|x_{1}) . Specifically, adopting the Rectified Flow ( 26 ) formulation, the forward process defines a linear interpolation between noise x_{0}\sim p_{0} and data x_{1}\sim p_{1} : x_{t}=tx_{1}+(1-t)x_{0}, which induces a constant conditional vector field, u_{t}(x_{t}|x_{1})=x_{1}-x_{0}=\frac{x_{1}-x_{t}}{1-t} . The parameterized network v_{\theta}(x_{t},t) is then trained to regress this conditional vector field via the standard flow matching objective: \mathcal{L}_{\text{FM}}(\theta)=\mathbb{E}_{t\sim\mathcal{U}[0,1],x_{1}\sim p_{1},x_{0}\sim p_{0}}\left[\|v_{\theta}(x_{t},t)-u_{t}(x_{t}|x_{1})\|^{2}\right]. (1) The global minimum of this objective yields the optimal marginal velocity field, which is explicitly formulated as the expectation of the conditional fields over the posterior data distribution ( 24 ; 2 ) : v(x_{t},t)=\mathbb{E}_{x_{1}\sim p(x_{1}|x_{t})}\left[\frac{x_{1}-x_{t}}{1-t}\bigg|x_{t}\right]=\frac{\hat{x}_{1}(x_{t})-x_{t}}{1-t}, (2) where \hat{x}_{1}(x_{t}) denotes the posterior mean estimation of the data x_{1} given the intermediate state x_{t} . 2.2 Reinforcement Learning as Supervised Fine-tuning A distinct paradigm seeks to bypass the complexities of policy gradients by treating RL as a supervised regression task. DiffusionNFT ( 49 ) grounds this in the decomposition of the reference policy \pi^{\mathrm{old}} into positive ( \pi^{+} )

Read the complete text