Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent
Zeyuan Wang, Yulin Chen, Yuehu Gong, Yanming Guo, Ye Shi, Liang Bai, Tianyuan Yu, Yanwei Fu, [2mm] Laboratory for Big Data, Decision, National University of Defense Technology, China · 2026
Read Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent by Zeyuan Wang, Yulin Chen, Yuehu Gong, Yanming Guo, Ye Shi, Liang Bai, Tianyuan Yu, Yanwei Fu, [2mm] Laboratory for Big Data, Decision, National University of Defense Technology, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21282.txt.
Introduction
Zeyuan Wang and colleagues introduce Stochastic MeanFlow Policies, a one-step generative policy class for entropy-regularized mirror descent in off-policy reinforcement learning. Gaussian policies struggle with multimodal targets, while generative policies often need iterative sampling or lack tractable entropy estimates. Their MeanFlow noise-to-action mapping captures diverse action modes, and a learnable Gaussian scale gives an analytic entropy lower bound. That bound regulates stochasticity alongside value guidance and advantage-weighted regression. Across seven MuJoCo benchmarks, the one-step policies perform strongly against Gaussian and generative baselines.
Catalog record
| CREATOR | Zeyuan Wang, Yulin Chen, Yuehu Gong, Yanming Guo, Ye Shi, Liang Bai, Tianyuan Yu, Yanwei Fu, [2mm] Laboratory for Big Data, Decision, National University of Defense Technology, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66824 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21282
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent Zeyuan Wang, Yulin Chen, Yuehu Gong, Yanming Guo, Ye Shi, Liang Bai, Tianyuan Yu, Yanwei Fu, [2mm] Laboratory for Big Data, Decision, National University of Defense Technology, China Abstract Online off-policy reinforcement learning (RL) is governed by two coupled design dimensions: policy representation (e.g. Gaussian vs generative policies) and optimisation methodology (e.g. soft actor-critic (SAC) vs mirror descent (MD)). Gaussian policies offer fast inference and tractable entropy estimation but have limited ability to model multimodal action distributions, whereas generative policies provide richer action distributions at the cost of iterative sampling or intractable entropy estimation. On the optimisation side, SAC-style entropic exploration and MD updates can be interpreted as minimising distinct Kullback-Leibler divergences: SAC-style exploration performs soft policy improvement towards a value-induced Boltzmann distribution, whereas MD constrains updates to remain close to the previous policy. The combination of SAC-style entropy and MD constraint provides a complementary framework that enables efficient exploration and stable policy updates, yet yields a multimodal target that unimodal Gaussian policies fail to represent. This representation mismatch raises a research question of whether generative policies can fill this gap. To this end, we introduce Stochastic MeanFlow Policies (SMFP), a one-step generative policy class that combines MeanFlow-based noise-to-action mappings with Gaussian reparameterisation. This stochastic reparameterisation enables the derivation of a tractable entropy surrogate for MeanFlow policies while remaining integrable with off-policy mirror descent, yielding a unified objective for exploratory yet stable policy improvement. Worthy noting, SMFP not only makes entropy-regularised mirror-descent RL practically effective for the first time but also preserves the advantages of both policy representations. Empirically, on seven MuJoCo benchmarks, SMFP achieves strong performance against both Gaussian and generative baselines while maintaining single-step inference efficiency. 1 Introduction Online off-policy reinforcement learning (RL) has become a dominant paradigm for solving high-dimensional continuous control tasks, ranging from robotic manipulation to autonomous navigation [ 28 , 18 , 35 , 50 ] . Its success is largely governed by two key design axes: policy representation ( how actions are modelled ) and optimisation methodology ( how policies are updated ). Along the axis of policy representation, a fundamental trade-off arises between tractability and expressivity. Standard approaches rely on unimodal Gaussian policies [ 11 , 20 ] , which provide closed-form likelihoods, efficient entropy estimation, and fast inference, but are fundamentally restricted to Figure 1 : An illustration of SMFP Performance vs. Inference Speed. Evaluated on an NVIDIA RTX 5090 across Ant-v4, SMFP achieves strong performance with negligible inference latency. unimodal action distributions. In contrast, diffusion and flow-based policies [ 21 , 36 , 12 , 6 , 14 , 60 ] offer substantially greater expressivity and can model complex multimodal behaviours. However, these benefits come at the cost of intractable likelihoods and entropy estimation, as well as high inference latency due to iterative sampling [ 22 , 31 ] . Recent one-step variants, such as MeanFlow policies [ 56 ] , alleviate sampling overhead but are still incompatible with explicit entropy control. Along another axis of optimisation methodology, a similar tension arises between exploration-oriented entropy maximisation, as in SAC [ 20 ] , and stability-focused trust-region methods such as mirror descent [ 54 , 52 ] . Entropy-regularised approaches maximise a soft value function augmented with an entropy term, promoting broad exploration and mitigating premature convergence. However, they typically rely on soft maximisation and lack explicit control over policy updates, which can lead to instability during rapid learning. In contrast, mirror-descent-based methods [ 2 , 45 ] impose KL constraints that ensure each update remains close to the previous policy, thereby ensuring stable and conservative improvement. Fundamentally, these two types of optimisations can be interpreted as minimising distinct Kullback-Leibler divergences [ 52 ] , capturing complementary aspects of policy optimisation: exploration vs stability. Can SAC-style exploration be combined with the stability of mirror-descent updates? While these objectives are naturally complementary, their combination induces a multi-modal target distribution (as illustrated in Figure 2 ), arising from the competing pulls of value maximisation and trust-region regularisation. However, standard Gaussian policies are inherently unimodal and therefore fail to represent such targets. This mismatch between optimisation objectives and policy representation raises a fundamental question: can expressive generative policies bridge this gap? Figure 2 : Illustration of the composite SAC-MD target and induced multi-modal structure. Entropy-regularised objectives, such as SAC, require a tractable estimate of policy entropy. However, for expressive generative policies, exact likelihoods and entropies are generally intractable and often rely on costly estimators or trajectory-level approximations [ 23 , 6 , 59 ] . In contrast, diagonal Gaussian parameterisations [ 27 , 46 , 58 ] enable efficient stochastic sampling and closed-form entropy control, suggesting a principled route to bridge generative expressivity with entropy-driven optimisation. To this end, we introduce Stochastic MeanFlow Policies (SMFP), a class of one-step stochastic generative policies that combines MeanFlow-based noise-to-action mappings [ 56 ] with Gaussian reparameterisation. This stochastic reparameterisation enables the derivation of a tractable entropy surrogate, thereby eliminating the intractability of entropy control in vanilla MeanFlow [ 16 ] . The derived entropy surrogate is compatible with off-policy mirror descent, yielding a unified objective that supports both effective exploration and stable policy updates. Worthy noting, SMFP makes entropy-regularised mirror-descent RL practically effective for the first time, while preserving the complementary advantages of expressive generative and efficient Gaussian policies. Empirically, SMFP achieves strong performance across seven MuJoCo continuous-control benchmarks, matching or outperforming iterative diffusion and flow baselines while retaining efficient one-step inference. Notably, while prior work suggests that combining SAC and mirror descent is ineffective under Gaussian policy parameterisations [ 39 ] , our results demonstrate that this combination becomes beneficial when supported by expressive generative policies. 2 Related Works Maximum Entropy Reinforcement Learning . Maximum entropy (MaxEnt) RL, standardised by algorithms like SAC [ 20 ] , balances exploration and exploitation by augmenting the reward objective with entropy regularisation. While recent innovations such as CrossQ [ 4 ] and BRO [ 37 ] have significantly improved sample efficiency via batch renormalisation [ 24 ] and aggressive updates, they predominantly rely on Gaussian policies that limit the modelling of complex action distributions [ 9 , 40 ] . To address this expressivity gap within the MaxEnt framework, the field has evolved from computationally intensive particle-based sampling methods, ranging from SVGD [ 19 ] to the recent LSAC [ 25 ] , toward expressive generative policies. This emerging landscape includes diffusion-based algorithms like DIME [ 6 ] , DACER [ 55 ] , and MaxEntDP [ 14 ] that approximate score functions or entropy surrogates, alongside flow-based methods such as ReinFlow [ 59 ] and SAC Flow [ 60 ] that leverage velocity reparameterisation or noise injection for tractable entropy maximisation. Recent FLAC [ 33 ] further avoids explicit action-density estimation via kinetic-energy-regularised bridge matching, but uses path-space regularisation rather than explicit action-entropy control. Mirror Descent Policy Optimisation . A prominent lineage of algorithms, notably TRPO [ 47 ] and PPO [ 48 ] , guarantees stable improvement by enforcing trust-region constraints via Kullback-Leibler (KL) divergence. This paradigm is rigorously unified under the framework of Mirror Descent Policy Optimisation (MDPO) [ 52 , 38 , 53 ] ,