Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimization Ruijie Hao, Longfei Zhang, Yang Dai, Yang Ma, Xingxing Liang *, Guangquan Cheng * Abstract Reinforcement Learning (RL) has proven highly effective in addressing complex control and decision-making tasks. However, in most traditional RL algorithms, the policy is typically parameterized as a diagonal Gaussian distribution, which constrains the policy from capturing multimodal distributions, making it difficult to cover the full range of optimal solutions in multi-solution problems, and the return is reduced to a mean value, losing its multimodal nature and thus providing insufficient guidance for policy updates. In response to these problems, we propose a RL algorithm termed flow-based policy with distributional RL (FP-DRL). This algorithm models the policy using flow matching, which offers both computational efficiency and the capacity to fit complex distributions. Additionally, it employs a distributional RL approach to model and optimize the entire return distribution, thereby more effectively guiding multimodal policy updates and improving agent performance. Experimental trails on MuJoCo benchmarks demonstrate that the FP-DRL algorithm achieves state-of-the-art (SOTA) performance in most MuJoCo control tasks while exhibiting superior representation capability of the flow policy. 1 1 footnotetext: Corresponding author. 1 Introduction Recently, Reinforcement Learning (RL) has achieved remarkable success across a wide range of domains [ 1 , 2 ] . In most existing RL algorithms, the policy is parameterized as a deterministic function or a Gaussian distribution and the return is regarded as the mean [ 3 ] . However, the theoretically optimal policy may exhibit strong multimodality, which cannot be sufficient modeled by deterministic or Gaussian policies, and the return of state-action pairs also exhibit strong multimodality [ 4 ] , which is ignored by most existing algorithms. Restricted policy representations like unimodal Gaussians can trap algorithms in local optimal by exhibiting mode-covering behavior that concentrates probability density in low value regions between distinct high value actions, thereby severely impairing policy performance. Additionally, if one policy cannot cover the full range of optimal solutions in current state, which may also not find the optimal solution in next state. Moreover, two returns with the same mean can have vastly different distributions. Conditioning policy learning solely on the mean thus discards critical information about return distribution, which is essential for distinguishing between actions that are consistently rewarding and those that are merely average in expectation. Consequently, modeling the policy with a unimodal Gaussian distribution and guiding it by the mean of returns is likely to significantly impair policy learning. Lately, the generative model has been widely used in policy modeling for its powerful ability to fit multimodal distributions [ 5 , 6 ] . In offline RL, expert behaviors can exhibit significant variability even in the same state. This necessitates the use of generative models capable of fitting multimodal distributions [ 7 ] . In Online RL, policy also need to be multimodal to fit all theoretically optimal actions. This allows the agent to achieve superior performance by capturing diverse optimal behaviors and adapting more flexibly to dynamic environments [ 8 ] . However, most existing algorithms employ diffusion models as policies, which require numerous steps to generate the target distribution, making it difficult to meet the real-time constraints of complex control tasks. Additionally, the majority of existing algorithms treat the return as a mean value, failing to provide sufficient guidance for policies with complex distributions. Therefore, this paper focuses on how the Policy can be trained as Flow-based policy, which requires only few steps inference to generate actions, and investigates the performance of guiding such policies via distributional RL. In this work, we propose flow-based policy with distributional RL (FP-DRL), a generalized new approach to combine flow-based policy with distributional critic in online RL. A recent work by Wang et al. [ 9 ] only uses the reverse diffusion processes to approximate policy and the optimization objective is to maximize the expected Q-value, which ignores the loss derived from supervised learning of diffusion model. Inspired by this work, We utilize only the network architecture of the flow model to enable the model to generate multimodal distributions, and train it directly using RL objectives, without incorporating a loss function between current actions and target actions. Then, we integrate distributional RL and employ the distributional critic to guide the policy update more effectively. In summary, the key contributions of this paper are the following: 1) We propose to approximate the policy using only the architecture of the Flow Matching (FM) model, training it directly with RL objectives rather than with the objectives derived from supervised learning. 2) We introduce distributional RL and leverage the representation of the full return distribution to provide more discriminative guidance for the multimodal policy. 3) We evaluate the efficiency, generality and performance of our method on the popular MuJoCo [ 10 ] benchmarking. Compared with SAC-Flow [ 11 ] , DSAC-T [ 12 ] , DACER [ 9 ] , SAC [ 3 ] , TD3 [ 13 ] , our approach achieves the SOTA performance. 4) We provide the FP-DRL code written in JAX and Pytorch to facilitate future researchers to follow our work. Section 2 reviews generative policies and distributional RL, motivating our approach. Section 3 outlines the theoretical foundations, including flow matching and distributional RL. Section 4 details the FP-DRL framework, integrating flow-based policies with a distributional critic in Soft Actor-Critic. Section 5 evaluates FP-DRL on MuJoCo benchmarks against baselines and via ablation studies. Finally, Section 6 summarizes our contributions and findings. 2 Related Works 2.1 Generative Policy in RL Diffusion Policy in RL Diffusion models were first employed in offline RL for cloning multiple expert behaviors. Cheng et al. [ 14 ] proposed a visuomotor policy based on conditional denoising diffusion, which achieves superior performance in robotic manipulation tasks by effectively modeling multimodal action distributions and enabling stable training. Wang et al. [ 15 ] introduced Diffusion-QL, which uses a diffusion model as an expressive policy class and integrates Q-learning guidance to achieve effective policy regularization and improvement. The original diffusion models need labels to train, while expert behaviors can serve as labels in offline RL, they are not available in online RL. Wang et al. [ 9 ] proposed the DACER algorithm, which leverages the reverse process of the diffusion model as the policy function, estimates entropy using a Gaussian mixture model, and directly employs Q-values to guide policy updates without requiring labels, making it applicable to online RL. Flow Policy in RL Although diffusion policy can fit complex multimodal distributions, it suffers from low efficiency due to numerous steps sampling, making it inadequate for real-time control tasks. A flow model learns a velocity field, enabling sampling from an initial simple distribution to a complex multimodal distribution in just a few steps, or even a single step [ 16 , 17 ] . Research on flow-based policies is just beginning to emerge. Jiang et al. [ 18 ] proposed Streaming Flow Policy, which treats action trajectories as flow trajectories and enables streaming generation from a narrow Gaussian initialization while preserving multimodality. Zhang et al. [ 11 ] proposed SAC-flow, which stabilizes off-policy training of flow-based policies by reparameterizing the velocity network as a GRU or Transformer to mitigate gradient pathologies. 2.2 Distributional RL Distributional RL In value-based RL Distributional RL captures the full distribution of returns rather than just the expected value, providing a richer learning signal that enables more stable, efficient, and risk-aware policy optimization [ 19 , 20 , 21 ] . The groundbreaking paper of distributional RL is C51 [ 4 ] algorithm proposed by Bellmare et al. which parametrizes the return distribution as a categorical distribution over a discrete set of fixed supports, learning the probability mass per atom through minimization of the KL divergence. Quantile-based methods instead learn where the supports of the distribution should lie. QRDQN [ 22 ] uses a uniform mixture of Dirac delta functions to approximate the return distribution, where each quantile level is assigned equal probability. This formulation allows the minimization of the quantile regression loss, which is an approximation of the 1-Wasserstein distance. Further extensions, such as Implicit Quantile Networks (IQN) [ 23 ] or Fully Quantile Functions (FQF) [ 24 ] enhance the expressiveness of the learned distribution. IQN samples quantile fractions from some underlying base distribution and FQF learns the location of the fractions in an end-to-end manner. Distributional RL With Actor-Critic Integration Recent works have extended distributional RL to continuous control problems using the Actor-Critic (AC) framework [ 25 , 26 , 27 ] . Prior work, such as D4PG [ 28 ] , adapted the categorical projection from C51, but inherited the limitations of fixed supports and the use of KL divergence. More recently, methods like DSAC [ 29 ] and DSAC-T [ 12 ] integrated the AC framework by modeling returns with a Gaussian distribution. However, a critical limitation persists: these methods optimize via Maximum Likelihood Estimation (MLE), which effectively minimizes the KL divergence rather than the contractive Wasserstein metric. Our method is motivated to propose a flow-based policy and a distributional critic that can be combined with most existing actor-critic frameworks. We first use the flow model as a policy function with strong representational power. Then, we construct a distributional critic to provide sufficient guidance for the multimodal distribution policy and improve the performance of the policy. 3 Preliminaries 3.1 Online RL In online RL, the agent interacts with environment in real time, and this interaction can be modeled as a Markov Decision Process (MDP) defined by the tuple (\mathcal{S,A},R,P,\gamma) [ 30 , 31 ] . Here, \mathcal{S} denotes the state space, \mathcal{A} denotes the action space, R denotes the reward function under the state and action, P denotes the transition probability from state \mathcal{S} to \mathcal{S}^{\prime} under action a . \gamma\in(0,1) denotes the reward discount factor. For an agent following a fixed policy \pi , the discounted return is denoted by the random variable: Z^{\pi}(s,a)=\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t}), (1) where s_{0}=s , a_{0}=a , s_{t}\sim P(\cdot|s_{t-1},a_{t-1}) and a_{t}\sim\pi(\cdot|s_{t}) . The action-value function is defined as Q^{\pi}(s,a)=\mathbb{E}\left[Z^{\pi}(s,a)\right] and can be characterized by the Bellman equation: Q^{\pi}(s,a)=\mathbb{E}\left[R(s,a)\right]+\gamma\mathbb{E}_{P,\pi}\left[Q^{\pi}(s^{\prime},a^{\prime})\right]. (2) The objective in RL is to find an optimal policy \pi^{*} , which maximizes \mathbb{E}[Z^{\pi}] and satisfies Q^{\pi^{*}}(s,a)\geq Q^{\pi}(s,a) for all \pi and all s,a . 3.2 Maximum Entropy RL Standard RL aims to maximize the expected cumulative reward [ 32 ] . We consider a more general maximum entropy objective that augments the original objective with the expected entropy of the policy [ 33 ] : J_{\pi}=\mathbb{E}_{(s_{i\geq t},a_{i\geq t})\sim\rho_{\pi}}\left[\sum_{i=t}^{\infty}\gamma^{i-t}[r_{i}+\alpha\mathcal{H}(\pi(\cdot\mid s_{i}))]\right], (3) where \gamma\in(0,1) is the discount factor, \alpha is the temperature coefficient and the policy entropy \mathcal{H} is expressed as \mathcal{H}(\pi(\cdot\mid s))=\mathbb{E}_{a\sim\pi(\cdot\mid s)}[-\log\pi(a\mid s)]. We denote the entropy-augmented accumulated return from s_{t} as G_{t}=\sum_{i=t}^{\infty}\gamma^{i-t}[r_{i}-\alpha\log\pi(a_{i}\mid s_{i})] , also referred to as soft return. The soft Q of \pi is given as Q^{\pi}(s_{t},a_{t})=r_{t}+\gamma\mathbb{E}_{(s_{i>t},a_{i>t})\sim\rho_{\pi}}[G_{t+1}], (4) which defines the expected soft return for choosing a_{t} in state s_{t} and subsequently following policy \pi . We solve for the optimal policy using soft policy iteration, a method consisting of two interacting processes: evaluation and improvement. Soft policy evaluation fits the soft Q-function to the current policy \pi by iterating the Bellman operator \mathcal{T}^{\pi} : \mathcal{T}^{\pi}Q^{\pi}(s,a)=r+\gamma\,\mathbb{E}_{s^{\prime}\sim p,\,a^{\prime}\sim\pi}\big[Q^{\pi}(s^{\prime},a^{\prime})-\alpha\log\pi(a^{\prime}\mid s^{\prime})\big]. (5) Soft policy improvement updates the policy by maximizing the expected soft Q-value. The updated policy \pi{\text{new}} is obtained via: \pi_{\text{new}}=\arg\max_{\pi}\,\mathbb{E}_{s\sim\rho_{\pi},\,a\sim\pi}\big[Q^{\pi_{\text{old}}}(s,a)-\alpha\log\pi(a\mid s)\big]. (6) As established in prior work [ 33 ] , alternating between these evaluation and improvement steps ensures convergence to the global optimum. 3.3 Distributional RL Unlike traditional value-based RL, which estimates the scalar expectation of the cumulative return Q^{\pi}(s,a)=\mathbb{E}[Z^{\pi}(s,a)] , distributional RL seeks to model the full probability distribution of the random return Z^{\pi}(s,a) . The distribution of returns satisfies the distributional Bellman equation: Z^{\pi}(s,a)\overset{D}{=}R(s,a)+\gamma Z^{\pi}(S^{\prime},A^{\prime}), (7) where S^{\prime}\sim P(\cdot|s,a) , A^{\prime}\sim\pi(\cdot|S^{\prime}) and \overset{D}{=} denotes equality in distribution. The corresponding distributional Bellman operator \mathcal{T}^{\pi} is defined as \mathcal{T}^{\pi}Z(s,a){:=}R(s,a)+\gamma Z(S^{\prime},A^{\prime}). (8) For the control setting, the optimal return distribution Z^{*} is the fixed point of the distributional optimality operator \mathcal{T} : \mathcal{T}Z(s,a){:=}R(s,a)+\gamma Z(S^{\prime},\pi^{*}(S^{\prime})), (9) where \pi^{*} is the greedy policy that maximizes the expected return, i.e., \pi^{*}(s^{\prime})=\arg\max_{a^{\prime}\in\mathcal{A}}\mathbb{E}[Z(s^{\prime},a^{\prime})] . 3.4 Flow Matching FM is a family of generative models that learn to match the velocity fields between two probabilistic distributions, where the velocity is regarded as the flow [ 34 , 35 ] . Unlike diffusion models [ 36 ] , which require a two step process of first adding noise and then denoising, FM can directly transition from an initial simple distribution, such as a Gaussian, to a complex target distribution. The process of FM is shown as Fig. 1 . Figure 1: FM learns a velocity fields that transports samples from a simple prior distribution to the target data distribution via an ordinary differential equation (ODE). Given data x\sim p_{\text{1}}(x) and prior \epsilon\sim p_{\text{0}}(\epsilon) , a flow path can be constructed as z_{t}=a_{t}x+b_{t}\epsilon with time t , where a_{t} and b_{t} are predefined schedules. The velocity v_{t} is defined as v_{t}=z^{\prime}_{t}=a^{\prime}_{t}x+b^{\prime}_{t}\epsilon , where ′ denotes the time derivative. The schedule used in Rectified Flow and now widely adopted is a_{t}=1-t and b_{t}=t , which leads to v_{t}=\epsilon-x . Because a given z_{t} and its v_{t} can arise from different x and \epsilon , FM essentially models the expectation over all possibilities, called the marginal velocity: v(z_{t},t)\triangleq\mathbb{E}_{p_{t}(v_{t}|z_{t})}[v_{t}]. (10) A neural network v_{\theta} parameterized by \theta is learned to fit the marginal velocity field: \mathcal{L}_{\text{FM}}(\theta)=\mathbb{E}_{t,p_{t}(z_{t})}\|v_{\theta}(z_{t},t)-v(z_{t},t)\|^{2} . Although computing this loss function is infeasible due to the marginalization in Eq. (10), it is proposed to instead evaluate the conditional FM loss [28]: \mathcal{L}_{\text{CFM}}(\theta)=\mathbb{E}_{t,x,\epsilon}\|v_{\theta}(z_{t},t)-v_{t}(z_{t}\mid x)\|^{2} , where the target v_{t} is the conditional velocity. Minimizing \mathcal{L}_{\text{CFM}} is equivalent to minimizing \mathcal{L}_{\text{FM}} [28]. Given a marginal velocity field v(z_{t},t) , samples are generated by solving an ODE for z_{t} : \frac{d}{dt}z_{t}=v(z_{t},t) (11) starting from z_{0}=\epsilon\sim p_{\text{0}} . The solution can be written as: z_{r}=z_{t}-\int_{r}^{t}v(z_{\tau},\tau)d\tau , where we use r to denote another time step. In practice, this integral is approximated numerically over discrete time steps. For example, the Euler method, a first-order ODE solver, computes each step as: z_{t_{i+1}}=z_{t_{i}}+(t_{i+1}-t_{i})v(z_{t_{i}},t_{i}) . Higher-order solvers can also be applied. 4 Method In this section, we detail the design of our flow-based policy with distributional RL. First, we use FM as policy approximator, serving as the policy function in RL. Second, we directly optimize the flow-based policy using gradient descent, whose objective function is to maximize expected Q-values. At this point, the return is still regarded as the mean. However, the mean loses critical information such as risk and multimodality; only the full return distribution can provide all the necessary guidance for risk-aware, robustly optimized, and stably learned policies. Consequently, we employ a quantile-based approach to model the full return distribution and utilize distributional RL to update the returns. 4.1 Flow-based Policy Representation We conceptualize the flow-based policy as a conditional FM conditioned on states S. Starting from an initial Gaussian distribution p_{0}(\cdot\mid s)\sim\mathcal{N}(0,I) on an action space \mathcal{A}=\mathbb{R}^{d} , the target policy p_{1}(\cdot\mid s) is reached via a time-indexed mapping \varrho:[0,1]\times\mathcal{A}\times\mathcal{S}\to\mathcal{A} , with A_{t}:=\varrho_{t}(A_{0}\mid s) for t\in[0,1] , where A_{0}\sim p_{0}(\cdot\mid s) and A_{1}\sim p_{1}(\cdot\mid s) . The trajectory satisfies the ODE \frac{\mathrm{d}}{\mathrm{d}t}\varrho_{t}(A_{0}\mid s)=v(t,\varrho_{t}(A_{0}\mid s),s) , where v is the velocity field to be learned. The learned field is integrated numerically with flow rollout to obtain: A_{t_{i+1}}=A_{t_{i}}+\Delta t_{i}\,v_{\theta}(t_{i},A_{t_{i}},s),\quad 0=t_{0}<\cdots