Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent ∗ Corresponding author Nicole Bäuerle, Athanasios Vasileiadis ∗ Abstract . The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law, rather than optimizing an expected value or a risk functional. To solve the resulting distributional control problem in a model-free setting, we propose a policy-gradient algorithm based on neural-network parameterizations of randomized Markov policies, defined on an augmented state space and a sample-based evaluation of the characteristic-function loss. Under mild regularity and growth assumptions, we prove convergence of the algorithm to stationary points using stochastic approximation techniques. Several numerical experiments illustrate the ability of the method to match complex target distributions, recover classical optimal policies when they exist, and reveal intrinsic non-uniqueness phenomena specific to distributional control. Key words : Distributional MDPs, Policy Gradient, Randomized controls 1. Introduction Since the introduction of Reinforcement Learning (RL), tremendous progress has been achieved in designing algorithms that allow agents to learn optimal behavior through interaction with uncertain environments. Markov Decision Processes (MDPs) has been so far the theoretical framework to model these problems, 35 ; 14 ; 33 ; 10 . When all the involved functions i.e. the dynamics and rewards/costs, are known and the spaces finite, we can use exact dynamic programming methods usually in a tabular form. When the spaces are continuous we need some form of function approximation. For what concerns this present article, RL is what we do when we only observe samples from the states actions and rewards, without access to any part of the theoretical model. Classical RL methods focus on estimating the expected cumulative return, the classical value function of control, which has led to impressive successes across domains ranging from notoriously hard games like chess and GO to modern life robotics. We call this mathematical formulation MDPs of the first kind . However, not all problems we are facing can be solved by the classical value functions, for example portfolio selection under model uncertainty, robust control and in general risk aware optimization, 19 ; 7 ; 34 ; 16 ; 17 ; 9 ; 24 call for a different formulation under coherent risk measures and a time consistent dynamic programming principle, 4 . We call collectively this mathematical formulation MDPs of the second kind . Finally, motivated by recent advances in finance, 28 ; 18 and AI (especially generative AI), we are interested in optimal controls that stir the whole distribution of (discounted) cumulative rewards towards a certain target distribution. We call these (Distributional) MDPs, MDPs of the third kind . In this article we restrict to problems with a finite time hoizon. In complete analogy, Distributional Reinforcement Learning (DRL) addresses the problem in a practical way, using only samples by modeling the entire distribution of returns, providing a richer representation that can enhance stability, performance, and risk-sensitive decision making. 1.1. Motivation To motivate our results we start by giving an answer to a simple variant of our problem that we can actually solve analytically, so we get a flavor of what to expect in more complicated and general cases that we treat in the rest of the article. Problem 1 . Find optimal controls that shift the distribution of a terminal cumulative (discounted) reward (cost) such that it matches a predefined target. On a compact space, the Fourier representation of a probability measure is a sequence of coefficient and matching this is a finite dimensional problem. In particular we can match directly by identifying the coefficients of sequence. To illustrate this, consider \mathbb{T}^{1}=\mathbb{R}/\mathbb{Z} , the one dimensional torus endowed with its Borel \sigma -algebra. For a finite horizon T\in\mathbb{N} consider the Markov Decision Process s_{t+1}=\bigl(s_{t}+a_{t}+\varepsilon_{t+1}\bigr)\bmod 2\pi,\qquad t=0,\dots,T-1, where s_{0}\in\mathbb{T}^{1} is an \mathcal{F}_{0} -measurable initial condition with distribution \mu_{0} . The reward is given by r_{t}(s_{t},a_{t})=0 for t=0,1,\ldots,T-1 and r_{T}(s_{T},a_{T})=s_{T}. Next, a_{t}\in\mathcal{A}\subset\mathbb{T}^{1} is the action chosen by the decision maker at stage t which may depend on the observed history of the process. We consider randomized policies here. Finally, (\varepsilon_{t})_{t\geq 1} is an i.i.d. white noise sequence on \mathbb{R} , independent of s_{0} . All sums are taken in \mathbb{R} and reduced modulo 2\pi to lie again in \mathbb{T}^{1} . The objective is to control the process in such a way that the distribution \mu_{T} of the accumulated reward \sum_{t=0}^{T}r_{t}(s_{t},a_{t})=s_{T} is as close as possible to a given target distribution \mu^{*} on \mathbb{T}^{1} . The distance is measured as a distance of characteristic functions. We will be more precise later. For the purpose of illustration let us assume that T=1 . In this case we can compute the characteristic function of s_{1} explicitly. First, consider y_{1}=s_{0}+a_{0}+\varepsilon_{1}. Then the law of y_{1} is a Gaussian centered at s_{0}+a_{0} if a_{0} is deterministic and if a_{0} is stochastic with distribution \nu then \mathbb{P}(y_{1}\leq z)=\int_{\mathbb{R}}\Phi(\frac{z-x_{0}-u}{\sigma})d\nu(u) i.e. the distribution of y_{1} is given by the convolution \delta_{x_{0}}*\nu*\Phi. Second, we need to consider the nonlinear transformation w:\mathbb{R}\to\mathbb{T}^{1} given by y\bmod{2\pi}=w(y)=y-2\pi\lfloor\frac{y}{2\pi}\rfloor. In such a situation, if y\sim\mathcal{N}(m,\sigma^{2}) on \mathbb{R} and s\;=\;w(y)\;\in\;\mathbb{T}^{1}, then the density of s with respect to Lebesgue measure on [0,2\pi) is the wrapped Gaussian (or circular normal ): \Phi^{\text{wrap}}(x;m,\sigma^{2})\;=\;\frac{1}{\sqrt{2\pi\sigma^{2}}}\sum_{k\in\mathbb{Z}}\exp\!\Bigl(-\frac{(x-m+k)^{2}}{2\sigma^{2}}\Bigr)=\frac{1}{\sqrt{2\pi}}\Bigl\{1+2\sum_{k=1}^{\infty}e^{\frac{k^{2}\sigma^{2}}{2}}\cos\bigl(2k\,(x-m)\bigr)\Bigr\}. Thus, wrapping the distribution in \mathbb{T}^{1} gives \mathbb{P}(s_{1}\leq z)=\int_{\mathbb{T}^{1}}\Phi^{\text{wrap}}(\frac{z-x_{0}-u}{\sigma})d\nu(u). When we want to match a certain characteristic function we can do it by choosing \nu accordingly. Note that the characteristic function of a distribution \mu on \mathbb{T}^{1} is determined by the sequence \hat{\mu}(n):=\int_{\mathbb{T}^{1}}e^{-inu}\,d\mu(u) for n\in\mathbb{Z} , see for example ( 36 , Chapter 1) . Further note that the characteristic function of the wrapped Gaussian density centered at s_{0} is given by \widehat{\Phi}^{\text{wrap}}(n;s_{0},\sigma^{2})=e^{-ins_{0}}e^{-\frac{1}{2}\sigma^{2}n^{2}},\quad n\in\mathbb{Z}. Thus, identifying the characteristic function of s_{1} with the one of target distribution \mu^{*} yields \mu^{*}(n)=\widehat{\mu}_{1}(n) for all n\in\mathbb{Z} . Using the convolution property of the characteristic function and solving for the policy that we want to determine yields \widehat{\nu}(n)=\frac{\widehat{\mu}^{*}(n)}{\widehat{\Phi^{\text{wrap}}_{s_{0},\sigma}}(n)}=\widehat{\mu}^{*}(n)\cdot e^{ins_{0}}\;e^{\frac{1}{2}\sigma^{2}n^{2}}\quad\forall\,n\in\mathbb{Z}. It is relatively obvious from the discussion above that when we are considering only deterministic controls we just shift the initial condition without any hope to match any nontrivial target distribution. 1.2. Extension to Continuous Non Compact Spaces Of course as illustrative and appealing as it is, this example is of limited use and we present it only for pedagogical reasons. The main question is how to extend to continuous non compact Borel spaces with non linear dynamics? When we cannot trace the functional form of the distribution of controls since we don’t have anymore convolutions and the Fourier representation cannot be broken down analytically. To motivate our choices in Section 2 we first notice that to extend the problem in case of continuous non compact spaces we need to define a distance between the characteristic functions of the reward R_{T}=\sum_{k=0}^{T-1}r_{k}(s_{k},a_{k}) and a target characteristic function \varphi^{*} : \int_{\mathbb{R}}\big|\varphi^{*}(u)-\mathbb{E}[e^{iuR_{T}}]\big|^{2}\,w(u)\,du, with some weight function w (e.g. Gaussian) to make the integral finite and emphasize relevant frequencies. A natural choice for working directly with probability measures could also be the Wasserstein distance but with a serious caveat that it is not differentiable out of the box and thus not easy to use as a loss function for our purposes. The theoretical foundations of problems like this have been laid in 8 . The authors there give a dynamic programming equation for the solution. However, the question still remains how these problems can numerically be solved efficiently. Thus, in this paper we use artificial neural networks (ANNs) to parametrize the controls where we condition them on the current state and reward and using some external noise in order to learn to sample optimally from the unknown action distribution \nu when we have no access and or assumption on it. The ANN parameters are trained by a gradient descent scheme. To wrap up, we design a model free policy gradient algorithm using only samples from the dynamics and rewards to minimize a characteristic function loss. 1.3. Choosing the Target There are several ways to choose the target distribution. This may either be a desired risk-return distribution when we think about investment problems where a certain risk profile is targeted. Another setting could be the problem of imitating others: Suppose you can observe the rewards/outcome of an opponent but you do not know her/his actions. Then you can easily obtain her/his characteristic target function of cumulative rewards and use this to find out a control which is consistent with this behavior. A third application is to design a Markov model such that the behavior in terms of cumulative rewards behaves as observed in nature. 1.4. Literature Review Distributional Reinforcement learning has gained in recent years a lot of attraction with several important contributions. One of the most influential advances is the quantile-regression formulation of 20 , which replaces the fixed-support categorical approximation of C51 from the seminal paper 11 with a trainable quantile parameterization. This representation yields a theoretically consistent approximation of value-return distributions in the 1-Wasserstein metric and resolves several limitations of earlier approaches, most notably by providing unbiased stochastic gradient estimates via asymmetric quantile losses and by ensuring that the projected Bellman operator is a contraction. Further along this line of research on value-based control the article of 13 introduces the first convergence guarantees for a distributional RL algorithm combined with function approximation, addressing a central theoretical gap left open by earlier works such as C51 and its analysis by 11 . Its key contribution is the construction of a fully Cramér-based loss free of softmax and KL divergence which extends the Cramér distance to arbitrary real vectors and augments it with a normalization penalty, enabling a new algorithm (S51) whose updates are mathematically tractable. Using this generalized loss, the authors prove that the projected distributional Bellman operator converges under linear function approximation and quantify the resulting approximation error, showing, perhaps unexpectedly, that distributional methods can yield worse expected-value accuracy than classical TD under function approximation. In 1 , the authors introduce a simplified distributional framework that resolves one of the central theoretical difficulties of classical distributional RL: the instability of the optimality operator in the control case. By restricting attention to the randomness induced by the first transition only, the authors construct one-step distributional Bellman operators that unlike the original distributional operators are \gamma -contractions in all Wasserstein metrics for both policy evaluation and control, leading to a unified almost-sure convergence theory even without assuming uniqueness of the optimal policy. Using these operators, they develop new Cramér-projected algorithms (tabular and deep variants such as OS-C51) whose categorical updates are simpler, cheaper, and provably convergent. Nevertheless, this line of work remains fundamentally oriented toward value-based control : the goal is to estimate the distribution of the return induced by a fixed policy and to extract improved greedy or risk-sensitive actions from it. By contrast, the problem studied in this article is of a different nature: instead of estimating an endogenous return distribution, we aim to steer the distribution of a terminal random variable toward a prescribed target law by directly optimizing the policy. Whereas quantile-based DRL methods rely on Bellman recursions, additive rewards, and the value-function framework, our approach operates on a lifted state space of probability measures and optimizes a characteristic-function divergence akin to a policy iteration method. We show in Example 5.4 that our framework is general and flexible enough that it can accommodate solutions to some classical RL problems with minimal modifications. Recent work has extended distributional reinforcement learning to the continuous-time setting by characterizing the evolution of return distributions through a distributional analogue of the backward Kolmogorov equation. In 40 the authors derived what they call the distributional HJB equation, but which is in fact a linear backward Kolmogorov PDE for the state-conditioned return CDF under a fixed policy, thereby providing the first continuous-time description of return distributions beyond expectations. To make this infinite-dimensional PDE numerically tractable, they introduced the statistical HJB (SHJB) loss, which replaces the full distribution by a finite set of statistics (e.g., quantiles) and measures the discrepancy between the approximate CDF and the PDE operator. However, that work did not establish whether minimizing the SHJB loss actually produces consistent approximations of the true return distribution. This gap is filled by 2 , who prove that if the chosen imputation strategy (mapping finite statistics to a distribution) satisfies a mild topological condition, namely convergence in the sense of tempered distributions then SHJB minimization is provably convergent: as the number of statistics increases, the approximate solutions converge to the true solution of the Kolmogorov PDE. Moreover, they show that the widely used quantile representation satisfies this condition and achieves an \mathcal{O}(\frac{1}{N}) approximation error for the CDF, thereby giving the first rigorous justification that quantile-based continuous-time DRL is a sound and convergent discretization scheme for the distributional Kolmogorov equation. In contrast to these works, which focus primarily on estimating the return distribution under a policy (or its continuous-time evolution), 8 ; 31 introduce distributional dynamic programming, a general DP framework for optimizing arbitrary statistical functionals of the return distribution . Their approach is the closest to ours in this article. We agree on the definition of the distributional MDP which augments states with cumulative reward (stock) and combines with a distributional Bellman recursion defined over this augmented MDP, enabling principled optimization of objectives such as quantiles, CVaR, and other risk-sensitive functionals. In this way, distributional DP broadens the scope of distributional RL by integrating distributional objectives directly into the dynamic programming principle, complementing earlier work on distributional estimation in both discrete and continuous time. In contrast with 31 , like in 8 we resort to a lifted distributional MDP (see Section 2.2 ) to formally describe the Bellman recursion and identify the class of optimal policies, mainly for two reasons, first one being the deterministic nature of the lifted MDP and the time homogeneous Markovian structure of the augmented MDP. Last but not least we mention two works that connect stochastic control with optimal transport and are very close in spirit to ours but different in methods. In 3 the authors study time series generation through the Schrödinger Bridge (SB) framework, which formulates generative modeling as a stochastic control problem on path space: one selects a drift process minimizing a quadratic control (relative-entropy) cost with respect to a Brownian reference measure, subject to matching prescribed marginal distributions of the state process at given time points. The main conceptual difference lies in the location of the distributional objective: while the SB approach enforces distributional constraints directly on the state process itself, we consider controlled dynamics augmented with a cumulative reward variable and impose the distribution-matching objective on this derived random quantity rather than on the state trajectory. In this sense, rewards act as an additional layer on top of the dynamics, transforming path distributions into outcome distributions. Unlike the SB formulation, we are not aware of any optimal transport or entropic regularization interpretation on path space for our loss. Nevertheless, 3 is very interesting in conjunction with 40 for a general bridge interpretation. In 38 the authors consider fleets of identical agents whose dynamics are non-interacting, each agent evolves independently according to a common controlled transition map, and the population state evolves as the pushforward of the current distribution through this single-agent dynamics. Any coupling between agents arises exclusively through the objective function, via optimal transport discrepancies that compare the induced population distribution to a prescribed reference measure. As a result, the separation principle identified in that work critically relies on this absence of interaction in the dynamics. Here, we stress once more the difference that we operate in a lifted distributional MDP and rely on a direct policy method. However we find the connection with optimal transport very interesting at least in the case of value iteration. 1.5. Summary and Organisation of the Article The remainder of the article is organized as follows. In Section 2 , we formally introduce the distributional control problem, define the characteristic–function matching objective, and reformulate it as a lifted distributional Markov Decision Process on an augmented state space. We also specify the class of randomized Markov policies considered and state the standing assumptions on the dynamics, rewards, and neural-network parametrization. Section 3 presents the policy-gradient algorithm for distribution matching, including the discretization of the Fourier domain, the construction of an unbiased (up to finite-sample effects) stochastic gradient estimator, and the full learning procedure. In Section 4, we establish our main theoretical result, proving convergence of the proposed algorithm to a stationary point under standard stochastic approximation conditions. Section 5 illustrates the methodology on a series of numerical examples, namely linear–quadratic control, an investment problem, a compactly supported distribution and classical benchmark MDPs, highlighting both the flexibility and the intrinsic non-uniqueness of distributional optimal controls. Technical proofs and auxiliary results are collected in the Appendix. 2. The Problem and Assumptions 2.1. Problem Formulation The main purpose of this section is to give a formal definition to our general problem. To this end, we assume a finite time horizon T\in\mathbb{N} and let us denote the state space and action space with \mathcal{S} and \mathcal{A} , respectively. We assume that both are Borel subsets of \mathbb{R} . Let (\Omega,\mathcal{F},\mathbb{P}) be a complete probability space and (\varepsilon_{t})_{t=0}^{T-1} a sequence of i.i.d. standard normal random variables defined on this space. For ease of notation we consider a stationary model here, but the extension to non-stationary systems is immediate. The system evolves according to the stochastic dynamics: s_{t+1}=F(s_{t},a_{t},\varepsilon_{t+1}),\quad\varepsilon_{t+1}\sim\mathcal{N}(0,1), (1) with initial state s_{0} deterministic or random \mathcal{F}_{0} -measurable. Furthermore, F:\mathcal{S}\times\mathcal{A}\times\mathbb{R}\to\mathcal{S} is a measurable transition function and a_{t}\in\mathcal{A} is the action chosen at time point t . We assume that the decision maker can decide about a policy \sigma=(\sigma_{0},\ldots,\sigma_{T-1}) where \sigma_{t} is the randomized, history dependent decision rule at time t , i.e. a_{t}\sim\sigma_{t}(\cdot|s_{0},a_{0},\ldots,s_{t}) . Assume a measurable one-stage reward r:\mathcal{S}\times\mathcal{A}\to\mathbb{R} . The cumulative reward for a trajectory up to time t is: R_{t}=\sum_{k=0}^{t-1}r(s_{k},a_{k}). (2) For this work we choose to see R_{T} as a random variable and we are interested in its distribution. The distribution depends on the chosen policy \sigma , hence we denote it by \text{Law}^{\sigma}(R_{T}) . When we are given a target distribution \mu^{\star} on \mathbb{R} , our goal is to find a policy \sigma such that \text{Law}^{\sigma}(R_{T})\approx\mu^{\star}. (3) To formalize the matching, we introduce a divergence between probability distributions, the weighted squared L^{2} distance between the characteristic functions: \mathcal{L}(\sigma):=\int_{\mathbb{R}}\left|\varphi^{\star}(u)-\varphi_{\sigma}(u)\right|^{2}w(u)\,du, (4) where \varphi_{\sigma}(u):=\mathbb{E}^{\sigma}[e^{iuR_{T}}] is the characteristic function of the real-valued random variable R_{T} when policy \sigma is chosen, \varphi^{\star} is the characteristic function of \mu^{*} and w:\mathbb{R}\to\mathbb{R}_{+} is a suitable weighting function satisfying the weight moment assumptions \int_{\mathbb{R}}w(u)du<\infty,\quad\int_{\mathbb{R}}|u|\cdot w(u)du<\infty,\quad\int_{\mathbb{R}}u^{2}\cdot w(u)du<\infty. Distances like these are popular in statistics for testing distribution hypothesis, 21 ; 6 . The problem becomes: \min_{\sigma}\mathcal{L}(\sigma). (5) Now, we reformulate Problem ( 5 ) according to 8 as a lifted MDP in order to distinguish classes of controls among which we will look for optimal ones. 2.2. Lifted Distributional MDP Formulation We now recast optimization problem ( 5 ) as a Markov Decision Process on an augmented state space that captures the distribution of outcomes. To formulate first the distributional MDP , we retain the original notation and introduce an augmented state variable (s_{t},R_{t}) taking values in the product space \mathcal{S}\times\mathcal{R} , where \mathcal{R}\subseteq\mathbb{R} is the set of possible cumulative rewards. The law of (s_{t},R_{t}) under a policy \upsigma (and fixed initial distribution) is denoted \mathsf{F}_{t}^{\upsigma}\in\mathcal{P}(\mathcal{S}\times\mathcal{R}) , i.e., the joint distribution of (s_{t},R_{t}) . In particular, \mathsf{F}_{0}=\delta_{s_{0}\times 0} is the initial distribution of (s_{0},R_{0}) . The terminal law \mathsf{F}_{T}^{\sigma} encodes the distribution of both the terminal state s_{T} and the cumulative reward R_{T} . Note that the marginal of \mathsf{F}_{T}^{\sigma} on the R -coordinate is precisely \text{Law}(R_{T}) . We can now define a lifted distributional MDP whose state at each time is the joint distribution \mathsf{F}_{t}^{\sigma} of (s_{t},R_{t}) . The lifted model is specified as follows: a) State space: \mathcal{X}:=\mathcal{P}(\mathcal{S}\times\mathcal{R}) . An element \mathsf{F}\in\mathcal{X} is a joint distribution of the state and accumulated reward. At time t , the lifted state is \mathsf{F}_{t}=\text{Law}^{\sigma}(s_{t},R_{t}) under the current policy \sigma . This lifted state encodes all the necessary information of the process. b) Action space: \Pi^{M}:=\{\pi:\mathcal{S}\times\mathcal{R}\to\mathcal{P}(\mathcal{A})\} , the set of Markov (possibly randomized) decision rules on the augmented state. That is, \pi\in\Pi^{M} is a conditional distribution \pi(da|s,R) prescribing an action distribution given state s and cumulative reward R . A lifted policy is a sequence \pi_{0},\pi_{1},\ldots,\pi_{T-1} with \pi_{t}\in\Pi^{M} . c) One-stage reward: 0 for each t