Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cramér Surrogate

Simo Alami C, Rim Kaddah, Marie-Paule Cani, Jesse Read · 2025

Read Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cramér Surrogate by Simo Alami C, Rim Kaddah, Marie-Paule Cani, Jesse Read (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04310.txt.

Introduction

Simo Alami C and co-authors model full return distributions for reinforcement learning with far fewer parameters in 2025, using team framing since the head author order differs from the catalog list. Standard methods predict only average reward. Distribution methods fix support points or stack flat slices, so size grows with detail. Their model uses smooth flows that bend a base curve into the return shape.

Size then stays fixed no matter the detail wanted. A new distance, built on mass gaps, trains the flow. The team proves it is a true distance with shrinking update steps. Fair sample slopes come free. Toy tasks show rich multi peak returns captured. Game tests matched classic rivals at a fraction of the size.

Catalog record

CREATORSimo Alami C, Rim Kaddah, Marie-Paule Cani, Jesse Read
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04310

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cramér Surrogate Simo Alami C, Rim Kaddah, Marie-Paule Cani, Jesse Read Abstract Distributional Reinforcement Learning (DistRL) improves upon expectation-based methods by modeling full return distributions, but standard approaches often remain far from parsimonious. Categorical methods (e.g., C51) rely on fixed supports where parameter counts scale linearly with resolution, while quantile methods approximate distributions as discrete mixtures whose piecewise-constant densities can be wasteful when modeling complex multi-modal or heavy-tailed returns. We introduce NFDRL, a parsimonious architecture that models return distributions using continuous normalizing flows. Unlike categorical baselines, our flow-based model maintains a compact parameter footprint that does not grow with the effective resolution of the distribution, while providing a dynamic, adaptive support for returns. To train this continuous representation, we propose a Cramér-inspired, geometry-aware distance defined over probability masses obtained from the flow. We show that this distance is a true probability metric, that the associated distributional Bellman operator is a γ \sqrt{\gamma} -contraction, and that the resulting objective admits unbiased sample gradients—properties that are typically not simultaneously guaranteed in prior PDF-based DistRL methods. Empirically, NFDRL recovers rich, multi-modal return landscapes on toy MDPs and achieves performance competitive with categorical baselines on the Atari-5 benchmark, while offering substantially better parameter efficiency. 1 Introduction Traditional reinforcement learning (RL) algorithms aim to estimate the expected return from a given state or state-action pair [ 1 ] . However, this expectation provides only a partial view of the underlying return distribution, omitting critical information about uncertainty, risk, and variability. Distributional reinforcement learning (DistRL) addresses this limitation by modelling the full return distribution, providing a richer and more informative signal for decision-making. Why distributional RL? Modeling the full return distribution can improve learning and control beyond what is possible with expectation-only value functions. Prior work reports that distributional critics provide richer training signals and can yield stronger empirical performance, while also enabling risk-aware decision making through functionals of the return distribution (e.g., tail probabilities or CVaR) rather than a single mean value [ 2 , 3 ] . In addition, explicit access to distributional information supports robustness considerations and uncertainty-aware policies in stochastic environments, where variability and rare outcomes matter. A critical challenge in DistRL is achieving an expressive representation of the return distribution without incurring excessive computational or parametric costs. Early approaches such as Categorical DQN (C51) [ 2 ] approximate the return distribution using a histogram over a fixed discrete support. This formulation suffers from a fundamental lack of parsimony: the model’s parameter count scales linearly with the support resolution. To achieve high precision, one must drastically increase the number of atoms, inflating the model size. Furthermore, the fixed support imposes a hard bound on returns, limiting applicability in domains with dynamic or highly variable rewards, such as robotic control [ 4 ] or finance [ 5 ] . Existing alternatives face theoretical or representational trade-offs. Bellemare et al. [2] showed that the Distributional Bellman operator is a contraction in the Wasserstein metric. However, C51 is trained using KL divergence because empirical Wasserstein minimization suffers from biased sample gradients. This creates a disconnect between the contraction metric and the optimization objective. Moreover, using the KL divergence is also a major drawback in the unbounded setting, where predicted and target distributions may have negligible overlap, resulting in poor gradient signals. Subsequent quantile-based methods like QR-DQN [ 6 ] and IQN [ 3 ] or FQF [ 7 ] address the support limitation by learning quantile locations and train with quantile regression losses (pinball, often with a Huber smoothing). By construction, quantile models handle unbounded returns and optimize a surrogate consistent with the 1-Wasserstein distance in one dimension. While highly effective, these methods approximate distributions as mixtures of Dirac deltas (step functions). This representation can be representationally inefficient for smooth, multi-modal, or heavy-tailed distributions, often requiring a large number of quantiles to approximate simple densities [ 8 ] . A complementary approach is to model the return distribution through a continuous probability density. Continuity matters for two reasons. First, it yields an adaptive support and avoids committing to a fixed return range, which is a limitation of categorical methods in domains with dynamic or heavy-tailed rewards. Second, a smooth density facilitates the computation of risk-sensitive criteria (e.g., CVaR or tail probabilities) by direct integration, without introducing an additional discretization level or increasing the number of parameters to refine resolution. This motivates continuous density models for DistRL. While recent work has explored normalizing flows for DistRL [ 9 ] , achieving a training objective that is both stable in RL and theoretically consistent (contraction and unbiased sample gradients) remains challenging. Contribution. We propose NFDRL, a parameter-efficient DistRL method that models return distributions using continuous normalizing flows [ 10 ] . To train this continuous representation, we introduce a geometry-aware surrogate of the Cramér distance defined over probability masses. Our approach offers three distinct advantages. First, it ensures representational efficiency: unlike categorical baselines where model size scales linearly with resolution, NFDRL maintains a compact, constant parameter footprint while offering infinite effective resolution and dynamic support. Second, it guarantees theoretical soundness ; we prove that our mass-based distance is a true metric that induces a \sqrt{\gamma} -contraction in the distributional Bellman operator and admits unbiased sample gradients, resolving the inconsistencies of prior PDF-based approaches. Finally, the method demonstrates superior expressivity , capturing complex, multi-modal return landscapes that quantile baselines often fail to resolve. We validate our approach on toy MDPs and the Atari-5 benchmark [ 11 ] . We show that NFDRL achieves performance competitive with categorical baselines while utilizing significantly fewer parameters, offering a theoretically rigorous and parsimonious alternative for Distributional RL. Scope and positioning. Our objective is not to propose a replacement for strong quantile-based agents such as IQN on Atari, but to study a parsimonious continuous return representation with well-controlled metric properties. Quantile methods represent \eta^{\pi}(x,a) as a finite mixture of Dirac masses; they are highly effective for control, but their discrete nature can make fine-grained characterization of distributional shape (e.g., multimodality, tail behavior) dependent on the number of quantiles. In contrast, NFDRL models a smooth density with adaptive support and a constant parameter footprint, and is paired with a Cramér-inspired objective for which we prove (i) metric axioms, (ii) a \sqrt{\gamma} -contraction of the distributional Bellman operator, and (iii) unbiased sample gradients. We view NFDRL as a complementary tool with different trade-offs (parsimony, continuous PDFs, and theoretical consistency), particularly relevant when downstream risk-sensitive quantities depend on the detailed shape of the return distribution. 2 Background We consider the standard RL setting, where the interaction between an agent and an environment is modelled as a Markov Decision Process (MDP) \mathcal{M}=(\mathcal{X},\mathcal{A},R,P,\gamma) [ 1 ] . Here, \mathcal{X} and \mathcal{A} denote the state and action spaces, R:\mathcal{X}\times\mathcal{A}\to\mathbb{R} is the reward function, P(\cdot|x,a) is the transition kernel, and \gamma\in(0,1) is a discount factor. A policy \pi(\cdot|x) maps a state x\in\mathcal{X} to a distribution over actions. Under a fixed policy \pi , the return G^{\pi}(x) is defined as the random variable representing the discounted sum of rewards collected along a trajectory starting from state x : G^{\pi}(x):=\sum_{t=0}^{\infty}\gamma^{t}R(x_{t},a_{t}),\quad x_{0}=x,a_{t}\sim\pi(\cdot|x_{t}),x_{t+1}\sim P(\cdot|x_{t},a_{t}). (1) The goal is to estimate the expected return i.e. the

Read the complete text