Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces

Saket Tiwari, Omer Gottesman, George Konidaris · 2025

Read Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces by Saket Tiwari, Omer Gottesman, George Konidaris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20853.txt.

Introduction

Saket Tiwari, Omer Gottesman, and George Konidaris study the shape of learning in smooth control tasks in a 2025 paper. Neural reinforcement learning wins on tasks with endless states and actions, yet theory mostly covers finite ones. The team ask what set of states a trained policy can actually reach.

Training a two-layer neural policy with an actor-critic scheme traces a low-dimensional sheet inside the vast state space. Under stated conditions, the sheet's dimension tracks the action space's size. Four standard robot benchmarks back the bound, and a toy task confirms it as dimensions vary. A manifold layer built on the result lifts control in high-freedom tasks.

Catalog record

CREATORSaket Tiwari, Omer Gottesman, George Konidaris
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces Saket Tiwari, Omer Gottesman, George Konidaris Abstract Advances in reinforcement learning (RL) have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and action spaces. We propose building a theoretical understanding of continuous state and action spaces by employing a geometric lens to understand the locally attained set of states. The set of all parametrised policies learnt through a semi-gradient based approach induces a set of attainable states in RL. We show that the training dynamics of a two-layer neural policy induce a low dimensional manifold of attainable states embedded in the high-dimensional nominal state space trained using an actor-critic algorithm. We prove that, under certain conditions, the dimensionality of this manifold is of the order of the dimensionality of the action space. This is the first result of its kind, linking the geometry of the state space to the dimensionality of the action space. We empirically corroborate this upper bound for four MuJoCo environments and also demonstrate the results in a toy environment with varying dimensionality. We also show the applicability of this theoretical result by introducing a local manifold learning layer to the policy and value function networks to improve the performance in control environments with very high degrees of freedom by changing one layer of the neural network to learn sparse representations. 1 Introduction The goal of a reinforcement learning (RL) agent is to learn a policy that maximises its expected, time discounted cumulative reward ( Sutton & Barto 1998 ) . Recent advances in RL have lead to agents successfully learning in environments with enormous state spaces, such as games ( Mnih et al. 2015 ; Silver et al. 2016 ; Wurman et al. 2022 ) , robotic control in simulation ( Lillicrap et al. 2016 ; Schulman et al. 2015 ; Schulman et al. 2017 ) and real environments ( Levine et al. 2016 ; Zhu et al. 2020 ; Deisenroth & Rasmussen 2011 ; Kaufmann et al. 2023 ) . However, we do not have an understanding of the intrinsic complexity of these seemingly large problems. We investigate the complexity of RL environments through a geometric lens. We build on the intuition behind the manifold hypothesis , which states that most high-dimensional real-world datasets actually lie on or close to low-dimensional manifolds ( Tenenbaum 1997 ; Carlsson et al. 2007 ; Fefferman et al. 2013 ; Bronstein et al. 2021 ) ; for example, the set of natural images is a very small, smoothly varying subset of all possible value assignments for the pixels. A promising geometric approach is to model this data as a low-dimensional structure—a manifold —embedded in a high-dimensional ambient space. In supervised learning, especially deep learning theory, researchers have shown that the approximation error depends strongly on the dimensionality of the manifold ( Shaham et al. 2015 ; Pai et al. 2019 ; Chen et al. 2019 ; Cloninger & Klock 2020 ) , thus connecting the learning complexity to the complexity of the underlying structure of the dataset. RL researchers have previously applied the manifold hypothesis —- i.e. by hypothesizing that the effective state space lies on a low-dimensional manifold ( Smart & Kaelbling 2002 ; Mahadevan 2005 ; Machado et al. 2017 ; Machado et al. 2018 ; Banijamali et al. 2018 ; Wu et al. 2019 ; Liu et al. 2021 ) , but the assumption has never been theoretically and empirically validated. RL shares many similarities with control theory ( Bertsekas 2012 ; Bertsekas 2024 ) . In a control-theoretic framework, the objective is to drive the system, over time, to a desired state or goal. Consequently, theoreticians and practitioners are often interested in the reachability of a control system to understand what state is reachable given how system changes under control inputs, i.e. the system dynamics. Locally reachable states are the set of states to which the system can possibly transition, starting from a fixed state, under all smooth time variant controls. Control theorists have long studied the set of reachable states ( Kalman 1960 ) using a differential geometric framework ( Sussmann 1973 ; Sussmann 1987 ) . Theoretical research in the study of control systems is often focused on finding necessary and sufficient conditions in the system dynamics such that all states are reachable ( Isidori 1985 ; Sun et al. 2002 ; Respondek 2005 ; Sun 2007 ) under all the admissible time-variant policies. In RL, the objective is to maximize the discounted return via gradient-based updates to the policy parameters, so the focus is on states attained through a sequence of policies determined by these parameters. Furthermore, a theoretical understanding of states attainable using neural network (NN) policies gives us insight into the geometry and low-dimensional structure of data in RL. This requires utilising an analytically tractable model of NNs. Ever since the remarkable success of neural networks, researchers have developed various theoretical models to better understand their efficacy. A theoretical model intended to study a complex object, such as a neural network, often ends up making simplifying assumptions for tractability. One such theoretical model studies the evolution of neural networks linearly in parameters during training of wide neural networks ( Lee et al. 2019 ; Jacot et al. 2018 ) , meaning in a setting where the width approaches infinity. This has helped researchers develop theories of the generalization properties of neural networks ( Jacot et al. 2018 ; Allen-Zhu et al. 2019a ; Wei et al. 2019 ; Adlam & Pennington 2020 ) . We similarly utilize a single hidden layer neural network model for the policy that is linear in terms of its parameters, not linear in the state, as the width approaches infinity, as has previously been applied to RL ( Wang et al. 2019 ; Cai et al. 2019a ) . Within this theoretical framework, we provide a proof of the manifold hypothesis for deterministic continuous state and action RL environments with wide two-layer neural networks. We prove that the effective set of attainable states is subset of a manifold and its dimensionality is upper bounded linearly in terms of the dimensionality of the action space, under appropriate assumptions, independent of the dimensionality of the nominal state space. The primary intuition is that the set of states locally attained are restricted by two factors: 1) the policy is time invariant and state dependent, and 2) the set of policies is constrained by the optimization of a wide , two-layer neural network using stochastic policy gradients. Our theoretical results are for deterministic environments with continuous states and actions; we empirically corroborate the low-dimensional structure of attainable states on MuJoCo environments ( Todorov et al. 2012 ) by applying the dimensionality estimation algorithm by Facco et al. 2017 . To show the applicability and relevance of our theoretical result, we empirically demonstrate that a policy can implicitly learn a low-dimensional representation with marginal computational overhead using the CRATE framework ( Yu et al. 2023a ; Yu et al. 2023b ; Pai et al. 2024 ) . We present an algorithm

Read the complete text