Task-Induced Representational Invariances Depend on Learning Objective in Deep RL

Manu Srinath Halvagal, Sebastian Lee, SueYeon Chung · 2026

Read Task-Induced Representational Invariances Depend on Learning Objective in Deep RL by Manu Srinath Halvagal, Sebastian Lee, SueYeon Chung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01868.txt.

Introduction

Halvagal and co-authors study what deep RL agents keep and drop in memory. RL models animal goal-driven behavior in neuroscience. Deep RL now succeeds across many domains. Its skill rests on abstract codes of complex states. But theory for those learned codes stays thin.

The team views the codes through MDP reduction theory. Agents run a navigation task under two methods. Value-based DQN and policy-gradient PPO score alike. Yet their codes differ in what they ignore. DQN codes ignore changes tied to MDP symmetries. PPO codes ignore changes tied to action symmetries. The split shows up across domains, they report. It also shapes later transfer learning results.

Catalog record

CREATORManu Srinath Halvagal, Sebastian Lee, SueYeon Chung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Task-Induced Representational Invariances Depend on Learning Objective in Deep RL Manu Srinath Halvagal, Sebastian Lee, SueYeon Chung Abstract Reinforcement Learning (RL) has long served as a model for goal-directed animal behavior in neuroscience. Modern deep RL has shown remarkable success across many domains, further strengthening this connection. The ability to learn abstract representations of high-dimensional state spaces underlies much of this success. However, theoretical understanding of these learned representations remains limited, hindering direct comparisons between models and animal learning. We address this gap by analyzing deep RL representations through the lens of MDP reduction theory. Investigating canonical RL algorithms in a navigation task, we find that even when performance is comparable, the value-based method (DQN) learns representations that are invariant to MDP homomorphism symmetries, while the policy-gradient method (PPO) learns representations invariant to action symmetries. These differences emerge consistently across domains, have downstream consequences for transfer learning, and appear in LLMs in a prompt-dependent manner. Our findings provide a principled approach to comparing learned representations across RL algorithms, with demonstrated practical implications and possible insights for neural coding in the brain. 1 Introduction Deep Reinforcement Learning (RL) has proved very successful in a range of tasks in recent years, including games such as Atari [ 39 ] and Go [ 56 ] , and more recently in fine-tuning large language models (LLMs) to human preferences [ 44 ] and formal reasoning abilities [ 22 ] . In neuroscience, RL has a long history as a framework for understanding animal learning [ 43 ] and is frequently used to model behaviour and explain neural data, particularly from the midbrain dopamine system [ 55 , 21 ] . More generally, however, barriers remain: direct comparisons between such computational models and animal learning is challenging [ 68 ] and subject to debate across levels of analysis [ 30 , 20 ] . While there has been a recent surge of work characterizing the representations of standard deep neural networks and language models [ 12 , 57 , 42 , 24 , 2 , 3 ] , relatively little attention has been paid to understanding Deep RL algorithms in these terms. On the one hand, there is a strong literature on learning theory in tabular RL [ 47 , 66 , 28 ] that is steadily growing to incorporate function approximation of increasing generality [ 62 , 27 , 65 ] . On the other hand, the empirical community has made remarkable algorithmic and computational strides and is enjoying yet another renaissance in its post-training applications to LLMs. However, understanding model representations and how they relate to task structure has been largely overlooked, with a few notable exceptions [ 71 , 64 ] . In this work, we seek to narrow this gap by analysing RL representations through the lens of invariances to symmetries. Although some symmetries in neural representations have been documented [ 32 ] , the role of task-induced symmetries, those arising from the structure of the behavioral problem itself, remains largely unexplored. In vision, the view that good representations progressively become invariant to nuisance variation while preserving task-relevant structure, formalized through "untangling" [ 17 , 12 ] , has been a productive bridge between computation, behavior, and neural representation. We adopt an analogous perspective for RL. In particular, we draw on ideas from the MDP reduction literature to understand encoding of so-called homomorphic images, as well as policy symmetries in the learned representations of a number of canonical deep RL algorithms. We find that the representational invariances which emerge depend systematically on the learning objective: value-based methods develop invariances to MDP homomorphism symmetries, while policy-gradient methods develop invariances to action symmetries. We perform this analysis in a range of RL domains before probing two further questions: do these learned symmetries have downstream functional consequences for transfer learning? And what kinds of symmetries are present in LLM representations when solving similar simple navigation tasks framed as an interactive RL problem? Overall, our contributions are: • We show across a range of domains that the symmetries encoded by the representations of deep RL agents depend on the learning objective: value-based objectives (DQN) learn representations that are invariant to MDP homomorphism symmetries, while policy-gradient objectives (PPO) learn representations invariant to action symmetries, even when both solve the same underlying task. • We empirically demonstrate both kinds of symmetry emerging in LLM representations when the model is prompted with different descriptions of a graph navigation task. • We show empirically on selected Atari games that algorithms that learn symmetries reflecting the structure of the environment (i.e. MDP homomorphisms) transfer better than those attuned to action symmetries. 2 Background 2.1 Reinforcement Learning The RL problem is frequently formulated as a Markov Decision Process (MDP) consisting of a tuple \langle\mathcal{S},\mathcal{A},\mathcal{T},\gamma,\mathcal{R}\rangle where \mathcal{S} is the state space, \mathcal{A} is the action space, \mathcal{T} is the state transition function, \gamma is the discount factor and \mathcal{R} is the reward function. RL algorithms aim to find a mapping from states to actions, denoted \pi and known as a policy, that maximises cumulative discounted reward. Two canonical methods which we focus on in this work are DQN [ 38 ] and PPO [ 54 ] , which sit either side of a common dividing line of RL algorithms; the former is a value-based method in which a policy is derived from a learned value-function, while the latter is a policy-gradient method in which the policy is directly optimised, and a value-function is learned as an auxiliary objective to reduce variance of the policy gradient estimates. 2.2 MDP Reduction A central idea behind the present work is that of reducing an MDP, which can be thought of as distilling the problem to a more compact form or representation. MDP reduction has a long history in RL [ 16 , 48 , 35 , 26 ] and has been studied under various names including MDP abstraction, state abstraction and state aggregation. The core idea is to group states or state-action pairs together based on some equivalence relation that exploits symmetries and environment structure. This grouping or abstraction essentially constructs an equivalent but smaller MDP, which is often easier to solve. One might naively consider the size of an RL problem to be given by the cardinality of the state/action spaces or the complexity of the policy function class. In simple settings, these concepts overlap: when the state space is small enough and we have a discrete set of actions, we can enumerate each state-action pair and learn properties of each independently. This regime is known as tabular RL. Consider the toy example in Fig. 1 , with 4 states and 4 actions and a single reward. Q-learning could solve this with 16 values. However, reduction techniques help uncover the core structure of the MDP, which may reveal a smaller underlying problem. Figure 1 : MDP homomorphisms can reduce problem sizes. (left) toy 2x2 gridworld task, (right) reduced MDP based on symmetry under reflection group. Bisimulation There are two principal groups of methods for reducing RL problems via abstraction. Under bisimulation [ 16 ] , two states s and t are considered bisimilar if \forall a\in\mathcal{A} : \displaystyle r(s,a) \displaystyle=r(t,a),\quad\text{and}\quad\sum_{s^{\prime}\in C}P(s,a,s^{\prime})=\sum_{s^{\prime}\in C}P(t,a,s^{\prime}), (1) where

Read the complete text