Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

Andrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli · 2026

Read Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching by Andrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27044.txt.

Introduction

Fraschini and co-authors compress RL policies by matching state occupancy instead of single actions. Deep RL policy spaces are huge and full of redundant settings. Earlier compression matched immediate actions, an error-prone proxy for behavior. Their occupancy-based method checks long-horizon state visitation mixtures directly. A uniqueness metric first curates a diverse pool of policies to learn from. The compression objective is smooth end to end. Tests on control benchmarks favor the occupancy approach.

Catalog record

CREATORAndrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63988
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27044

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching Andrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli Abstract Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space. A recent framework, which we refer to as Action-based Policy Compression (APC), mitigates this issue by compressing the parameter space Θ \Theta into a low-dimensional latent manifold 𝒵 {\mathcal{Z}} using a learned generative mapping g : 𝒵 → Θ g:{\mathcal{Z}}\to\Theta . However, its performance is severely constrained by relying on immediate action-matching as a reconstruction loss, a myopic proxy for behavioral similarity that suffers from compounding errors across sequential decisions. To overcome this bottleneck, we introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage. Specifically, we propose two principal improvements: (1) we curate the dataset generation with an information-theoretic uniqueness metric that delivers a diverse population of policies; and (2) we propose a fully differentiable compression objective that directly minimizes the divergence between the true and reconstructed mixture occupancy distributions. These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors while retaining most of the original parameter space’s expressivity. Finally, we empirically validate the advantages of our contributions across multiple continuous control benchmarks. 1 Introduction The success of Deep Reinforcement Learning (DRL) in complex continuous control relies heavily on the highly expressive nature of deep neural networks ( Lillicrap et al. 2016 ; Schulman et al. 2017 ) . However, optimizing these high-dimensional policies tabula rasa is notoriously sample-inefficient ( Duan et al. 2016 ; Agarwal et al. 2022 ) . This inefficiency stems in part from a fundamental functional redundancy: Due to the complex symmetries of neural architectures, vast and disparate regions of the parameter space often collapse into identical agent behaviors. Searching this massive, unstructured space blindly ignores the underlying functional geometry of the environment. This limitation becomes a severe bottleneck in multi-task or Unsupervised Reinforcement Learning (URL) settings ( Laskin et al. 2021 ) , where an agent must train a model that rapidly adapts to a variety of previously unknown downstream tasks, instead of learning from scratch. To address this redundancy in the policy parameter space, prior work has proposed various approaches to compress the policy space onto lower-dimensional latent manifolds ( Eysenbach et al. 2018 ; Mutti et al. 2022 ; Faccio et al. 2023 ; Tenedini et al. 2026 ; Li et al. 2026 , e.g.,) . In this stream, Action-based Policy Compression ( Tenedini et al. 2026 , APC,) explicitly shifts the learning paradigm from parameter optimization to behavior optimization. APC leverages a generative model to compress a highly redundant policy parameter space into a low-dimensional, behaviorally structured latent manifold. By curating a diverse dataset of random policies and training an autoencoder to reconstruct them, agents can subsequently bypass the raw parameter space entirely. Instead, they perform task-specific fine-tuning by directly navigating the compact latent space. By following this recipe, APC compresses policy architectures with several thousand parameters into tiny two- or three-dimensional latent spaces, while retaining most of the functional expressivity to address a variety of tasks in classical continuous control domains, such as Mountain Car and Reacher. While conceptually and empirically promising, the practical adaptability of APC beyond these domains is severely constrained by the technical solutions adopted to train the latent space. In particular, to organize this latent space functionally rather than geometrically, APC relies on an action-matching objective, training the autoencoder to minimize the discrepancy between the immediate action outputs of the original and reconstructed policies. However, measuring behavioral similarity through immediate action divergence is a mathematically loose and myopic proxy for long-horizon control ( Metelli et al. 2018 ) . Because errors compound over sequential decisions, policies that appear functionally identical at the single-action level can induce drastically different trajectories and global state visitations. Furthermore, Tenedini et al. 2026 curate the initial pre-training dataset of policies using novelty search in the action space, which frequently discards rare, high-performing behaviors, severely impoverishing the topological diversity of the resulting manifold. To overcome these limitations, we introduce Occupancy-based Policy Compression (OPC), representing a paradigm shift in latent manifold learning for DRL. We abandon immediate action-matching entirely in favor of global state-space coverage. Specifically, we introduce two principal algorithmic enhancements. First, we curate the pre-training dataset using an information-theoretic uniqueness metric that actively selects policies based on their contribution to the global state-visitation entropy, retaining a much broader spectrum of rare behaviors. Second, we propose a fully differentiable compression objective that explicitly matches the true mixture occupancy distribution of the policy population. By evaluating policies based on the exact states they visit over the full horizon, OPC completely bypasses the compounding errors of action-matching, forcing the generative model to organize the latent space around true, long-horizon functional similarity. Then, to support the design choices behind OPC, we address the following: Research Questions: ( Q1 ) Does curating policies based on their contribution to state-space coverage preserve a wider spectrum of rare, high-performing behaviors compared to action-space novelty search? ( Q2 ) Does optimizing a compression objective based on state-space visitation patterns yield a more behaviorally meaningful latent space than action-based behavioral reconstruction? ( Q3 ) Does navigating an occupancy-matched latent space improve sample efficiency and task performance compared to standard baselines and action-matched manifolds? Content Outline and Contributions. In Section 2 , we establish the necessary notation. In Section 3 , we detail the existing APC framework and expose the limitations of action-matching. In Section 4 , we introduce our OPC framework. Finally, in Section 5 , we address our Research Questions with extensive empirical evidence, demonstrating that OPC retains richer behavioral modes, constructs superior latent topologies, and enables fast few-shot adaptation to various downstream tasks. 2 Preliminaries Notation. For a set {\mathcal{A}} of finite size |{\mathcal{A}}| , we denote \Delta({\mathcal{A}}) its probability simplex. For two distributions p,q , the Kullback-Leibler (KL) divergence is D_{\mathrm{KL}}(p\parallel q) and the differential entropy of p is H(p) . Interaction Protocol. We model the interactions between the agent and the environment as a (finite-horizon) Controlled Markov Process (CMP). A CMP is a tuple {\mathcal{M}}\coloneq({\mathcal{S}},{\mathcal{A}},{\mathbb{P}},\mu,T) , where {\mathcal{S}} is the state space and {\mathcal{A}} is the action space. An episode begins with the initial state s_{0}\sim\mu\in\Delta({\mathcal{S}}) , where \mu is the initial state distribution. At each timestep t=0,\ldots,T , where T<\infty is the episode horizon, the agent observes the current state s_{t} , takes action a_{t}\in{\mathcal{A}} , such that the model transitions to the next state s_{t+1}\sim{\mathbb{P}}(\cdot\mid s_{t},a_{t}) according to the transition model {\mathbb{P}}:{\mathcal{S}}\times{\mathcal{A}}\to\Delta({\mathcal{S}}) . The agent acts according to a stochastic policy \pi:{\mathcal{S}}\to\Delta({\mathcal{A}}) such that \pi(a\mid s) denotes the conditional probability of taking action a in state s . This interaction induces a trajectory \tau\coloneq(s_{0},a_{0},s_{1},a_{1},\ldots,s_{T}) . Occupancy Measure. The deployment of policy \pi in {\mathcal{M}} induces a marginal distribution over the state space, known as the (state) occupancy measure d_{\pi}\in\Delta({\mathcal{S}}) , which represents the average visitation probability of state s over the episode length: d_{\pi}(s)=\frac{1}{T+1}\sum_{t=0}^{T}\Pr(s_{t}=s\mid\pi) . If we consider the

Read the complete text