Regularized Latent Dynamics Prediction is a Strong Baseline for Behavioral Foundation Models

Pranaya Jajoo 1,2 Harshit Sikchi 4,† Siddhant Agarwal 4,† Amy Zhang 4 Scott Niekum 5 Martha White 1,2,3 1 Department of Computing Science, University of Alberta, Canada 2 Alberta Machine Intelligence Institute (Amii) 3 Canada CIFAR AI Chair 4 The University of Texas at Austin 5 University of Massachusetts Amherst † Equal contribution · 2026

Read Regularized Latent Dynamics Prediction is a Strong Baseline for Behavioral Foundation Models by Pranaya Jajoo 1,2 Harshit Sikchi 4,† Siddhant Agarwal 4,† Amy Zhang 4 Scott Niekum 5 Martha White 1,2,3 1 Department of Computing Science, University of Alberta, Canada 2 Alberta Machine Intelligence Institute (Amii) 3 Canada CIFAR AI Chair 4 The University of Texas at Austin 5 University of Massachusetts Amherst † Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15857.txt.

Introduction

Jajoo and co-authors simplify how behavioral foundation models learn state features. Such models promise agents that adapt to any new reward. Their reach stops at rewards spanned by preset state features. Complex training goals were thought vital for rich features. Plain next-state prediction in latent space forms features too. That plain goal shrinks feature spread and span, though. Their RLDP method adds a simple orthogonality penalty. Feature diversity then holds through training. Low-coverage tests favor RLDP where prior methods falter.

Catalog record

CREATORPranaya Jajoo 1,2 Harshit Sikchi 4,† Siddhant Agarwal 4,† Amy Zhang 4 Scott Niekum 5 Martha White 1,2,3 1 Department of Computing Science, University of Alberta, Canada 2 Alberta Machine Intelligence Institute (Amii) 3 Canada CIFAR AI Chair 4 The University of Texas at Austin 5 University of Massachusetts Amherst † Equal contribution
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76997
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15857

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Regularized Latent Dynamics Prediction is a Strong Baseline for Behavioral Foundation Models Pranaya Jajoo 1,2 Harshit Sikchi 4,† Siddhant Agarwal 4,† Amy Zhang 4 Scott Niekum 5 Martha White 1,2,3 1 Department of Computing Science, University of Alberta, Canada 2 Alberta Machine Intelligence Institute (Amii) 3 Canada CIFAR AI Chair 4 The University of Texas at Austin 5 University of Massachusetts Amherst † Equal contribution Abstract Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task. These methods, however, are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existing state features , making the choice of state features crucial to the expressivity of the BFM. As a result, BFMs are trained using a variety of complex objectives and require sufficient dataset coverage, to train task useful spanning features. In this work, we examine the question: are these complex representation learning objectives necessary for zero-shot RL? Specifically, we revisit the objective of self-supervised next-state prediction in latent space for state feature learning, but observe that such an objective alone is prone to increasing state-feature similarity, and subsequently reducing span. We propose an approach, Regularized Latent Dynamics Prediction ( RLDP ), that adds a simple orthogonality regularization to maintain feature diversity and can match or surpass state-of-the-art complex representation learning methods for zero-shot RL. Furthermore, we empirically show that prior approaches perform poorly in low-coverage scenarios where RLDP still succeeds. 1 Introduction Zero-shot reinforcement learning (RL) ( 49 ) is a problem setting where we learn an agent that can solve any task in the environment without any additional training or planning, after an initial pretraining phase. Zero-shot RL has significant practical potential in developing generalist agents with wide applicability. For instance, robotics applications, like robotic manipulation or drone navigation, often require agents to solve a wide variety of unknown tasks. A general-purpose household robot needs to possess the capability to flexibly adapt to various household chores without explicit training for each new task. Behavioral Foundation Models (BFMs) have been shown to be promising for zero-shot RL ( 49 ; 4 ) . BFMs are trained on a dataset of reward-free interactions, with the aim to provide a near-optimal policy for a wide class of reward functions without additional learning or training during test-time. BFMs are trained by a) learning a state representation \varphi:s\rightarrow\mathbb{R}^{d} , and b) learning a policy \pi conditioned on a latent vector z\in\mathbb{R}^{d} , where the z can be seen as a task embedding for reward r(s)=\varphi(s)^{\top}z . In this way, the BFM consists of a space of policies, where different policies can be extracted by querying the learned policy using different z . At test time, given any reward function r^{test}(s) , the near-optimal policy \pi_{z_{r^{test}}} is obtained zero-shot by solving for z_{r^{test}} such that r^{test}(s)\approx\varphi(s)^{\top}z_{r^{test}} . This assumption on the reward is also used for successor features ( 6 ), which consist of the discounted cumulative sum of feature vectors under a policy. Successor features zero-shot produce the action-values for a new reward, given by vector z , and BFM approaches often use successor features to learn the policies. The performance of the BFM relies heavily on the state representation, which is both used to extract z for the reward and for the policy. State-of-the-art methods 48 ; 4 ; 34 usually learn state representations that retain information to represent successor measures under a wide class of policies. A successor measure captures the (discounted) state visitation of a policy, given any starting state. They are the generalization of successor representations 10 to continuous states, and have a simple linear relationship to successor features 48 . They can therefore be used to both encourage learning a generalizable state representation as well as simultaneously learning the successor features for the BFM. Successor measures are usually learned for an explicitly defined class of policies ( 4 ) or implicitly by first defining a class of reward functions ( 49 ; 34 ) and considering optimal policies for those reward functions as the set of policies. The main intuition behind predicting successor measures as a target for state representation learning is that representations sufficient to explain future state-visitation for a wide range of policies capture features that are relevant for sequential decision making under various reward functions. Unfortunately, learning state representations by estimating successor measures requires iteratively applying Bellman evaluation backups or Bellman optimality backups, both of which are known to result in a variety of learning difficulties. They can suffer from various forms of bias 45 ; 17 ; 30 ; 14 and can suffer from feature collapse due to the instability inherent in bootstrapping in the function approximation regime ( 26 ) . Using Bellman backups to learn a representation requires choosing a class of policies or a class of reward functions a priori. Further, because the state representation is trained from a batch of offline data, unless chosen carefully, the policies may select out-of-distribution actions, leading to incorrect generalization and degenerate representations. A simple alternative that sidesteps these issues is to use latent dynamics learning: predicting future latent states given the current state and the sequence of actions. Learning the state representation by predicting the latent dynamics has the benefit of being independent of the policy and thus avoids using Bellman backups and these out-of-distribution issues. This work investigates the following question: Does latent next-state prediction produce state features that enable performant zero-shot RL? Our investigation is inspired by the work of 15 , which showed that using dynamics prediction losses as auxiliary losses boosted performance of a single-task RL agent. Unlike the single task RL setting examined by 15 , we find that in its naive form, this objective leads to a mild form of feature collapse where the representation of different states increase in similarity over training. This collapse results in poor zero-shot RL performance when evaluated on a number of downstream tasks. With a simple orthogonality regularization to prevent collapse, we show that the representations learned are competitive and present a scalable alternative to representations learned via complex successor measure estimation methods for zero-shot RL. In summary, the contributions of this paper are as follows. 1. We propose regularized latent-dynamics prediction ( RLDP ) as a simple alternative to learn state features for zero-shot RL. We identify as well as mitigate feature collapse plaguing latent dynamics prediction. 2. We show that our method remains competitive through an extensive empirical evaluation of representations for task generalization across a variety of domains, in online and offline RL settings, including in humanoid with a large state-action space. 3. We show that the RLDP objective can learn performant policies in low-coverage settings where other methods fail. 2 Related Work Unsupervised RL encompasses the class of algorithms that enable learning general-purpose skills and representations without relying on reward signal in the data. Works that have focused on intent or skill discovery have used diversity-driven objectives ( 11 ; 1 ) , maximizing mutual information ( 50 , 11 , 1 , 12 ) or minimizing the Wasserstein distance ( 35 ) between latents and the induced state-visitation distribution. These discovered skills can be

Read the complete text