Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics
Maksim Bobrin 1,2 Ilya Zisman 5 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 5 Humanoid Alexander Nikulin 1,3 Vladislav Kurenkov 1,4 Dmitry Dylov 1,2 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 3 MSU 4 Innopolis University · 2025
Read Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics by Maksim Bobrin 1,2 Ilya Zisman 5 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 5 Humanoid Alexander Nikulin 1,3 Vladislav Kurenkov 1,4 Dmitry Dylov 1,2 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 3 MSU 4 Innopolis University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13150.txt.
Introduction
Maksim Bobrin and four co-authors tackle a weakness of behavioral foundation models in this 2025 paper. Such models learn, from offline data, how to act across many tasks without extra training at test time. They stumble, though, when the rules of motion in the world change, as happens often in robotics.
The authors show that one leading method, the forward-backward representation, cannot tell two different sets of dynamics apart, so its stored policies blur together. Their fix adds a transformer-based module that infers which dynamics are in play from context. They also split the policy code space into clusters, one per dynamics type. In tests with changing dynamics, the new model earned up to twice the reward of the baselines on both discrete and continuous tasks, and it coped with dynamics it had never seen.
Catalog record
| CREATOR | Maksim Bobrin 1,2 Ilya Zisman 5 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 5 Humanoid Alexander Nikulin 1,3 Vladislav Kurenkov 1,4 Dmitry Dylov 1,2 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 3 MSU 4 Innopolis University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics Maksim Bobrin 1,2 Ilya Zisman 5 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 5 Humanoid Alexander Nikulin 1,3 Vladislav Kurenkov 1,4 Dmitry Dylov 1,2 1 AXXX 2 Applied AI Institute, Computational Imaging Lab 3 MSU 4 Innopolis University Abstract Behavioral Foundation Models (BFMs) have proven successful at producing near-optimal policies for arbitrary tasks in a zero-shot manner, requiring no test-time retraining or task-specific fine-tuning. Among the most promising BFMs are those that estimate the successor measure, learned in an unsupervised way from task-agnostic offline data. However, these methods fail to react to changes in the dynamics, making them ineffective under partial observability or when the transition function changes. This hinders the applicability of BFMs in real-world settings (e.g., robotics), where the dynamics can change unexpectedly at test time. In this work, we demonstrate that the Forward–Backward (FB) representation, cannot produce reasonable policies under distinct dynamics, leading to interference among latent policy representations. To address this, we propose an FB model with a transformer-based belief estimator, which greatly facilitates zero-shot adaptation. Additionally, we show that partitioning the policy-encoding space into dynamics-specific clusters aligned with context-embedding directions yields further performance gains. These traits allow our method to respond to the dynamics mismatches observed during training and to generalize to unseen dynamics changes. Empirically, in settings with changing dynamics, our approach achieves up to 2× higher zero-shot returns than baselines on both discrete and continuous tasks. Project page: https://belief-fb.dunnolab.ai/ 1 Introduction † † footnotetext: corresponding author: [email protected] † † footnotetext: work is done in collaboration with dunnolab.ai One very desirable property of reinforcement learning (RL) agents is their ability to adapt during inference to new tasks without requiring any additional learning. Achieving this in as few trials as possible would be even better: the ideal being the zero-shot adaptation ( Barreto et al., 2017 ; Touati et al., 2022 ) , where the agent never interacts with the environment at test-time and relies solely on the task-agnostic data. Behavioral Foundational Models (BFMs) ( Pirotta et al., 2023 ; Sikchi et al., 2024 ; Tirinzoni et al., ) may be considered as a step in this direction, because they can learn a variety of behaviors from offline data without task specific labels. During inference, it is possible to extract a task-specific policy that is theoretically optimal in terms of performance. Recent works ( Tirinzoni et al., ) demonstrates that methods based on successor measure estimation through Forward-Backward (FB) decomposition ( Touati and Ollivier, 2021 ) , is especially versatile and can successfully extract diverse policies from data. At the same time, FB has a fundamental drawback that limits its adaptation ability to more general settings. In our paper, we show that FB is unable to generalize across different environment configurations ( i.e. , dynamics), such as changes in a transition function ( e.g. , new obstacles or environment configuration) or some latent factor variation ( e.g. , wind direction or change in mass). This limitation stems from the way the approximation of the successor measure ( Blier et al., 2021 ) is estimated: FB averages the discounted future-occupancy state distribution over all observed dynamics, which inevitably causes interference in a policy representation space. This fact alone may severely constrain the applicability of FB in the real-world scenarios. For example, one of the largest robotics dataset, Open X-Embodiment ( Collaboration, 2023 ) , consists of 22 different robot embodiments, and training FB on each of them independently is infeasible. In Section 3.1 , we discuss the underlying cause in more details by providing both empirical and theoretical support. To remedy such limitation, we introduce Belief-FB (BFB) , a conditioning method for FB through a belief estimation, a popular technique of uncertainty quantification in Meta-RL ( Zintgraf et al., 2020 ; Dorfman et al., 2021 ) . To implement this, we employ a permutation-invariant transformer encoder, denoted as f_{\text{dyn}} , which processes a given trajectory from the dataset to produce a dynamics-encoding vector h . This representation is subsequently utilized as a conditioning input to the successor feature network, expressed as F(\cdot,\cdot,h,\cdot) . We pre-train f_{\text{dyn}} in a self-supervised fashion, thus posing no additional requirements on the data structure or the trajectory re-labeling, while maintaining theoretical guarantees. We discuss the implementation of Belief-FB in Section 3.2 . Remarkably, Belief-FB enables the generalization capabilities of FB not only through the dynamics seen in the training dataset, but also on the unseen test configurations. Additionally, we find that in order to align belief estimation better with FB, one also needs to partition the policy space encodings prior into dynamics-specific clusters, which significantly improves generalization abilities of FB. We propose Rotation-FB (RFB) that accomplishes this partitioning. We present the theoretical support and the implementation details of Rotation-FB in Section 3.3 . Empirically, both BFB and RFB outperform baselines for seen and unseen dynamics, as gathered in Figure 1 and discussed in Section 4.3 . Figure 1: Summary of results . Aggregate mean performance over seen (train) and unseen (test) dynamics for zero-shot RL. The error bars indicate standard deviation over 5 seeds. Notably, both BFB and RFB adapt not only to the dynamics seen during training, but are also able to generalize to unseen dynamics. There are 30(20) training (test) dynamics for Randomized-FourRooms and PointMass and 16(4) for AntWind environments. We believe that our work sufficiently broadens the possible applicability of BFMs, yet keeping all of the zero-shot properties unchanged. Our contributions are as follows: • We demonstrate the limitation of Forward-Backward (FB) representations ( Touati and Ollivier, 2021 ) , which lies in its inability to generalize per se across different dynamics both from train and test, where dynamics shift constitute of new layout grids or changes in the transition function that are hidden from an agent. Refer to Section 3.1 for more discussion. • We propose Belief–FB (BFB) , which employs a permutation-invariant transformer encoder to infer a belief over the current dynamics ( Zintgraf et al., 2020 ; Dorfman et al., 2021 ) . Analyzing BFB’s policy encoding space reveals that additional disentanglement is beneficial, motivating our Rotation–FB (RFB) extension. Section 3.2 examines Belief-FB, and Section 3.3 details Rotation-FB’s theoretical motivation and implementation. • We empirically demonstrate that both BFB and RFB can adapt to different dynamics , unlike its counterparts in the zero-shot setup. Refer to Section 4.3 for the discussion and Figure 1 for results. 2 Behavioral Foundation Models (BFMs) A Behavioral Foundation Model (BFM) ( Pirotta et al., 2023 ; Tirinzoni et al., ; Frans et al., 2024 ; Park et al., 2024 ; Sikchi et al., 2025 ) is an RL agent trained in an unsupervised manner on a task-agnostic dataset to approximate optimal policies for diverse reward functions (tasks) specified at inference (test-time). Forward-Backward Representation (FB) ( Touati and Ollivier, 2021 ) approximates a discounted successor measure ( Blier et al., 2021 ) for various behaviors across diverse tasks. The successor measure M^{\pi}(s_{0},a_{0},X) for subset X\subset\mathcal{S} is defined as cumulative discounted time spend at X starting at (s_{0},a_{0}) and following \pi thereafter, defined as (for finite state-action space case) M^{\pi}(s_{0},a_{0},X)=\sum_{t\geq 0}\gamma^{t}\mathrm{P}(s_{t+1}\in X|s_{0},a_{0},\pi) (1) with the corresponding Q-function for a specific