Optimistic Task Inference for Behavior Foundation Models
Thomas Rupf, Marco Bagatella, Marin Vlastelica, Andreas Krause · 2025
Read Optimistic Task Inference for Behavior Foundation Models by Thomas Rupf, Marco Bagatella, Marin Vlastelica, Andreas Krause (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20264.txt.
Introduction
Thomas Rupf and three co-authors help Behavior Foundation Models learn a new task by acts, not by tagged gains. Such models can pull a strong plan for any gain rule named at test time. The catch is data cost: gains must be scored on a fair set of past acts first. Their method, OpTI-BFM, infers the task from live trial acts at test time.
An upbeat choice rule tracks doubt over gain rules and steers what data to take next. The team proves a regret bound by a direct link to upper-bound bandit rules. On set zero-shot tests, the method finds and tunes a new gain rule in a few rounds, with small extra cost.
Catalog record
| CREATOR | Thomas Rupf, Marco Bagatella, Marin Vlastelica, Andreas Krause |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimistic Task Inference for Behavior Foundation Models Thomas Rupf, Marco Bagatella, Marin Vlastelica, Andreas Krause Abstract Behavior Foundation Models (BFMs) are capable of retrieving high-performing policy for any reward function specified directly at test-time, commonly referred to as zero-shot reinforcement learning (RL). While this is a very efficient process in terms of compute, it can be less so in terms of data: as a standard assumption, BFMs require computing rewards over a non-negligible inference dataset, assuming either access to a functional form of rewards, or significant labeling efforts. To alleviate these limitations, we tackle the problem of task inference purely through interaction with the environment at test-time. We propose OpTI-BFM, an optimistic decision criterion that directly models uncertainty over reward functions and guides BFMs in data collection for task inference. Formally, we provide a regret bound for well-trained BFMs through a direct connection to upper-confidence algorithms for linear bandits. Empirically, we evaluate OpTI-BFM on established zero-shot benchmarks, and observe that it enables successor-features-based BFMs to identify and optimize an unseen reward function in a handful of episodes with minimal compute overhead. Code is available at https://github.com/ThomasRupf/opti-bfm . 1 Introduction Zero-shot reinforcement learning ( Touati et al. 2023 ) has gradually gained relevance as a powerful generalization of standard, single-reward RL ( Sutton & Barto 2018 ; Silver et al. 2021 ) . Zero-shot agents are designed to distill optimal policies for a set of reward functions from unlabeled, offline data ( Touati & Ollivier 2021 ; Agarwal et al. 2024 ; Park et al. 2024 ; Jajoo et al. 2025 ) . As these methods scale to more complex and broader environments, they are often referred to as Behavior Foundation Models (BFMs) ( Park et al. 2024 ; Tirinzoni et al. 2025 ) . At their core, the majority of BFMs rely on Universal Successor Features ( Ma et al. 2020 , USFs). These methods build upon explicit representations of states ( features ): given a set of policies, the expected discounted sum of features along each policy’s trajectory may be estimated completely offline (e.g. , through TD learning) ( Dayan 1993 ; Barreto et al. 2017 ) . As long as a reward function lies within the span of features, zero-shot policy evaluation can be performed through a simple scalar product, which in turn enables zero-shot policy improvement, i.e. , each policy is implicitly paired with a reward function and updated towards optimality ( Touati & Ollivier 2021 ) . Once this unsupervised pre-training phase is complete, USFs yield a set of learned policies. Given a reward function that is linear in the features, its optimal policy is by construction indexed by the linear weights describing the reward function in the basis of features, which can in turn be interpreted as task embeddings . The process of finding the optimal policy, which we refer to as task inference , thus corresponds to solving a linear regression problem. While this process is remarkably efficient in terms of compute, it retains strict requirements in terms of data: a dataset of labeled (state, reward) pairs needs to be provided. In simple settings, this necessity is not particularly problematic: BFMs have been largely trained in simulations ( Touati et al. 2023 ; Park et al. 2024 ; Tirinzoni et al. 2025 ) , which makes generating reward labels for the pre-training dataset, or additional data, particularly convenient. Realistically, however, (i) the pre-training dataset might be unavailable or proprietary and, most importantly, (ii) labeling states with rewards might incur significant costs. For instance, when BFMs are pre-trained directly from pixels, evaluating success from a single image is not a straightforward or cheap operation. Figure 1: In contrast to the standard offline task inference pipeline for BFMs, we explore an alternative online framework: instead of producing a point-estimate of the task embedding from an existing dataset, we actively collect data to build a belief over task embeddings, which results in milder labeling requirements and fast retrieval of a near-optimal policy. In order to alleviate these issues, we explore an alternative framework for task inference, which instead aims to collect a small amount of data directly during deployment (see Figure 1 ). This removes the need to access the pre-training dataset (i) and may require fewer labels (ii), as the data can be collected actively . To navigate this setting, we propose OpTI-BFM, a decision criterion that curates a sequence of task embeddings. Starting from an uninformed prior, OpTI-BFM leverages the linear relationship between features and rewards to update its belief over the space of rewards. A BFM conditioned on meaningfully chosen task embeddings can then interact with the environment for a few steps: labeling the states that are visited allows accurate task estimation with minimal labeling efforts. Crucially, if the underlying BFM is well-trained, we show that control reduces to a bandit problem over tasks. Under this assumption, we can provide regret guarantees for OpTI-BFM. When considering established zero-shot benchmarks in the Deepmind Control Suite (DMC) ( Tassa et al. 2018 ) , we find that OpTI-BFM requires only a handful of episodes to correctly identify the task, eventually matching or surpassing the performance that the standard offline reward inference pipeline achieves with significantly more data. 2 Preliminaries Notation We model the environment as a reward-free Markov Decision Process (MDP) \mathcal{M}=(S,A,P,\mu,\gamma) where S and A are state and action spaces, respectively, P(ds^{\prime}|s,a) is a probability measure describing the likelihood of transitions, \mu(ds) is a measure describing the initial state distribution, and \gamma is a discount factor. Given a policy \pi:S\to\Delta(A) and a state-action pair (s_{0},a_{0})\in S\times A we use \mathbb{E}\left[{\cdot}\ \middle|\ {s_{0},a_{0},\pi}\right] to denote expectations w.r.t. state-action sequences ((s_{t},a_{t}))_{t\geq 0} defined by sampling a_{t}\sim\pi(s_{t}) and s_{t+1}\sim P(s_{t},a_{t}) . For a given reward function r:S\to\mathbb{R} , we define the policy’s state-action value function as the discounted sum of future rewards Q_{r}^{\pi}(s_{0},a_{0})=\sum_{t\geq 0}\gamma^{t}\mathbb{E}\left[{r(s_{t})}\ \middle|\ {s_{0},a_{0},\pi}\right] . Successor Features and Behavior Foundation Models For a specific feature map \phi:S\to\mathbb{R}^{d} , Successor Features ( Barreto et al. 2017 , SFs) generalize state-action value functions by modeling the expected discounted sum of features under a policy \pi : \displaystyle\psi^{\pi}(s_{0},a_{0})=\sum_{t\geq 0}\gamma^{t}\,\mathbb{E}\left[{\phi(s_{t})}\ \middle|\ {s_{0},a_{0},\pi}\right]. (1) SFs allow zero-shot policy evaluation for any reward that lies in the span of the features: if r(s)=z^{\top}\phi(s) for some z\in\mathbb{R}^{d} , the Q-function is a linear function of SFs: \displaystyle Q_{r}^{\pi}(s_{0},a_{0})=\sum_{t\geq 0}\gamma^{t}\,\mathbb{E}\left[{r(s_{t})}\ \middle|\ {s_{0},a_{0},\pi}\right]=\sum_{t\geq 0}\gamma^{t}\,\mathbb{E}\left[{z^{\top}\phi(s_{t})}\ \middle|\ {s_{0},a_{0},\pi}\right]=z^{\top}\psi^{\pi}(s_{0},a_{0}) (2) A similar structure also holds for value functions: V_{r}^{\pi}(s)=z^{\top}\psi^{\pi}(s) , when defining \psi^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}\psi^{\pi}(s,a) . Behavior Foundation Models (BFMs) generally 1 1 1 There are BFMs that are not based on USFs. We refer the reader to Agarwal et al. 2025a for a comprehensive overview. capitalize on the opportunity of evaluating policies for multiple reward functions, by additionally learning a family of parameterized policies (\pi_{z})_{z\in\mathcal{Z}} with respect to all rewards in the span of features \phi ( Borsa et al. 2018 ; Touati & Ollivier 2021 ; Park et al. 2024 ; Agarwal et al. 2025b ) . Concretely, BFMs train a family of parameterized policies so that each policy \pi_{z} is optimal for the reward function r(s)=\phi(s)^{\top}z : \displaystyle\textstyle\pi_{z}(a|s)\in\argmax_{a}\psi^{\pi_{z}}(s,a)^{\top}z\quad\text{for each }z\in\mathcal{Z}. (3) In practice, the set of policies, their SFs, and features \phi may be represented through function