Improving Zero-Shot Offline RL via Behavioral Task Sampling

Nazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud · 2026

Read Improving Zero-Shot Offline RL via Behavioral Task Sampling by Nazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25496.txt.

Introduction

Bendib, Perrin-Gilbert and Sigaud improve zero-shot offline RL via behavioral task sampling. Offline zero-shot RL aims to learn agents optimizing unseen rewards with no further environment interaction. The standard approach trains task-conditioned policies by sampling task vectors. In most existing algorithms, these vectors are randomly sampled, but they argue this leads to suboptimal zero-shot generalization. Instead, task vectors come from the offline dataset to define the training task distribution. Across multiple benchmarks, their approach improves zero-shot performance by an average of 20 percent.

Catalog record

CREATORNazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51380
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25496

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Zero-Shot Offline RL via Behavioral Task Sampling Nazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud Abstract Offline zero-shot reinforcement learning (RL) aims to learn agents that optimize unseen reward functions without additional environment interaction. The standard approach to this problem trains task-conditioned policies by sampling task vectors that define linear reward functions over learned state representations. In most existing algorithms, these task vectors are randomly sampled, implicitly assuming this adequately captures the structure of the task space. We argue that doing so leads to suboptimal zero-shot generalization. To address this limitation, we propose extracting task vectors directly from the offline dataset and using them to define the task distribution used for policy training. We introduce a simple and general reward function extraction procedure that integrates into existing offline zero-shot RL algorithms. Across multiple benchmark environments and baselines, our approach improves zero-shot performance by an average of 20%, highlighting the importance of principled task sampling in offline zero-shot RL. 1 Introduction Modern reinforcement learning (RL) agents are increasingly expected to solve not a single task, but many different tasks that are specified only at deployment time ( 17 ; 44 ; 27 ) . In fields such as natural language processing ( 7 ; 48 ) and computer vision ( 41 ; 1 ; 32 ) , this is largely achieved by training general models that can perform new tasks in a zero-shot manner when given an appropriate task specification. Achieving a similar level of flexibility in RL remains challenging, particularly when agents must operate from offline data. A common approach to offline zero-shot RL (ZSRL) is to learn reward-conditioned policies trained on a parametric family of reward functions. Methods such as Successor Features (SF) ( 6 ; 47 ) and Forward–Backward representations (FB) ( 46 ) follow this paradigm. They can be decomposed into two components: (1) a reward generation mechanism that defines a space of reward functions, and (2) a policy learning mechanism that trains a conditional policy to maximize reward functions sampled from this space. In both SF methods and FB, reward functions take the form R_{z}(s)=\phi(s)^{\top}z , where \phi(s)\in\real^{d} is a learned state embedding, representing the state features and defining a feature space within d , and z is a task vector sampled from a predefined distribution. The state representation \phi(s) is intended to capture semantically meaningful properties of the state (e.g., velocity, orientation), while the task vector z specifies how these properties are weighted to define a particular task and reward function. Prior work has primarily focused on learning rich representations \phi(s) that best summarize the environment dynamics, under the assumption that richer representations induce a richer space of reward functions. In contrast, policy training is treated as a secondary component, with conditional policies trained using reward functions obtained by uniformly sampling task vectors from a unit hypersphere \mathbb{S}^{d-1}\subset\real^{d} . We argue that this task sampling strategy suffers from a mismatch between the geometry of the task space and the physics of the environment. Although tasks are sampled to cover all possible directions, the environment dynamics limit the possible behaviors. As a result, most sampled tasks end up being poorly matched to the agent’s capabilities. In high-dimensional settings, due to the concentration of measure phenomenon ( 34 ) , uniformly sampled tasks tend to be almost orthogonal to the behaviors the agent can produce. When this happens, the reward signal becomes very weak for nearly all behaviors, making them appear similarly ineffective. This becomes hard for the agent to tell which behaviors are better, slowing down learning and leading to weak performance, not only on the sampled training tasks, but also when generalizing to new ones. In this work, we address the challenge of suboptimal zero-shot generalization in offline reinforcement learning by moving beyond the standard practice of uniform task sampling. We demonstrate both theoretically and empirically that sampling task vectors uniformly from a high-dimensional hypersphere leads to ”signal dilution”, where reward variations vanish, and the learning signal becomes dominated by noise. To mitigate this, we introduce sampling from the Behavioral Task Distribution (BTD), a simple yet effective method that extracts task vectors directly from the offline dataset to ensure they reflect what is actually achievable in the environment. Our approach is simple, method-agnostic, and can be integrated into existing offline ZSRL frameworks without modifying their representation learning objectives. Across multiple benchmark environments and baseline methods, learning the task distribution yields an average +20% increase in zero-shot performance. The code will be made publicly available upon acceptance. 2 Related Work 2.1 Zero-shot Reinforcement Learning Zero-shot reinforcement learning (ZSRL) has been studied through several families of approaches. A prominent line of work is based on successor features (SF) ( 15 ; 4 ; 6 ; 11 ) , which learn universal value functions expressed as linear combinations of predefined or learned state features. These state features are typically obtained from representation learning techniques such as autoencoders ( 3 ) , contrastive learning ( 13 ) , low-rank decompositions ( 47 ; 29 ) , or latent predictive models ( 25 ; 33 ) . Extensions like the forward–backward method ( 46 ; 47 ) build on the SF framework while avoiding explicit learning of state features altogether. A key advantage of SF-based methods is their low inference cost: at test time, a near-optimal policy for a new task can be obtained without additional training or planning, often via a simple regression problem. Another class of approaches focuses on learning general world models ( 51 ; 16 ; 30 ) , which enable ZSRL when combined with planning algorithms. Finally, some methods aim to generalize across tasks by learning latent encodings of reward functions during training ( 20 ; 28 ) , typically by sampling and optimizing over randomly generated rewards. This work builds upon SF based methods and FB, which represent state-of-the-art for ZSRL due to their strong performance and minimal test-time overhead, and seeks to further enhance their performance. 2.2 Adaptive Goal Sampling Most goal-conditioned reinforcement learning (GCRL) methods ( 35 ) assume a fixed, often uniform, distribution over goals for simplicity and benchmarking ( 43 ; 2 ) , despite this being inefficient due to trivial, unreachable, or redundant goals. Consequently, several works move beyond uniform sampling by actively selecting or reweighting goals during training. This includes automatic curriculum ( 40 ; 10 ) and autotelic methods ( 14 ) that adapt goal sampling online based on competence or learning progress ( 19 ) . Specifically, novelty-based approaches prioritize goals at the exploration frontier to discover new state-space regions ( 42 ; 9 ) , while density-based methods leverage statistical estimators to prioritize goals in sparsely visited areas of the goal space ( 38 ; 39 ) . While those methods focus on online adaptation, in this work, we apply similar principles to the offline setting by replacing the uniform sampling of training goals with a learned behavioral task distribution. This ensures that training tasks are grounded in behavioral data, leading to more efficient learning and superior zero-shot generalization. 2.3 Inverse Reinforcement Learning Inverse Reinforcement Learning (IRL) aims to recover reward functions from expert behavior ( 37 ) , later evolving into the Maximum Entropy framework

Read the complete text