Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning

Stefan Stojanovic, Alexandre Proutiere · 2026

Read Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning by Stefan Stojanovic, Alexandre Proutiere (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13207.txt.

Introduction

Stojanovic and Proutiere extend successor measures so hierarchy can guide zero shot reinforcement learning. Older hierarchical methods lean on fixed time spans or goal only tasks, limiting them to narrow reward settings. Their switching measures yield both a high level choice policy and a low level control policy from one learned form. Tests on goal tasks and general reward tasks beat flat baselines and match top hierarchical methods on goal tasks.

Catalog record

CREATORStefan Stojanovic, Alexandre Proutiere
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96706
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13207

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning Stefan Stojanovic, Alexandre Proutiere Abstract Hierarchical reinforcement learning can improve generalization by decomposing long-horizon decision-making into simpler subproblems. However, existing approaches often rely on restrictive design choices, such as fixed temporal abstractions or goal-conditioned objectives, which largely confine them to goal-reaching tasks and limit their applicability to general reward functions. In this paper, we introduce switching successor measures , an extension of successor measures that enables hierarchical control in zero-shot reinforcement learning without additional supervision, fixed horizons, or manually designed subgoals. We show that switching successor measures arise naturally from classical successor measures while preserving their underlying structure. Building on this result, we propose FB π \pi -Switch, an algorithm that extracts both a high-level subgoal-selection policy and a low-level control policy directly from forward-backward (FB) representations, allowing hierarchical behavior to emerge from a single learned representation. Experiments on both goal-conditioned and general reward-based tasks show that FB π \pi -Switch improves over non-hierarchical baselines and matches state-of-the-art hierarchical methods in goal-conditioned settings. These results demonstrate that structured successor representations provide a flexible foundation for hierarchical zero-shot reinforcement learning beyond goal-reaching tasks. Our project website is available at: https://stestokth.github.io/switching-successors/ . 1 Introduction A central challenge in reinforcement learning (RL) is to design methods that are flexible and generalize well across a wide range of reward functions, enabling agents to adapt to new tasks without retraining from scratch. Goal-conditioned reinforcement learning (GCRL) has shown that structuring tasks around goals can significantly improve generalization ( 12 ; 18 ; 59 ) . Hierarchical approaches further extend this idea by decomposing long-horizon problems into sequences of shorter subproblems. For instance, methods such as HIQL ( 41 ) demonstrate that conditioning on intermediate subgoals, rather than only on global task specifications, can substantially improve performance. However, these approaches rely on strong design choices. In particular, HIQL enforces locality through fixed subgoal horizons, rather than allowing such locality to emerge from the learning process itself. Moreover, these methods are primarily studied in goal-conditioned settings, where tasks reduce to reaching specific states. In contrast, general RL involves arbitrary reward functions that cannot always be expressed as goal-reaching objectives. A prominent line of work addresses this setting through forward-backward (FB) representations of successor measures, which aim to represent value functions and policies in a shared embedding space ( 57 ; 58 ; 56 ) . Despite their promise, these approaches rely on a strong global factorization assumption of the form F(s,a,z)^{\top}B(g) for successor measures, which must simultaneously capture the dynamics of the environment and all downstream reward functions. Recent works have shown that such global low-dimensional structure can be difficult to learn and may be too restrictive in complex environments ( 11 ; 3 ; 20 ; 16 ) . This suggests that successor representations may be most reliable locally: they can capture short-range transitions effectively, while their accuracy may deteriorate as the effective planning horizon increases. Supporting this view, recent work on compositional planning ( 19 ) shows that combining local behaviors can improve generalization, although it requires an additional generative model. Finally, several recent methods treat representation learning as a first stage in zero-shot RL pipelines, with policy learning performed separately using standard offline algorithms ( 21 ; 42 ; 20 ) . While effective, this separation does not fully exploit the structure encoded in successor representations for joint planning and control. Together, these observations highlight an important gap: there is currently no unified approach that leverages successor representations to derive hierarchical policies directly from the learned representation, while retaining the ability to generalize across arbitrary reward functions. This motivates our central question: Can successor representations serve as a unified foundation from which both reward embeddings and hierarchical control emerge in zero-shot reinforcement learning? We address this question by introducing switching successor measures that allow us to derive hierarchical policies directly from successor representations, rather than introducing hierarchy as a separate supervised object. Concretely, in each state s , the agent first selects a latent subgoal w using a high-level policy \pi^{h}(\cdot|s,r) conditioned on the reward function r , and then acts according to a low-level policy \pi^{\ell}(\cdot|s,w) conditioned on the selected subgoal. Our main contributions are: (a) We introduce switching successor measures as a principled framework for extracting hierarchical structure from successor-based representations. A key insight is that the switching successor measures needed for high-level planning can be derived directly from a single classical successor measure. We establish this remarkable connection in Theorem 1 , showing that hierarchy is already implicitly encoded in standard successor representations and can be recovered without additional learning. (b) We propose FB \pi -Switch , a hierarchical zero-shot RL algorithm in which both the high-level subgoal-selection policy and the low-level control policy are derived from FB successor representations. The method follows a three-stage training procedure; in particular, the high-level policy learning stage can be incorporated into existing FB algorithms. (c) We empirically evaluate FB \pi -Switch on both goal-conditioned and more general reward-based tasks. The method achieves strong performance without explicit locality-inducing hyperparameters, and consistently outperforms other baselines, including FB with an induced high-level policy. 2 Related work Successor representations and skill-based methods. Successor features ( 9 ) provide a temporal-difference framework for zero-shot learning by decoupling dynamics from rewards. This idea has been extended to successor measures via FB representations, enabling zero-shot adaptation to new tasks ( 58 ) . Subsequent work improves these models through online embedding adaptation ( 52 ) and more stable training objectives ( 63 ) . Alternative parameterizations have also been explored, e.g., intention-conditioned value functions (ICVFs) ( 21 ) and proto successor measures ( 3 ) , which relax the linear reward assumption but require a discrete policy codebook. A related line of work learns reusable skills without explicit successor structure, including unsupervised skill discovery in online ( 17 ) and offline settings ( 5 ) , as well as methods that improve skill transfer ( 20 ) . More broadly, GCRL methods such as UVFA ( 50 ) and contrastive approaches ( 18 ) learn policies conditioned on goals and can be viewed as a special case of successor-based formulations. However, these approaches typically rely on conditioning on fixed goals and do not explicitly capture hierarchical composition. Hierarchical reinforcement learning. Hierarchical RL studies temporal abstraction, such as options, skills, and goals. The option framework ( 54 ) provides a principled formulation for hierarchical RL, with extensions including gradient-based methods ( 15 ) and actor-critic architectures ( 7 ) . Composing pretrained policies has also been explored in prior work ( 35 ; 44 ) , while subgoal-based methods such as HIRO ( 38 ) delegate goal generation to a high-level policy under a fixed reward function. Subgoal-based approaches have been widely studied in both online ( 28 ; 12 ; 23 ) and offline goal-conditioned RL. In particular, HIQL ( 41 ) extends hierarchical RL to the offline regime, with subsequent work improving subgoal generation ( 27 ; 13 ; 43 ) and value estimation ( 31 ; 22 ) . Hierarchical structure can also be induced in skill-based frameworks, e.g., by optimizing for midpoints in latent space ( 42 ) , or

Read the complete text