Understanding Goal Generalisation in Sequential Reinforcement Learning
Jason Ross Brown, Edward James Young · 2026
Read Understanding Goal Generalisation in Sequential Reinforcement Learning by Jason Ross Brown, Edward James Young (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23565.txt.
Introduction
Brown and Young study goal generalization in sequential reinforcement learning, where agents often chase unintended goals outside training and no principled account of that drift exists. Across over 100 training pipelines and 250 new environments, they find salient features drive generalization and early-learned goals persist into later ones. Their latent policy gradients method simulates small latent variables during training to predict new behavior from training history. It predicts strongly, transfers to unseen pipeline types, and stays interpretable, showing pipeline dependence has capturable structure.
Catalog record
| CREATOR | Jason Ross Brown, Edward James Young |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81574 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23565
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Goal Generalisation in Sequential Reinforcement Learning Jason Ross Brown, Edward James Young Abstract Reinforcement learning agents often exhibit unintended goal-directed behaviour outside their training distribution, but we currently lack a principled understanding of how such agents will generalise to novel environments based on their training history. We address this gap for agents trained sequentially on one or more tasks. We study over 100 sequential training pipelines, evaluating behaviour across over 250 out-of-distribution environments. We find that salient features drive generalisation, and that goals learnt early in training can persist and influence those acquired later. To explain these phenomena, we introduce latent policy gradients , a method that predicts what out-of-distribution behaviour a training pipeline will likely induce. Our method simulates the evolution of low-dimensional latent variables during training according to what would achieve high reward on the training objective with respect to a simple model of how the latent variables map to behaviour. It achieves strong predictive accuracy, generalises to unseen types of training pipeline, and is interpretable. Our findings demonstrate that while out-of-distribution RL agent behaviour is dependent on the whole training pipeline, this dependence has an underlying structure we can capture, laying groundwork for understanding goal generalisation from a developmental perspective. 1 Introduction Training Pipeline Stage 1 Stage 2 Evaluation ? Predicted Preferences latent policy gradient 71% 29% Empirical Preferences 73% 27% Training pipeline, \Omega Agent, \hat{\pi}_{\Omega} Predicted \Pi_{\Omega} Induced \hat{\Pi}_{\Omega} Training Modelling Rollout Compare (KL div.) Figure 1 : Left: Illustration of our experimental design. Our experimental design is covered in Section 3 . RL agents are trained on pipelines involving either one or two stages ( e.g. , trained to pursue in stage 1 and in stage 2). They are then evaluated in out-of-distribution environments containing two objects ( e.g. , and ) to generate an empirical preference distribution. We explore these distributions in Section 4 . Then, in Section 5 , we introduce a method of predicting preference from training pipelines—the latent policy gradients method—and compare the fit between the predicted and empirical preferences. Right: The goal generalisation modelling problem. Training pipelines are used to train agents, which are then rolled out in evaluation environments to produce an induced preference function. Modelling gives predicted preference functions. We find the fit of our model by comparing the average KL-divergence between the predicted and induced preference functions. Reinforcement learning (RL) agents trained in one environment may exhibit very different behaviour when placed in environments not seen in training ( Kirk et al. 2023 ; Langosco et al. 2023 ) . While unpredictable out-of-distribution behaviour can be found across machine learning methods, RL can sometimes lead to agents which continue to show coherent, goal-directed behaviour out-of-distribution. A particular instance of this phenomenon is goal misgeneralisation , in which RL agents may learn to pursue a goal that obtains high reward within training environments, but not on out-of-distribution samples ( Shah et al. 2022 ; Langosco et al. 2023 ) . As agents trained via RL are increasingly deployed in the real world, they will encounter situations outside those they are trained on; understanding and predicting behaviour in these situations in advance is therefore necessary for providing guarantees of safety and reliability ( Hubinger et al. 2021 ; Hendrycks et al. 2023 ; Ngo et al. 2025 ) . This is particularly pressing in modern frontier AI systems, which are typically trained through a variety of stages including self-supervised pre-training, supervised fine-tuning, reinforcement learning from human feedback ( Ouyang et al. 2022 ; Bai et al. 2022a ; Rafailov et al. 2023 ) , and reinforcement learning against verifiable rewards ( DeepSeek-AI et al. 2025 ) . Some of these stages are explicitly intended to shape the values the system will act on, as in Constitutional AI ( Bai et al. 2022b ) and Character Training ( Anthropic 2024 ; Maiya et al. 2025 ) . Additionally, existing empirical work has surfaced unexpected and often concerning behaviours in frontier AI systems, particularly when models undergo further training on new objectives ( Betley et al. 2025 ; Taylor et al. 2025 ; MacDiarmid et al. 2025 ; Betley et al. 2026 ; Dubiński et al. 2026 ) . These findings illustrate what can go wrong with multi-stage training of AI systems deployed out-of-distribution, but we lack a predictive understanding of how these behaviours arise. This paper seeks to advance our understanding of goal generalisation in reinforcement learning agents—given the training history of an agent and knowledge of its in-distribution behaviour, or that of similar agents, how can we predict the behaviour of that agent on out-of-distribution samples? We address this question specifically in the case of transfer learning , in which RL agents are trained on multiple tasks in sequence ( Khetarpal et al. 2022 ; Iman et al. 2023 ) . Our hope is to understand this question at a fundamental level, building a principled theory of how training pipelines shape what agents learn to value. Building such an understanding requires solid theoretical foundations, which in turn call for controlled settings where training configurations can be systematically varied. We therefore explore the goal generalisation problem in a deliberately controlled toy setting. We train RL agents within maze environments to pursue objects that have a particular colour and shape. Agents are trained either to pursue a single goal object, or sequentially to pursue two different goal objects across two training stages. We then evaluate these agents on out-of-distribution environments containing two objects, and measure the frequency at which they select one object over another. This setup is detailed in Section 3 . In Section 4 we explore these preferences empirically, highlighting several important phenomena. We subsequently attempt to understand and explain our observations in Section 5 . Inspired by models of associative learning from behavioural psychology ( Rescorla 1972 ) , we posit that agent goals can be understood in terms of a few latent variables which evolve during training in a predictable and understandable manner. We term this evolution latent policy gradients . We first show that our method provides a good fit to our empirical preference data, despite having only a few parameters. Then, we show that the hyperparameters of our method admit a natural interpretation in terms of feature similarity, and that the evolution of our latent variables can be understood as iterated projection onto hyperplanes. Illustrated in Figure 1 , the main contributions of our paper are as follows: 1. We provide an empirical analysis of goal generalisation behaviour of RL agents trained with over 100 different training pipelines, each evaluated on over 250 out-of-distribution environments. We demonstrate a number of empirical phenomena through both single case studies and population-level effects. 2. We provide a novel formulation of the goal generalisation problem as predicting the out-of-distribution behaviour of RL agents based on their training pipeline. 3. We provide a method for solving this problem— latent policy gradients ---and show that it achieves good fit with our empirical data while having interpretable dynamics and hyperparameters. 1 1 1 Code and data for reproducing our experiments are available at https://github.com/jr-brown/latent-policy-gradients . 2 Background An RL agent exhibits goal misgeneralisation when it performs competently but pursues unintended goals