From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers
Ching Fang, Kanaka Rajan · 2025
Read From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers by Ching Fang, Kanaka Rajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19686.txt.
Introduction
Ching Fang and Kanaka Rajan peer inside a transformer model to see how it learns to plan from raw experience, in a 2025 paper. Animals learn new mazes fast, drawing on remembered episodes rather than slow trial-and-error updates. Standard learning algorithms do not capture that speed. Transformers, whose memory design echoes the brain's episode store, offer a place to study the trick. The authors trained one on sets of planning tasks modeled on rodent behaviour.
Inside the trained model, they found the machine builds shared maps across tasks: layouts learned in one setting line up with another's, even when the sights differ. Its strategy matched neither textbook style of planning, neither step-by-step value updates nor full mental simulation. The paper argues the model grows a third kind of method, which may hint at how brains learn so quickly.
Catalog record
| CREATOR | Ching Fang, Kanaka Rajan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-19686
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers Thanks: corresponding author: Kanaka Rajan Ching Fang, Kanaka Rajan Abstract Humans and animals show remarkable learning efficiency, adapting to new environments with minimal experience. This capability is not well captured by standard reinforcement learning algorithms that rely on incremental value updates. Rapid adaptation likely depends on episodic memory—the ability to retrieve specific past experiences to guide decisions in novel contexts. Transformers provide a useful setting for studying these questions because of their ability to learn rapidly in-context and because their key-value architecture resembles episodic memory systems in the brain. We train a transformer to in-context reinforcement learn in a distribution of planning tasks inspired by rodent behavior. We then characterize the learning algorithms that emerge in the model. We first find that representation learning is supported by in-context structure learning and cross-context alignment, where representations are aligned across environments with different sensory stimuli. We next demonstrate that the reinforcement learning strategies developed by the model are not interpretable as standard model-free or model-based planning. Instead, we show that in-context reinforcement learning is supported by caching intermediate computations within the model’s memory tokens, which are then accessed at decision time. Overall, we find that memory may serve as a computational resource, storing both raw experience and cached computations to support flexible behavior. Furthermore, the representations developed in the model resemble computations associated with the hippocampal-entorhinal system in the brain, suggesting that our findings may be relevant for natural cognition. Taken together, our work offers a mechanistic hypothesis for the rapid adaptation that underlies in-context learning in artificial and natural settings. 1 Introduction Animals can learn efficiently from sparse data, rapidly adapting to new environments with minimal experience. For example, humans can infer underlying structure or learn new concepts from just a handful of examples and mice in maze tasks can identify optimal paths after only a few successful trials [ Meister 2022 ] . Standard reinforcement learning (RL) algorithms, which typically rely on incremental value updates to shape decision-making, does not capture this rapid learning behavior well [ Eckstein et al. 2024 ] . One explanation is that biological agents possess useful priors shaped by evolution and experience, allowing them to generalize quickly in naturalistic settings. They also rely on episodic memory, the ability to recall specific past experiences to guide decisions in novel situations. Here, we ask how episodic memory operates not just as storage, but as a computational substrate for rapid learning and decision-making. We train a transformer model to perform in-context reinforcement learning [ Lee et al. 2023 ] on navigation tasks inspired by rodent behavior. In each new environment, the model receives exploratory trajectories as context and infers a goal-directed policy. Transformers are especially relevant not only because of their established capabilities for rapid and flexible in-context learning [ Dong et al. 2022 , Brown et al. 2020 , Lampinen et al. 2024 ] , but also because their key–value memory architecture has been linked to models of episodic memory in the brain [ Krotov and Hopfield 2020 , Tyulmankov et al. 2021 , Fang et al. 2025 ] . Understanding the reinforcement learning strategy that emerges in these models can provide new hypotheses for how memory-based computations might support flexible decision-making in new environments. We focus on two task suites: spatially regular gridworlds and hierarchically structured tree mazes (Fig 1 BC). While both require memory to support goal-directed behavior, they differ sharply in geometry: gridworlds are Euclidean and spatially continuous, whereas tree mazes are non-Euclidean and branch-structured. This contrast allows us to evaluate how learned in-context strategies generalize across structural regimes known to challenge standard sequence models– for instance, language models are known to struggle with symbolic reasoning and hierarchical generalization in tree-structured domains [ Bogin and Berant 2022 , Ruiz and Nachum 2021 , Keysers et al. 2020 ] . In this paper, we make the following contributions: • We show that transformers trained to in-context reinforcement learn develop consistent representation learning strategies: structure learning within contexts, and alignment of representations across contexts with shared regularities. • The model learns computations similar to those that support natural cognition: the representation learning strategies are consistent with suggested roles for hippocampus and entorhinal cortex, and memory retrieval patterns at decision time is consistent with hippocampal replay. • Mechanistic analysis of the emergent RL strategy shows that the model does not use standard model-free or model-based RL methods. • We give descriptions for the in-context RL algorithm that emerged in transformers to solve the two planning tasks. We find strategies tend to rely on intermediate computations cached in memory tokens, demonstrating how episodic memory can be used as a computational workspace. Figure 1: A transformer is trained to in-context reinforcement learn in diverse planning tasks. A. Diagram of the meta-learning setup. Each task in the training distribution is defined by a state transition function \mathcal{T} , a state encoding function \mathcal{S} , and a reward function \mathcal{R} . For each task, the model is trained via supervision to predict the optimal action from a query state s_{\text{query}} , given a context dataset \mathcal{D}=\{(s_{1},a_{1},s^{\prime}_{1},r_{1}),(s_{2},a_{2},s^{\prime}_{2},r_{2}),\dots\} of RL transition tuples sampled from the task. B. Illustration of three training tasks (orange) and one test task (gray) from the gridworld distribution. In each task, the underlying graph structure is fixed (shown in gray), with a reward location (red star) that varies across tasks. Each state is encoded as a random Gaussian vector (bottom). Importantly, state encodings in test tasks are novel. C. As in (B), but for the tree maze distribution. The graph structures in the training set is drawn from probabilistically branching trees, while the test set graph structure is a full binary tree. 2 Related Works Meta-learning to discover in-context reinforcement learning algorithms In order to develop a model that can in-context reinforcement learn, we use a recently introduced meta-learning framework [ Lee et al. 2023 ] . Meta-learning is concerned with “learning-to-learn”, using prior knowledge learned from previous tasks to support rapid adaptation to new ones [ Beck et al. 2023 ] . That is, the goal is to learn an algorithm f that can be deployed in new tasks. The learning of f is called the outer-loop while f itself is referred to as the inner-loop [ Beck et al. 2023 ] . In many settings where f is a RL algorithm (meta-RL), the outer loop shapes the weights of the network, but learning within each task occurs via activation dynamics– through memory and internal state– rather than parameter updates [ Beck et al. 2023 , Lampinen et al. 2024 ] . The resultant f is considered an in-context learning algorithm [ Beck et al. 2023 , Sandbrink and Summerfield 2024 , Lampinen et al. 2024 ] . Early examples of this approach include RL 2 , which meta-trains a recurrent neural network using reinforcement learning in the outer loop, such that an in-context RL strategy emerges in the inner loop [ Wang et al. 2016 , Duan et al. 2016 ] . Subsequent work extended this approach to include explicit episodic memory mechanisms, combining RNNs with key–value memory architectures [ Ritter et al. 2018 , Ritter