Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines
Wilka Carvalho, Sam Hall-McMaster, Honglak Lee, Samuel J. Gershman · 2025
Read Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines by Wilka Carvalho, Sam Hall-McMaster, Honglak Lee, Samuel J. Gershman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05561.txt.
Introduction
Wilka Carvalho, Sam Hall-McMaster, Honglak Lee, and Samuel J Gershman argue that people quietly practice for tasks they are not currently doing, in a 2025 paper. A person can face endless possible tasks but handle only a few at once. The authors suggest the brain uses spare moments on one task to rehearse others that were within reach.
They formalize the idea as multitask preplay: replaying an experience, but steering the replay into a road not taken. In a small grid world, the method matched how humans later handled tasks they had passed by, better than standard planning tools did. The match held in a richer two-dimensional craft game. Agents given the same trick also transferred better to new worlds, suggesting preplay is a scalable way to learn across many tasks.
Catalog record
| CREATOR | Wilka Carvalho, Sam Hall-McMaster, Honglak Lee, Samuel J. Gershman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines Wilka Carvalho, Sam Hall-McMaster, Honglak Lee, Samuel J. Gershman Abstract Humans can pursue a near-infinite variety of tasks, but typically can only pursue a small number at the same time. We hypothesize that humans leverage experience on one task to preemptively learn solutions to other tasks that were accessible but not pursued. We formalize this idea as Multitask Preplay , a novel algorithm that replays experience on one task as the starting point for “preplay”—counterfactual simulation of an accessible but unpursued task. Preplay is used to learn a predictive representation that can support fast, adaptive task performance later on. We first show that, compared to traditional planning and predictive representation methods, multitask preplay better predicts how humans generalize to tasks that were accessible but not pursued in a small grid-world, even when people didn’t know they would need to generalize to these tasks. We then show these predictions generalize to Craftax, a partially observable 2D Minecraft environment. Finally, we show that Multitask Preplay enables artificial agents to learn behaviors that transfer to novel Craftax worlds sharing task co-occurrence structure. These findings demonstrate that Multitask Preplay is a scalable theory of how humans counterfactually learn and generalize across multiple tasks; endowing artificial agents with the same capacity can significantly improve their performance in challenging multitask environments. While searching for coffee in a new neighborhood, you may come across gyms, grocery stores, and parks, enabling you to quickly find these locations later when you need them. One theory of how humans achieve this form of generalization posits that they learn an internal model of the environment, representing what is where kool2016does . “Model-based” algorithms can use this model to chart a path from the current location to any goal. Model-based approaches provide maximum flexibility (generalization to arbitrary goals and starting points), but come at the cost of requiring a possibly prohibitive amount of computation at decision time. At the other extreme, “model-free” approaches directly learn predictions of future states or rewards based on interacting with the environment carvalho2024predictive . For example, model-free deep reinforcement learning (RL) systems modify the weights of a neural network that outputs predictions of future reward based on sensory input mnih2015human . Predictions are effectively “cached” in the sense that they can be retrieved with relatively little additional computation. By choosing the actions that maximize future reward, a near-optimal policy can be cheaply computed. However, this comes at the cost of flexibility: a change in the task can require learning new weights from on-task experience. In between these extremes are approaches that use offline computation (“background planning”) to compile model-based knowledge into cached predictions sutton1990integrated ; gershman2014retrospective . This combines the flexibility of model-based approaches with the computational efficiency of model-free approaches. Classically, such approaches only cache predictions for a single task. This means that they’re limited in their ability to flexibly generalize to new tasks without additional on-task experience, where each new task can require significant additional computation. Generalization is still bottlenecked by the need to rerun planning (either at decision-time or in the background) for each new task. In fact, it is not necessary to run planning from scratch when different tasks share structure. For example, your search for coffee may lead you to a city center where many different shops are located; navigating to any of these potential goals will entail navigating to the city center and you can leverage this experience to learn about each of these goals. This shared structure can be leveraged by parameterizing a reward-predicting neural network to take the goal as input (what we refer to as goal conditioning ; schaul2015universal ). However, learning to predict which actions will accomplish each goal still needs experience achieving that goal. Inspired by an old idea in RL that one can use experience with one goal to learn about many goals ( kaelbling1993learning ) , we hypothesize that people gain this experience by counterfactually simulating pursuit of locally accessible goals and providing this simulated experience as input to a learning algorithm. Doing so then enables people to generalize to these goals later on with relatively little additional computation. We refer to this as Multitask Preplay . Figure 1: Overview of Multitask Preplay. (1) At time t_{1} , an agent experiences a goal-directed task g (e.g., going to a coffee shop). (2) During offline replay, the agent counterfactually simulates alternative accessible tasks g’ (e.g., going to a grocery store) based on the same experience trajectory. Using temporal difference learning, the agent backs up information about both g and g’ to learn a shared predictive representation. This enables it to form an implicit predictive map describing how to reach multiple goals. (3) At a future time t_{2} , if goal {\color[rgb]{0.8906,0.1953,0.1328}{g^{\prime}}} becomes the actual task, the agent can perform it re-using the previously cached predictive representation—without requiring new experience or planning. This architecture allows generalization to unseen but structurally accessible tasks through preemptive counterfactual simulation. In this paper, we describe Multitask Preplay formally and then present a series of human behavioral experiments testing its distinctive predictions. In particular, we present evidence that humans counterfactually simulate pursuit of alternative goals, using these simulations to update cached predictions. We show that our findings hold in both simple grid-worlds and in a complex, partially observable domain (a 2D version of Minecraft) with numerous potential goals. Finally, we show that Multitask Preplay outperforms a range of competitive RL algorithms when an environment has many inter-dependent tasks. Taken together, these results suggest that preemptive counterfactual learning is a powerful algorithmic concept for both natural and artificial intelligence. Preemptively solving future tasks with Multitask Preplay We formalize Multitask Preplay within the framework of sequential decision problems. At any given time t , the environment is described by a state s_{t} . The agent selects an action a_{t} according to a probabilistic, state-dependent policy \pi(a_{t}|s_{t}) , and then receives reward r_{t} . The optimal policy \pi^{\ast} maximizes expected discounted future return, or value , defined as: \displaystyle Q^{\pi}(s,a) \displaystyle=\mathbb{E}\left[r_{1}+\gamma r_{2}+\gamma^{2}r_{3}+\ldots|s_{0}=s,a\sim\pi\right], (1) where \gamma\in[0,1) is a discount factor determining the relative weighting of long-term vs. short-term rewards. The agent can potentially have multiple goals, but is constrained to pursue them sequentially. We use R_{\color[rgb]{0.3359,0.707,0.9141}{g}}(s) to denote the goal-dependent reward function. Accordingly, we use Q(s,a,{\color[rgb]{0.3359,0.707,0.9141}{g}}) to express the fact that different tasks induce different value functions, which can be incorporated into a single “universal” value function schaul2015universal . Value functions are interesting because they can be written recursively. This enables using a form of temporal difference (TD) learning known as Q-learning to estimate it (see Methods for more details). The TD loss function for a single transition is: \displaystyle L_{\tt TD}=\Big(\underbrace{r_{t}+\gamma\max_{a}\hat{Q}_{\theta}^{\pi}(s_{t+1},a,{\color[rgb]{0.3359,0.707,0.9141}{g}})}_{\text{target}}-\underbrace{\hat{Q}_{\theta}^{\pi}(s_{t},a_{t},{\color[rgb]{0.3359,0.707,0.9141}{g}})\vphantom{\max_{a}}}_{\text{prediction}}\Big)^{2}, (2) where \hat{Q}_{\theta}^{\pi}(s_{t},a_{t},{\color[rgb]{0.3359,0.707,0.9141}{g}}) is an approximation of the value function parametrized by \theta . We use deep neural networks to approximate the value function, updating the parameters by backpropagating the gradient of the TD loss through the network. Figure 2: Overview of algorithms. (a) Q-learning is a model-free algorithm that updates cached reward predictions based on interactions with the environment. (b) Successor features are predictive representations that generalize reward-predictive representations to arbitrary features. Notably, if certain features (such as whether groceries are