Finding the Time to Think: Learning Planning Budgets in Real-Time RL
Aneesh Muppidi, Firas Darwish, Dylan Cope, João F. Henriques, Jakob Nicolaus Foerster · 2026
Read Finding the Time to Think: Learning Planning Budgets in Real-Time RL by Aneesh Muppidi, Firas Darwish, Dylan Cope, João F. Henriques, Jakob Nicolaus Foerster (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26463.txt.
Introduction
Aneesh Muppidi and four colleagues study agents that must act while the world keeps moving. In standard reinforcement learning, the environment politely waits for each decision. In real-time settings it does not. Thinking longer means the situation changes more before the agent acts. The authors call this variable-delay real-time reinforcement learning.
The right thinking time depends on the current state. Planning how long to plan can itself paralyse the agent. Their fix is a lightweight gating policy placed over a planner. The gate learns to pick a planning budget, a time allowance, for each state. Tests covered real-time Pac-Man, Tetris, Snake, and two board games. The gating policy beat fixed budgets and hand-written rules. It also transferred to a setup where the game and the agent ran on separate graphics processors.
Catalog record
| CREATOR | Aneesh Muppidi, Firas Darwish, Dylan Cope, João F. Henriques, Jakob Nicolaus Foerster |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26463
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Finding the Time to Think: Learning Planning Budgets in Real-Time RL Aneesh Muppidi, Firas Darwish, Dylan Cope, João F. Henriques, Jakob Nicolaus Foerster Abstract Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent’s decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent’s action. Building on prior real-time formalizations, we introduce variable-delay real-time RL , where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs. Code Checkpoints Project Page 1 Introduction Expert decision-makers do not think equally hard about every choice. They must recognize which decisions require more time or effort to think through and which decisions can be made quickly, as they are constrained in the cognitive resources they can devote to decision-making ( 45 ) . For example, a skilled chess player managing the clock in speed chess will play most moves quickly and reserve their time for critical board positions when they need to think for longer. In standard RL, and Markov Decision Processes (MDPs), the environment waits as the agent may deliberate indefinitely before committing to an action, and computation does not carry any cost. Real-time settings break this assumption: the world keeps progressing even if the agent takes longer to act. The cost of deliberation is not wall-clock latency, but steps taken by the environment while the agent is still deciding. A perfect action executed one moment too late may be far worse than a timely, good-enough action. 48 formalized this concern by fixing the action delay at exactly one timestep. We generalize 48 ’s framework and introduce variable-delay real-time RL , where the agent is a procedure that runs in time rather than a one-frame function, so the time any particular action-producing computation spans is paid as progress in the world. We explore this setting with agents that use anytime planning algorithms to improve the quality of their actions, but must confront that doing so takes time in which the environment continues to evolve. Specifically, we use AlphaZero-style models ( 58 ) that run Monte Carlo Tree Search (MCTS) at decision time to improve action quality (see Appendix B for a primer). A property of these agents is that more search produces better actions but takes longer to run, and we verify empirically (Figure 3 ) that planning quality and inference latency scale together. Our setting introduces three challenges. Firstly, choosing the right delay is state-dependent: some states reward careful planning while others demand immediate reaction. Secondly, the agent must commit to something during the intermediate frames, since the environment will not wait for the planner. Lastly, choosing how long to deliberate is fundamentally a problem of meta-reasoning ( 51 ; 32 ) , which risks an obvious paradox of planning-about-planning : the meta-decision occurs while the environment continues to evolve, so deciding how long to think could incur the same per-frame cost as thinking itself. We address these challenges by training a lightweight gating policy on top of a frozen planner that decides how long to plan at each decision point. Our design escapes the paradox in practice because a single gate forward pass is orders of magnitude cheaper than an MCTS rollout, and therefore adds negligible real-time overhead. We employ our gating policy in real-time Pac-Man, Tetris, Snake, in which the world progresses while the planner runs in the background (Figure 1 ), and clock environments (Speed Hex, Speed Go), in which the board is static but each player has a clock that depletes with thinking time. Section 3 formalizes both. Our contributions are the following: 1. Variable-delay real-time RL. A generalization of 48 ’s fixed-delay framework: under the real-time interaction protocol the agent is a procedure that runs in time rather than a function evaluated instantaneously, and the time spent producing any action is paid as progress in the world. 2. Planning quality vs. inference time. We empirically characterize how planning quality and real-time inference cost co-scale with MCTS simulation count, establishing the joint tradeoff that motivates adaptive allocation. 3. Adaptive gating on a frozen planner. A lightweight gating policy, trained with PPO on top of a frozen AlphaZero planner, that selects a state-dependent planning budget at each decision. Figure 1 : Given the current state, the gating policy chooses whether to react immediately or spend time planning, selecting the number of timesteps k over which to plan. The agent then takes k{-}1 committed actions using \pi_{\mathrm{reflex}} ( \pi_{\mathrm{0}} ) while MCTS plans, and finally executes the planned action. Across five environments, the gating policy which adaptively selects among a set of planning budgets outperforms baselines constrained to use a single fixed planning budget at every step. To validate that the environments and method we designed capture the challenges of real-time decision-making, we transfer the learned gating policy to a real-time hardware setting where the environment runs on one GPU and the agent runs on a second GPU without any retraining or architectural changes to demonstrate that it performs effectively. 2 Related Work Real-time RL and delayed MDPs. The standard MDP assumes the environment waits during action selection. 63 flagged this as a fundamental mismatch with real-time interaction, and 48 formalized an alternative in which the agent has exactly one timestep to compute its next action, mathematically equivalent to a 1-step constant-delay MDP ( 64 ) . Subsequent work extends this to fixed action and observation delays of larger length ( 18 ; 10 ; 34 ) , to concurrent control ( 66 ) , and to asynchronous and pipelined architectures that mitigate inference cost ( 49 ; 2 ) . Across this literature, delay is imposed by the system. We instead let the agent choose its delay at each decision point, turning delay selection itself into the learning problem. Value of computation and meta-reasoning. The view that rational agents must reason not only about what to do but about how much to think traces to bounded rationality ( 59 ) and Good’s Type II rationality ( 22 ) . 51 made this operational by defining the value of computation (VOC) as the expected improvement in decision quality minus its cost, and bounded optimality ( 50 ) reframed rational agency as the optimal use of fixed computational resources. Anytime algorithms ( 16 ; 67 ; 39 ; 36 ) produce monotonically improving answers given more time, and Bayesian flexible-computation methods ( 31 ) formalize when to stop. Applied to MCTS, value-of-computation ideas yield Bayesian formulations of which simulation to run next ( 30 ; 62 ; 56 ; 40 ) , while classical chess- and Go-engine time management allocates clock per move via hand-designed heuristics (