On the Role of Iterative Computation in Reinforcement Learning
Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko, Benjamin Eysenbach · 2026
Read On the Role of Iterative Computation in Reinforcement Learning by Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko, Benjamin Eysenbach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05999.txt.
Introduction
In reinforcement learning, compute and parameter count are usually tangled: bigger nets both hold more and compute more. Raj Ghugare and three co-authors untangle them by formalising policies with a hard compute budget. They prove that a policy allowed more compute can solve tasks and stretch to longer horizons that a cheaper policy cannot reach, even when parameter counts stay fixed.
They build a minimal variable-compute architecture from earlier work on algorithmic learning and model-free planning. Across thirty-one tasks, online and offline, more compute alone lifts performance, and long-horizon generalisation beats feedforward and deep residual nets with up to five times more parameters. The paper argues that how much a policy thinks, not just how big it is, sets what it can learn.
Catalog record
| CREATOR | Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko, Benjamin Eysenbach |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05999
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Role of Iterative Computation in Reinforcement Learning Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko, Benjamin Eysenbach Abstract How does the amount of compute available to a reinforcement learning (RL) policy affect its learning? Can policies using a fixed amount of parameters, still benefit from additional compute? The standard RL framework does not provide a language to answer these questions formally. Empirically, deep RL policies are often parameterized as neural networks with static architectures, conflating the amount of compute and the number of parameters. In this paper, we formalize compute bounded policies and prove that policies which use more compute can solve problems and generalize to longer-horizon tasks that are outside the scope of policies with less compute. Building on prior work in algorithmic learning and model-free planning, we propose a minimal architecture that can use a variable amount of compute. Our experiments complement our theory. On a set 31 different tasks spanning online and offline RL, we show that ( 1 ) (1) this architecture achieves stronger performance simply by using more compute, and ( 2 ) (2) stronger generalization on longer-horizon test tasks compared to standard feedforward networks or deep residual network using up to 5 times more parameters. 1 1 1 The code for our experiments is available on this website . † † affiliationnotice: * Equal contribution, author order decided via alphabetical order of the last name 1 Introduction The standard view in reinforcement learning (RL) is to treat RL policies as static functions that map states to actions. Policies typically spend a fixed amount of compute, regardless of the complexity of making the correct decision in the underlying state. For example, for a humanoid robot, the action of where to move the right foot is easy, but figuring out how to efficiently pack furniture in a truck requires more deliberation. In RL, this limitation is often solved by citing Moore’s law ( Moore 1965 ) and simply scaling the number of policy parameters. But the compute required to make good decisions in different situations can span many orders of magnitude; using a static massive policy with a large number of parameters is wasteful. Additionally, as compute becomes cheap, the complexity of the world and sensory data also increase ( Javed & Sutton 2024 ) . In this work we advocate for a computational view of RL, where computation time and parameter count are distinct axes. The first primary contribution of our paper is to take a step towards formalizing this view. We formulate RL policies as bounded models of computation and prove that there exist MDPs where policies with more compute time are arbitrarily better than policies with less compute. Intuitively, policies with more compute time should learn more generalizing solutions, where static or computationally limited policies might instead learn heuristics which overfit their training ( Zhou et al. 2023 ; Zhou et al. 2024b ) . In this vein, we prove that there exist MDPs where policies with less compute can overfit on the training tasks and fail to generalize, whereas policies with more compute can generalize to longer-horizon tasks during evaluation ( Myers et al. 2025 ) . The second primary contribution of our work is to empirically show that RL policies which use more compute achieve stronger performance as well as stronger generalization to longer-horizon unseen tasks . Indeed, prior empirical work has also recognized the importance of using more computation to improve the performance of RL agents. Model-Based RL attempts this by learning a transition model and performing search using a planning algorithm ( Sutton 1991 ) . Prior work in model-free planning uses various inductive biases like tree-search ( Farquhar et al. 2018 ; Coulom 2007 ) or latent space planning ( Oh et al. 2017 ; Silver et al. 2017 ) to train policies which generate solutions iteratively. Notably Guez et al. 2018 ; Tamar et al. 2016 have shown that using iterative architectures for RL policies with purely model-free RL objectives can display planning-like behavior ( Bush et al. 2025 ) . One limitation of these works is that the architectures proposed are restricted to tasks with a complete top down view. In our work, we propose a minimal recurrent architecture that can be trained on any task, to use a variable amount of compute with the same number of parameters ( Figure 1 ). On over 30 tasks, we show that these recurrent policies improve significantly when using more recurrent steps. When trained to reach only nearby goals, we show that these policies outperform residual networks with 5 times more parameters when evaluated on unseen far-away goals. We also propose a way to estimate a numerical measure of the value of compute , analogous to the value of information studied in POMDPs. Our work is reminiscent of recent work on length generalization in language models ( Zhou et al. 2023 ; Zhou et al. 2024b ) that make use of iterative computation to promote this sort of generalization. Our experiments complement these results by empirically demonstrating that iterative computation can unlock horizon generalization in RL, and can do so without a special training dataset or curriculum ( Cho et al. 2024 ; Lee et al. 2025 ) . While prior work often attributes the success of iterative reasoning to the tasks that a model has seen before ( Hanna & Corrado 2025b ; Lester et al. 2021 ) , both our experiments and theory highlight that increasing compute (including via iterative function application) can improve performance on RL tasks in settings without language, pre-training, or multi-task training. 2 Related Work Model-based RL and planning. There has been significant prior work in model-based RL on using additional compute to improve RL policies using planning ( Coulom 2007 ; Schrittwieser et al. 2020 ; Hafner et al. 2019 ; Janner et al. 2021 ; Hansen et al. 2022 ; Heess et al. 2015 ) . The model of the world ( Sutton 1991 ) is typically learned independent of the primary learning process. Previous works have proposed model-learning objectives that support the primary RL objective ( Farahmand et al. 2017 ; Grimm et al. 2020 ; Voelcker et al. 2023 ) or learning policies and models using a single objective ( Eysenbach et al. 2023a ; Ghugare et al. 2023 ) . Our work is more closely related to approaches that propose architectures that can learn to plan, without explicitly learning a model of the world ( Tamar et al. 2016 ; Farquhar et al. 2018 ; Guez et al. 2019b ; Oh et al. 2017 ; Guez et al. 2018 ) . Guez et al. 2019a show that planning-like computations can be learned using a convolutional LSTM architecture ( Shi et al. 2015 ) and a purely model-free RL objective. Recurrent architectures for learning algorithmic solutions. In the supervised learning literature, prior works have shown that recurrent networks ( Rumelhart et al. 1986 ; Hochreiter & Schmidhuber 1997 ; Graves et al. 2014 ; He et al. 2015 ) trained on outputs of an algorithm can learn solutions that extrapolate to much longer inputs ( Graves 2017 ; Wang et al. 2025a ; Jolicoeur-Martineau 2025 ; Geiping et