Toward Agents That Reason About Their Computation

Adrian Orenstein, Jessica Chen, Gwyneth Anne Delos Santos, Bayley Sapara & Michael Bowling · 2025

Read Toward Agents That Reason About Their Computation by Adrian Orenstein, Jessica Chen, Gwyneth Anne Delos Santos, Bayley Sapara & Michael Bowling (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22833.txt.

Introduction

Skilled humans get faster and use less effort with practice. AI agents rarely do the same. Adrian Orenstein leads a team with four co-authors. In 2025 they test agents that reason about their own compute. Agents see compute cost. They can choose when to spend it.

Tests run in the Arcade Learning Environment. That suite holds classic video games. With the same training budget, such agents did better. They won on three quarters of games. They also used three times less compute on average. Per game analysis shows where savings arise. The work hints at more energy efficient agents.

Catalog record

CREATORAdrian Orenstein, Jessica Chen, Gwyneth Anne Delos Santos, Bayley Sapara & Michael Bowling
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward Agents That Reason About Their Computation Adrian Orenstein, Jessica Chen, Gwyneth Anne Delos Santos, Bayley Sapara & Michael Bowling Abstract While reinforcement learning agents can achieve superhuman performance in many complex tasks, they typically do not become more computationally efficient as they improve. In contrast, humans gradually require less cognitive effort as they become more proficient at a task. If agents could reason about their compute as they learn, could they similarly reduce their computation footprint? If they could, we could have more energy efficient agents or free up compute cycles for other processes like planning. In this paper, we experiment with showing agents the cost of their computation and giving them the ability to control when they use compute. We conduct our experiments on the Arcade Learning Environment, and our results demonstrate that with the same training compute budget, agents that reason about their compute perform better on 75% of games. Furthermore, these agents use 3 3 times less compute on average. We analyze individual games and show where agents gain these efficiencies. 1 Introduction As reinforcement learning (RL) agents get better at a task, they typically do not become more energy efficient with experience. This is unlike humans, who adapt their cognitive process over time and can learn to devote cognitive effort with intention ( Miller and Cohen 2001 ) . Agents, by contrast, often have their computational processes (a.k.a. compute ) fixed; their processes for sensing, acting, learning and planning do not change after design time. So, when the problem demands more computation in some moments and less in others, the agent’s computational process does not adapt. There are many ways that agents are designed to use compute, and these choices have mostly been found by agent designers ( Mnih et al. 2015 ; Silver et al. 2016 ; Vinyals et al. 2019 ; Bellemare et al. 2020 ; Wurman et al. 2022 ; Degrave et al. 2022 ; Mankowitz et al. 2023 , e.g.,) . Designers shape how and with what frequency the agent should use compute so that it can more effectively scale with available resources. As one simple example, the Arcade Learning Environment ( Bellemare et al. 2013 ) (ALE) outputs 60 frames per second. Instead of making agents process each frame, making the agent process only every fifth frame as is the default with the ALE speeds up the execution by five times, with a negligible reduction in performance ( Bellemare et al. 2012 ) . If agents could control their compute and received feedback on how they used it, they could begin to make these choices for themselves. But at present, designers choose how compute is used — designers decide when the agent should act or learn or plan, and the “cost” of compute is not made available to the agent. Additionally, agents are not typically long lived ( Sutton et al. 2022 ) , they are created to use compute in a certain way and replaced once their design limits them. This creates a cycle of design : where an agent is created, their performance is observed, the design is evaluated, and a new agent is improved that fits the trade-off known at design time. This cycle of design suggests that agents could do the same for themselves if they had access to similar actions and received feedback about their compute costs. This work represents a foray into agents that can reason about their own computational processes. Since agents are capable of learning to make effective choices within the RL formalism, we will use the very same RL formalism to frame the challenge of reasoning about compute. We provide the agent with additional actions that do not have direct external effects, but rather change the agent’s usage of compute. Additionally, we provide feedback that makes the cost of compute explicit alongside traditional environment reward. We focus on providing the agent with actions that change the rate at which observations are processed and environment actions are computed, while modifying its rewards to reflect the cost of computation. We will show that in this setting agents can effectively learn to adapt their computational processes at run time from just their experience. Our contributions are as follows: 1. We show that an agent with the ability to control its compute and see the consequences of its computation can learn to adapt its compute at run time. 2. These agents perform better on 75% of the Arcade Learning Environment games, while using the same training compute budget. They learn compute efficient behaviors per-game that are on average 3 times more efficient. We show that efficiencies are learned and tailored to each game, and adapt to the changing circumstances within a game. 3. We show that agents are responsive to the experienced compute cost, showing that an agent’s computational processes adapt to the RL task posed to it by the designer. 2 Reasoning about computation In this section, we show that we can use the RL problem formulation to enable agents to reason about their compute. Our control problem consists of observations and actions. The agent processes an observation into a state, S_{t} , at a time step, t , and computes an action, A_{t} , in the same time step. This action is then sent to the environment, and in response, a new observation is perceived in the next time step resulting in the next state S_{t+1} . A special part of the agent’s observation is a scalar reward signal, R_{t+1} , received at every time step. This cycle of interaction repeats in a synchronized manner, and the objective of the RL agent is to maximize the expected episodic discounted return: G=\mathbb{E}\left[\sum_{k=1}^{T-1}\gamma^{k-1}R_{k+1}\right] . 2.1 Framing the objective The agent’s goals are specified in the reward signal ( Sutton 2004 ; Bowling et al. 2023 ) , and so, the scalar reward should also provide the consequences of the agent’s computational processes. For instance, consider how an agent may learn when it is important to act while operating a solar panel tracking station. The solar panel needs to be continually oriented toward the sun to capture sunlight throughout the day, where a better agent is one that maximizes the power exported to the grid. The agent can learn to track the sun accurately by actuating the panel and observing the exported energy change in value. However, the agent’s computation consumes energy as well, which is energy not exported to the grid. If the objective incorporated the energy that has been gathered, r^{\text{task}}_{t} , and the compute cost of the agent, c_{t} , a scalar reward can be constructed representing the total power exported to the grid: r_{t}=r^{\text{task}}_{t}-c_{t} . This frames the problem the designers face — balancing the agent’s compute use with performance on the task, but now allows the agent to see feedback on its compute usage. In the next section, we show how actions that change the cost can enable the agent to choose when to use compute. 2.2 Controlling the computational processes The rate at which agents process observations and take action greatly changes their compute cost. If the agent processes at half the rate, the agent can halve its compute cost.

Read the complete text