Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
Matthew Riemer, Gopeshh Subbaraj *, Glen Berseth, Irina Rish · 2024
Read Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference by Matthew Riemer, Gopeshh Subbaraj *, Glen Berseth, Irina Rish (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14355.txt.
Introduction
Matthew Riemer, Gopeshh Subbaraj, Glen Berseth, and Irina Rish study reinforcement learning that must act in real time at large scale in a 2024 study. In live settings the world keeps changing while the agent thinks, so slow inference means acting on stale information. Modern networks are bigger and slower, which clashes with the fast reaction such settings demand.
The authors derive lower bounds on regret for real-time environments, showing perfect long-term performance is generally impossible under the usual step-by-step loop. Their staggered asynchronous design lets inference and learning overlap across agents. The analysis reframes what real-time learning systems should promise and how they should be built.
Catalog record
| CREATOR | Matthew Riemer, Gopeshh Subbaraj *, Glen Berseth, Irina Rish |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference Matthew Riemer, Gopeshh Subbaraj *, Glen Berseth, Irina Rish Abstract Realtime environments change even as agents perform action inference and learning, thus requiring high interaction frequencies to effectively minimize regret. However, recent advances in machine learning involve larger neural networks with longer inference times, raising questions about their applicability in realtime systems where reaction time is crucial. We present an analysis of lower bounds on regret in realtime reinforcement learning (RL) environments to show that minimizing long-term regret is generally impossible within the typical sequential interaction and learning paradigm, but often becomes possible when sufficient asynchronous compute is available. We propose novel algorithms for staggering asynchronous inference processes to ensure that actions are taken at consistent time intervals, and demonstrate that use of models with high action inference times is only constrained by the environment’s effective stochasticity over the inference horizon, and not by action frequency. Our analysis shows that the number of inference processes needed scales linearly with increasing inference times while enabling use of models that are multiple orders of magnitude larger than existing approaches when learning from a realtime simulation of Game Boy games such as Pokémon and Tetris. 1 Introduction An often ignored discrepancy between the discrete-time RL framework and the real-world is the fact that the world continues to evolve even while agents are computing their actions. As a result, agents are limited in the types of problems that they can solve because the speed at which they can compute actions dictates a particular stochastic or deterministic time discretization rate. Agents that take infrequent actions require some lower-level program to manage behavior between actions, often through simple policies like remaining still or repeating the last action. Ideally, intelligent agents would exert more control over their environment, but this conflicts with the trend of using larger models, which have high action inference and learning times. Consequently, as typically deployed with sequential interaction, large models, which are often found to be essential for complex tasks, increasingly rely on low-level automation, reducing their control over realtime environments. This paper examines this discrepancy and explores alternative asynchronous interaction paradigms, enabling large models to act quickly and maintain greater control in high-frequency environments. Figure 1(a) shows the standard sequential interaction paradigm of RL. In this setup, the agent receives a state from the environment, learns from the state transition, and then infers an action. Each process must be completed before the agent can process a new state, limiting the action frequency and increasing reliance on low-level automation as the model size grows. In contrast, Figure 1(b) illustrates the asynchronous multi-process interaction paradigm we propose. Our key insight is that even models with high inference times can act at every step using sufficiently many staggered inference processes. Similarly, sufficiently many asynchronous learning processes can maintain rapid updates without blocking progress, despite high learning times. This work formalizes and empirically tests the benefits and limitations of this approach, making the following contributions: (a) Sequential Interaction and Learning (b) Asynchronous Multi-process Interaction and Learning Figure 1: Frameworks for Environment Interaction in RL . a) The typical sequential interaction paradigm where both learning and action inference block the environment from moving forward. b) The more realistic setting considered in this work where the environment, the agent’s inference process, and agent’s learning process all proceed at their own rate and interact asynchronously. Multiple self-loops are depicted to denote multiple asynchronous processes. \tau_{\mathcal{M}} denotes the frequency of the environment, \tau_{\theta} denotes the frequency of each inference process, and \tau_{\mathcal{L}} denotes the frequency of each learning process. Sequential interaction and learning has a frequency of \tau_{\mathcal{M}}+\tau_{\theta}+\tau_{\mathcal{L}} . 1. We formalize how the choice of a particular time discretization induces a new learning problem and how that problem relates to the original problem in Definition 1 . 2. We derive worst-case lower bounds on regret for solving the new problem rather than the original in Theorem 1 , leading us to conclude in Remark 1 that typical sequential interaction (Figure 1(a) ) scales poorly with model size. 3. We propose novel methods for staggering asynchronous inference in Algorithms 1 and 2 , addressing the poor scaling properties of sequential interaction (Remark 2 ). 4. We conduct comprehensive experiments to verify our theory, demonstrating the use of models that are orders of magnitude larger for realtime games like Pokémon and Tetris. 2 Regret Decomposition in Realtime Reinforcement Learning Background - Sequential Interaction: Most RL research focuses on agents interacting sequentially with a Markov Decision Process (MDP) [ 71 ; 94 ] \mathcal{M}_{\text{seq}}=\langle\mathcal{S},\mathcal{A},p,r\rangle , where \mathcal{S} is a set of states, \mathcal{A} is a set of actions, r(s,a) is a reward function with outputs bounded by r_{\text{max}} , and p(s^{\prime}|s,a) is a state transition probability function. Agents take actions based on a policy \pi_{\theta}(a|s) that maps states to action probabilities parameterized by \theta . An unrealistic implicit assumption of this setting is that the time between decisions is fixed and only depends on the MDP. It is also unrealistically assumed that the environment can be paused while the policy generates an action a from state s . Asynchronous Interaction Environments: The standard MDP formalism lacks a crucial element for realtime settings where the environment cannot be "paused," and the agent interacts with it asynchronously, as described by Travnik et al. [102] . In this case, it is necessary to define the environment’s behavior when the agent has not selected an action. We believe the most general solution is to use a preset default behavior if there is no available action a_{t} by the agent \pi at time-step t . This behavior follows a\sim\beta(s) , where a\in\mathcal{A}_{\beta} is possibly from a different action space than \mathcal{A} , requiring p and r to be defined over \mathcal{A}\cup\mathcal{A}_{\beta} . Now we can define an asynchronous MDP \mathcal{M}_{\text{async}}=\langle\mathcal{S},\mathcal{A},p,r,\beta\rangle as an extension of a sequential MDP \mathcal{M}_{\text{seq}} with the addition of the default behavior policy \beta . Note that \beta does not need to be non-Markovian, because the state space should be defined to include any intermediate computations needed to generate the actions of \beta . Defining the default behavior as a policy is no more than a useful interpretation of what happens and is equivalent to saying the environment follows a Markov chain p^{\beta}(s^{\prime}|s) when no action is available where p^{\beta}(s^{\prime}|s):=\sum_{a\in\mathcal{A}_{\beta}}p(s^{\prime}|s,a)\beta(a|s) with expected reward r^{\beta}(s)=\sum_{a\in\mathcal{A}_{\beta}}\beta(a|s)r(s,a) . Time Discretization Rates: The real environment evolves in continuous time, so we must define time discretization rates to describe each component of the agent-environment interface in discrete steps. We treat the environment step time as a random variable \mathrm{T}_{\mathcal{M}} with sampled values \tau_{\mathcal{M}}\sim\mathrm{T}_{\mathcal{M}} and expected value \bar{\tau}_{\mathcal{M}}:=\mathbb{E}[\mathrm{T}_{\mathcal{M}}] . Similarly, the inference time of the policy for a single action 1 1 1 See Appendix B.5 for a comparison with action chunking methods that produce multiple actions at a time. is another random variable \mathrm{T}_{\theta} with sampled values \tau_{\theta}\sim\mathrm{T}_{\theta} and expected value \bar{\tau}_{\theta}:=\mathbb{E}[\mathrm{T}_{\theta}] . 2 2 2 While policies in general could have adaptive computation times based on the state, this is relatively uncommon in the literature and will be left to future work for simplicity of the discourse. We can now introduce yet another random variable \mathrm{T}_{\mathcal{I}} with