Handling Delay in Real-Time Reinforcement Learning
Ivan Anokin, Rishav Rishav, Matthew Riemer, Stephen Chung, Irina Rish, Samira Ebrahimi Kahou · 2025
Read Handling Delay in Real-Time Reinforcement Learning by Ivan Anokin, Rishav Rishav, Matthew Riemer, Stephen Chung, Irina Rish, Samira Ebrahimi Kahou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23478.txt.
Introduction
Ivan Anokhin and co-authors tackle delay in real-time reinforcement learning, where a learning agent must act on a fixed clock, in this 2025 arXiv paper. While the network computes, the world keeps moving, so each action rests on stale sight. Deeper networks see more but lag more, since every layer adds its own delay.
The team's fix joins two ideas: connections that skip forward in time, and observations enriched with recent history. Agents built this way stayed strong across neuron speeds, learning methods, and test worlds, from simulated robot bodies to classic arcade-style games. Running neurons in parallel also sped up decisions by six to three hundred fifty percent on ordinary hardware. The authors frame the work as groundwork for real-time agents.
Catalog record
| CREATOR | Ivan Anokin, Rishav Rishav, Matthew Riemer, Stephen Chung, Irina Rish, Samira Ebrahimi Kahou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23478
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Handling Delay in Real-Time Reinforcement Learning Ivan Anokin, Rishav Rishav, Matthew Riemer, Stephen Chung, Irina Rish, Samira Ebrahimi Kahou Abstract Real-time reinforcement learning (RL) introduces several challenges. First, policies are constrained to a fixed number of actions per second due to hardware limitations. Second, the environment may change while the network is still computing an action, leading to observational delay . The first issue can partly be addressed with pipelining, leading to higher throughput and potentially better policies. However, the second issue remains: if each neuron operates in parallel with an execution time of τ \tau , an N N -layer feed-forward network experiences observation delay of τ N \tau N . Reducing the number of layers can decrease this delay, but at the cost of the network’s expressivity. In this work, we explore the trade-off between minimizing delay and network’s expressivity. We present a theoretically motivated solution that leverages temporal skip connections combined with history-augmented observations. We evaluate several architectures and show that those incorporating temporal skip connections achieve strong performance across various neuron execution times , reinforcement learning algorithms, and environments, including four Mujoco tasks and all MinAtar games. Moreover, we demonstrate parallel neuron computation can accelerate inference by 6-350% on standard hardware. Our investigation into temporal skip connections and parallel computations paves the way for more efficient RL agents in real-time setting. 1 Introduction (a) (b) Figure 1: (a) Parallel computations of layers speed-up inference time. Speed-up on GPU is achieved using default Pytorch software and widely accessible Nvidia GPU. (b) Normalized averaged performance and standard error of agents in parallel computation framework. Agents with skip connections and history- aug mented observations exhibit strong performance. Performance is averaged across the following environments: HalfCheetah-v4, Walker2d-v4, Ant-v4 and Hopper-v4 on Mujoco, all six environments on MinAtar, and Empty-Random-5x5-v0 and DoorKey-5x5-v0 on MiniGrid. Performance on Mujoco is also averaged across four different neuron execution times. Neural network inference presents several challenges in real-time reinforcement learning (RL) as the environment can change significantly, even during the networks’ inference process. One major challenge is that the inference time directly impacts throughput, the number of actions the agent can produce per second. High throughput is important in domains like robotics, algorithmic trading, and real-time gaming, where frequent decision-making can significantly improve policy performance. To address this challenge, a straightforward approach is to speed up inference by employing pipelining techniques. In a pipelined architecture, instead of waiting for the entire neural network to complete its forward pass on one input before processing the next, each layer begins processing the subsequent input as soon as it produces its output for the current one ( Carreira et al., 2018 ; Iuzzolino et al., 2021 ) . This approach increases the throughput of a neural network (see Fig. 1(a) ), as layers are effectively working in parallel but on different inputs. Throughout this paper, we refer to this approach as the parallel computation framework . However, even within the parallel computation framework, a traditional N -layer feed-forward neural network still suffers from another issue known as observational delay : the agent’s action at time step t is based on an observation from time step t-N\delta where \delta denotes execution time of each layer. This delay arises because, in a pipelined system, each layer is processing data from different time steps simultaneously – layer 1 processes input from time t , layer 2 processes the output of layer 1 from time t-\delta , and so on (as illustrated in the center graph of Fig. 2 ). This challenge leads us to the central question of this paper: If we use parallel computations of layers, how do we address observational delay? Reducing the number of layers can mitigate delay but limits the network’s expressivity. To overcome this, we propose using temporal skip connections . Traditionally, skip connections are used to stabilize training and allow gradient flow in deep networks ( Ronneberger et al., 2015 ; He et al., 2016 ) . However, within the parallel computation framework, skip connections offer another advantage: they do not only shortcut between layers along depth, but also along time, by sending activations forward in time (see the rightmost graph in Fig. 2 ). This temporal application of skip connection in the parallel computation setting reduces the observational delay. Nevertheless, the computational paths through these temporal skip connections are shorter than those without them and thus offer limited expressivity compared to longer paths through more neurons. Figure 2: Computation flow of agents. Left graph represents sequential computations and the central graph – parallel computations of layers. \delta is execution time of each neuron (or layer). All nodes at each column are available at the same time and can be processed further in parallel. The right architecture with skip connections exhibits less delay as it performs shortcuts along time-steps. We explore the trade-off between delay and network expressivity and investigate various types of architectures to find an optimal balance. Our theoretical analysis quantifies the impact of skip connections on reducing the regret associated with observational delay. Furthermore, we justify the importance of augmenting observations with past data in architectures with temporal skip connections. Experiments confirm the importance of skip connections and history-augmented observation (see Fig. 1(b) ), and our analysis shows that the skip connection offers a fast but less refined path for processing inputs, while the main connections provide a slower but more refined path. Our results show that in many environments this allows the policy in a parallel computation setting to achieve similar performance to an oracle agent with an instantaneous forward pass, provided the inference time of a layer is not large. While the parallel layer computation approach and temporal skip connections were proposed before to accelerate predictions on image ( Iuzzolino et al., 2021 ; Fischer et al., 2018 ) and video ( Carreira et al., 2018 ; Kugele et al., 2020 ) domains, this is the first application in RL – a domain where one bad action can critically impact the entire trajectory due to the agent’s influence on the environment. To summarize, we introduce a solution to real-time RL : speeding up inference time by parallel computations of layers and addressing associated observational delay. We demonstrate that parallel computations significantly improve throughput on modern hardware like GPUs. To address the observational delay, we provide a theoretically justified solution using temporal skip connections and history-augmented observations. Our experiments demonstrated its effectiveness across various cases, paving the way for more efficient RL agents in real-time setting. 2 Related Work Parallel computations of neurons (or layers). Parallel processing of information is consistent with popular mathematical models of the human cortex ( Tomita et al., 1999 ; Betti & Gori, 2019 ; Kubilius et al., 2018 ; Larkum, 2013 ) , where neurons operate asynchronously. Inspired by this, several attempts have been made to parallelize neural networks, aiming to maximize processing resource utilization and reduce latency. Carreira et al. (2018) introduced parallel video networks that employ parallel layer computations and temporal skip connections, significantly boosting throughput (or frame rate) during inference. Similarly, Iuzzolino et al. (2021) explored this approach for still images, enabling fast “anytime predictions” that improve over time. Additionally, Fischer