The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks

Walter Mayor, Johan Obando-Ceron, Aaron Courville, Pablo Samuel Castro · 2025

Read The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks by Walter Mayor, Johan Obando-Ceron, Aaron Courville, Pablo Samuel Castro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03404.txt.

Introduction

Walter Mayor and co-authors study how gathering data in parallel shapes deep reinforcement learning in a 2025 paper. Agents often learn from many copies of a task running at once, and choices about how many copies and how long each run lasts affect the data they learn from. More passes over the same data can also help or hurt.

The team ran a careful set of tests with a popular algorithm and varied these choices. Bigger data sets raised final scores across many settings. Adding more parallel copies worked better than lengthening each run. The authors tie the patterns to how well networks keep learning and to stability of training, and they argue data collection choices deserve as much care as model design.

Catalog record

CREATORWalter Mayor, Johan Obando-Ceron, Aaron Courville, Pablo Samuel Castro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03404

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks Walter Mayor, Johan Obando-Ceron, Aaron Courville, Pablo Samuel Castro Abstract The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance. † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning (RL) is a promising framework for addressing a wide range of decision-making tasks, from robotics and autonomous driving to game playing and resource optimization ( Schwarzer et al. 2023 ; Vinyals et al. 2019 ; Bellemare et al. 2020 ; Fawzi et al. 2022 ) . Central to the success of RL is the iterative process of data collection and policy learning, wherein agents interact with an environment to gather experience, which is subsequently used to improve decision-making policies ( Sutton & Barto 2018 ) . This interaction forms the foundation of RL’s ability to discover and refine strategies that can handle high-dimensional and dynamic decision spaces. However, many RL tasks, particularly those involving complex environments, suffer from a critical bottleneck: the need for extensive and diverse training data to effectively generalize across the state space ( Cobbe et al. 2020 ) . This bottleneck is further compounded in online RL, where data arrives sequentially over time in a non-stationary fashion ( Khetarpal et al. 2022 ) . Traditional RL methods often rely on single-environment simulations which limits the speed and diversity of data collection. This approach, while straightforward, is inherently slow and can restrict the scope of exploration, leading to suboptimal policy learning ( Rudin et al. 2022 ; Singla et al. 2024 ) . To address these limitations, recent advances in parallelized simulations, powered by GPU technology and distributed computing, have enabled simultaneous interaction with multiple instances of the environment ( Handa et al. 2023 ; Gallici et al. 2024 ; Singla et al. 2024 ) . This innovation allows RL agents to gather significantly more data within a given timeframe, which can result in more diverse training data and richer training signals ( Horgan et al. 2018 ; Espeholt et al. 2018 ; Petrenko et al. 2020 ) . Parallelized data collection has the potential to facilitate better state space coverage, reduce variance in gradient estimates, and accelerate the learning process by providing a larger and more diverse data for each policy update ( Schulman et al. 2017 ) . However, the practical realization of these benefits might be hindered by intrinsic challenges within deep RL algorithms. One such challenge is the loss of plasticity, which refers to a network’s ability to adapt to new information without degrading its performance on previously learned tasks ( Berariu et al. 2021 ; Lyle et al. 2023 ; Juliani & Ash 2024 ; Moalla et al. 2024 ) . Recent work has also demonstrated that scaling data can often saturate an agent’s performance ( Singla et al. 2024 ) . With this work we systematically investigate the interplay between parallelized data collection, plasticity, learned representations, and sample efficiency. Our findings reveal that parallelized data collection can help mitigate common optimization challenges in deep RL, ultimately resulting in a significant positive impact on final performance. We show that increasing the number of parallel environments ( N_{\textrm{envs}} ) leads to stabler training dynamics, as evidenced by reductions in weight norm and gradient kurtosis, and that this strategy is generally more effective than increasing rollout length ( N_{\textrm{RO}} ) for a fixed data budget (see Sec. 4 and Sec. 5 ). We conclude by providing guidance for maximizing data efficiency when increasing the data collected via parallelization. 2 Related Work Early efforts to scale RL focused on distributing computation across multiple actors and learners. A3C ( Mnih et al. 2016 ) introduced an asynchronous multi-threaded framework to parallelize experience collection and learning. IMPALA ( Espeholt et al. 2018 ) extended this idea using a decoupled actor-learner architecture with off-policy correction, enabling efficient scaling to thousands of environments. Other systems explored synchronous data collection using large batches ( Stooke & Abbeel 2018 ; Horgan et al. 2018 ) , improving hardware utilization but often increasing training instability. Recent advances in GPU-accelerated simulators such as Isaac Gym ( Makoviychuk et al. 2021 ) , EnvPool ( Weng et al. 2022 ) , and PGX ( Koyamada et al. 2024 ) have significantly increased the throughput of data collection, enabling thousands of environment instances to run in parallel on a single device. These tools have enabled researchers to scale simulation-based learning to previously inaccessible domains, such as dexterous manipulation ( Rudin et al. 2021 ) , legged locomotion ( Agarwal et al. 2022 ) , and complex coordination tasks ( Handa et al. 2023 ) . Several recent studies have revisited the role of parallel data collection in improving learning performance. Li et al. 2023 showed that increasing the number of environments during training can improve exploration and stabilize learning. Liu et al. 2024 demonstrated that combining parallel environment rollout with distributed learners leads to faster convergence and stronger final performance. Despite these benefits, scaling rollout length ( N_{\text{RO}} ) versus environment count ( N_{\text{envs}} ) remains underexplored. While high-throughput simulators provide vast amounts of experience, prior work has shown that simply increasing batch size or rollout length is often suboptimal. Singla et al. 2024 showed diminishing returns when scaling data volume without accounting for structural factors such as rollout diversity or update frequency. Similar results were observed in single-environment settings, where larger batches hindered generalization and slowed adaptation ( Obando Ceron et al. 2024 ) . These findings suggest that the structure of collected data, not just its volume, is critical for effective and stable training. Our work builds on these insights by systematically analyzing how different modes of data scaling (via N_{\text{envs}} and N_{\text{RO}} ) affect performance, sample efficiency, and network plasticity in PPO and PQN. This study isolates these variables under fixed data budgets and links their effects to optimization stability and final returns across discrete and continuous control tasks. 3 Background RL is a machine learning paradigm that enables agents to map observations to actions. When these actions are executed in an environment, the environment provides a numerical reward and transitions the agent to a new state. This is typically formalized as a Markov Decision Process (MDP) ( Puterman 1994 ) . M:=(S,A,R,P,\gamma), where S is a finite set of states, A is

Read the complete text