Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story
Vincenzo De Paola, Riccardo Zamboni, Mirco Mutti, Marcello Restelli · 2025
Read Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story by Vincenzo De Paola, Riccardo Zamboni, Mirco Mutti, Marcello Restelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01336.txt.
Introduction
Vincenzo De Paola and co-authors ask whether twin robot learners should be taught to differ. Parallel reinforcement learning runs many agents in copies of one world to gather data faster. Identical agents may waste the speedup by collecting the same experience. Their 2025 framework instead maximizes the entropy of the pooled data. A balance is struck: each agent stays varied, and agents differ from each other. A centralized policy gradient method steers the whole fleet at once.
Tests show the specialized fleet beats identical-agent systems. It also pairs well with batch methods that feed on diverse data. A new concentration analysis proves faster learning rates for varied sampling. Diversity, the math says, is not noise but fuel.
Catalog record
| CREATOR | Vincenzo De Paola, Riccardo Zamboni, Mirco Mutti, Marcello Restelli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01336
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Diversity In Parallel Agents: A Maximum State Entropy Exploration Story Vincenzo De Paola, Riccardo Zamboni, Mirco Mutti, Marcello Restelli Abstract Parallel data collection has redefined Reinforcement Learning (RL), unlocking unprecedented efficiency and powering breakthroughs in large-scale real-world applications. In this paradigm, N N identical agents operate in N N replicas of an environment simulator, accelerating data collection by a factor of N N . A critical question arises: Does specializing the policies of the parallel agents hold the key to surpass the N N factor acceleration? In this paper, we introduce a novel learning framework that maximizes the entropy of collected data in a parallel setting. Our approach carefully balances the entropy of individual agents with inter-agent diversity, effectively minimizing redundancies. The latter idea is implemented with a centralized policy gradient method, which shows promise when evaluated empirically against systems of identical agents, as well as synergy with batch RL techniques that can exploit data diversity. Finally, we provide an original concentration analysis that shows faster rates for specialized parallel sampling distributions, which supports our methodology and may be of independent interest. 1 Introduction Exploration , or taking sub-optimal decisions to gather information about the task, lies at the heart of Reinforcement Learning ( Bertsekas, 2019 , RL,) from its very foundation. When the task of interest is defined through a reward function, exploration shall be carefully balanced with the exploitation of the current information to maximize the rewards collected over time ( Mnih et al., 2013 ) . When a task is not known at the time of the interaction, exploration becomes an objective per se , in order to foster the collection of maximally informative data. A common formulation of the latter pure exploration setting is state entropy maximization ( Hazan et al., 2019 ) , in which the agent aims to maximize the entropy of the data distribution induced by its policy. The state entropy objective has been widely studied in the literature, e.g., to provide data collection strategy for offline RL ( Yarats et al., 2022 ; Park et al., 2023 ) , experimental design ( Tarbouriech & Lazaric, 2019 ) , or transition model estimation ( Tarbouriech et al., 2020 ) , and as a surrogate loss for policy pre-training in reward-free settings ( Mutti & Restelli, 2020 ) . All of the above uphold the status of state entropy maximization as a flexible tool for exploration. To this day, however, an important gap remains in how to deploy state entropy maximization on parallel simulators. Let us picture an illustrative example to guide our thoughts: A robotic arm control task. We may want the robot to be able to perform a wide range of tasks, such as interacting with a diverse set of objects. A good strategy is to exploit a simulator of the robotic arm to learn good manipulation policies in advance, instead of relying on costly online interactions, in which exploration may be unsafe. Since not all of the future tasks may be known in advance, one could opt for state entropy maximization, in order to collect exploration data once and for all. Collecting those data on complex simulators may be inefficient though, resulting in long simulation times. However, one crucial advantage of sampling in simulation rather than the physical system is that the sampling process does not have to be sequential. In many modern simulators ( NVIDIA Isaac Sim, ; Genesis, 2024 ; Todorov et al., 2012 ) , the process of learning is accelerated by instantiating vectorized instances of agents and environments simultaneously. Parallelization was shown to be a game-changing approach that allowed for unprecedented performance in standard RL ( Espeholt et al., 2018 ) , and the use of a multitude of identical agents operating on copies of the same environment allowed RL systems to tackle increasingly complex and large-scale problems. While the maximum entropy formulation has been widely adopted in prior works, how to better take advantage of parallel exploration agents is still to be investigated. This is far from being a trivial problem. Zhong et al. (2024) recently showed that not explicitly caring for redundant behaviors among the agents often results in inefficient use of computational resources, which is why parallelization is employed in the first place. Additionally, as highlighted in Rudin et al. (2022) , solving complex problems necessitates intelligent strategies for constructing experience that accelerate algorithmic convergence. This challenge is further compounded in scenarios where access to real-world process data is limited, examples are scarce, or simulators are computationally expensive. Following these considerations, our study is centered around the following questions: How should parallel agents optimize their policies for efficient exploration? What is the best way to leverage parallelism in state entropy maximization? In this paper, we generally address this question by extending state entropy maximization to incorporate the role of parallel agents, enabling a deeper investigation into their exploration efficiency. First, we introduce a specific problem formulation that can explicitly leverage parallelism. Rather than considering agents independently, we look at them as a whole, and consider the distribution a single one virtual agent would induce if it was following a uniform mixture of the agents’ policies. Additionally, we theoretically characterize how the entropy of the distribution of state visits concentrates in the single-agent and parallel-agent cases and demonstrate the significant impact of parallelization on the rate of entropy stabilization and exploration diversity. Then, we shift our attention to how to solve the problem of parallel state exploration in practice , by proposing a policy gradient strategy ( Williams, 1992 ) that explicitly optimizes for the objective at hand. Finally, we use illustrative experiments to show that the proposed algorithm is indeed able to optimize for the state entropy and that the induced behavior enables agents to balance individual exploration with inter-agent diversity. This synergy is particularly relevant when integrated with batch RL techniques ( Levine et al., 2020 ) . Original Contributions. Throughout the paper, we make the following contributions: • We define the problem of State Entropy Maximization in Parallel MDPs (Section 3 ); • We provide high-probability concentration bounds describing the rate at which the entropy of empirical distributions converges to the entropy of their stationary distributions. The proofs employ novel techniques that may be of independent interest (Section 4 ); • We introduce a policy gradient algorithm to take advantage of parallel agents for maximizing the entropy of the visited states (Section 5 ); • We provide numerical experiments to corroborate our findings, both in terms of state entropy maximization and batch RL on the collected data (Section 6 ). 2 Preliminaries Before diving into the technical contributions of our work, we introduce the notation and the relevant background. Notation. We denote [N]:=\{1,2,\ldots,N\} for a constant N<\infty . We denote a set with a calligraphic letter \mathcal{A} and its size as |\mathcal{A}| . We denote \mathcal{A}^{T}:=\times_{t=1}^{T}\mathcal{A} the T -fold Cartesian product of \mathcal{A} . The simplex on \mathcal{A} is denoted as \Delta_{\mathcal{A}}:=\{p\in[0,1]^{|\mathcal{A}|}|\sum_{a\in\mathcal{A}}p(a)=1\} and \Delta_{\mathcal{A}}^{\mathcal{B}} denotes the set of conditional distributions p:\mathcal{A}\to\Delta_{\mathcal{B}} . Let X,X^{\prime} random variables on the set of outcomes \mathcal{X} and corresponding probability measures p_{X},p_{X^{\prime}} , we denote the Shannon entropy of X as \mathcal{H}(X)=-\sum_{x\in\mathcal{X}}p_{X}(x)\log(p_{X}(x)) and the Kullback-Leibler (KL)