Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
Nicholas E. Corrado, Josiah P. Hanna · 2025
Read Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies by Nicholas E. Corrado, Josiah P. Hanna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01049.txt.
Introduction
Nicholas Corrado and Josiah Hanna tackled a quiet flaw in 2025 in how teams of learning agents train. Each agent learns from its own stream of play, and the joint stream can drift from what the team plan says it should be. That drift warps the slope each agent climbs. The result can be a poor joint plan even when each agent's own slope points the right way on average.
Their fix, called CoSER, steers play from the center toward joint moves that the stream has under-sampled. Across a range of team games, it cut joint sampling error faster than plain play. Agents trained on that stream reached the best joint plan more often. The authors pitch it as a trust boost for team learning.
Catalog record
| CREATOR | Nicholas E. Corrado, Josiah P. Hanna |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01049
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies Nicholas E. Corrado, Josiah P. Hanna Abstract Independent on-policy policy gradient algorithms are widely used for multi-agent reinforcement learning (MARL) in cooperative and no-conflict games, but they are known to converge suboptimally when each agent’s policy gradient points toward a suboptimal equilibrium [ 1 , 2 , 3 , 4 ] . In this work, we identify a subtler failure mode that arises even when the expected policy gradients of all agents point toward an optimal solution. After collecting a finite set of trajectories, stochasticity in independent action sampling can cause the joint data distribution to deviate from the expected joint on-policy distribution. This sampling error w.r.t. the joint on-policy distribution produces inaccurate gradient estimates that can lead agents to converge suboptimally. In this paper, we investigate if joint sampling error can be reduced through coordinated action selection and whether doing so improves the reliability of policy gradient learning in MARL. Toward this end, we introduce an adaptive action sampling approach to reduce joint sampling error. Our method, M ulti- A gent P roximal R obust O n- P olicy S ampling (MA-PROPS), uses a centralized behavior policy that we continually adapt to place larger probability on joint actions that are currently under-sampled w.r.t. the current joint policy. We empirically evaluate MA-PROPS in a diverse range of multi-agent games and demonstrate that (1) MA-PROPS reduces joint sampling error more efficiently than standard on-policy sampling and (2) improves the reliability of independent policy gradient algorithms, increasing the fraction of training runs that converge to an optimal joint policy. 1 Introduction On-policy policy gradient methods are among the most popular algorithms used for multi-agent reinforcement learning (RL) in cooperative and no-conflict games [ 5 , 6 , 7 ] . 1 1 1 In no-conflict games, all agents share the same set of preferred equilibria. Cooperative games are a subset of no-conflict games in which all agents share the same reward function. A common way to apply these algorithms is to treat agents as independent learners: each agent samples trajectories independently from its own policy (without observing other agents’ actions), estimates a Monte Carlo approximation of its policy gradient [ 8 ] , and performs a local policy update to maximize its expected return via gradient ascent. While independent policy gradient algorithms have demonstrated strong empirical performance across a range of multi-agent benchmarks [ 3 , 9 ] and powered several high-profile success stories [ 5 , 6 , 7 ] , it is well-understood that they may not converge to the most preferred equilibrium even in no-conflict games [ 1 , 2 , 3 , 4 ] . In this paper, we identify a subtler, lesser-known failure mode: even when the expected policy gradients align with optimal behavior , stochasticity in action sampling can nevertheless cause agents to converge to suboptimal solutions. Agent 2 A B Agent 1 A 12,12 0,6 B 6,0 2,2 Figure 1 : 2\times 2 matrix game where r_{1},r_{2} denotes rewards for Agent 1 ( r_{1} ) and 2 ( r_{2} ). To illustrate this issue, consider the 2\times 2 matrix game in Fig. 1 . Suppose each agent assigns equal probability to actions A and B and selects actions independently. For both agents, the expected reward of A is (12+0)/2=6 , and the expected reward of B is (6+2)/2=4 . Thus, both agents are incentivized to increase the probability of A , steering them toward the optimal joint action (A,A) . Now suppose the agents play the game four times. In expectation, each agent will sample both actions twice and observe each joint action once. However, due to randomness in action selection, Agent 1 may sample A,B,A,B while Agent 2 samples B,A,B,A so that (A,A) and (B,B) are under-sampled w.r.t. the expected joint on-policy distribution. Consequently, both agents observe reward 0 for action A and reward 6 for action B and thus increase the probability of B , driving agents toward the suboptimal outcome (B,B) . The core issue is joint sampling error : randomness in action sampling causes the empirical joint data distribution to deviate from the expected joint on-policy distribution, leading to inaccurate policy gradient estimates. Moreover, joint sampling error exists even though both agents sample A and B twice and thus have zero sampling error w.r.t. their own policies. Under on-policy sampling, the only way to reduce sampling error is to collect more data. Recently, Corrado and Hanna 2023 introduced an action sampling algorithm (PROPS) that reduces sampling error more efficiently than standard on-policy sampling. However this work focused on reducing sampling error w.r.t. a single policy in single-agent RL settings. As the above example shows, reducing sampling error w.r.t. each agent’s policy individually does not necessarily reduce sampling error w.r.t. the joint policy in multi-agent settings. Thus, agents must coordinate their action selection to reduce joint sampling error. These observations motivate the central question of this work: Can adaptive action sampling reduce joint sampling error and enable independent on-policy policy gradient algorithms to more reliably converge to an optimal joint policy? To answer this question, we introduce M ulti- A gent P roximal R obust O n- P olicy S ampling (MA-PROPS), an action sampling algorithm that adaptively corrects joint sampling error during multi-agent on-policy data collection. Fig. 2 provides an overview of MA-PROPS. Rather than sampling actions from each agent independently, MA-PROPS samples actions from a separate data collection policy that we continually update to increase the probability of under-sampled joint actions. We evaluate MA-PROPS on a diverse set of multi-agent games and first show that it reduces joint sampling error more efficiently than standard on-policy sampling. Next, we answer our central question affirmatively and show that reducing joint sampling error during independent on-policy policy gradient learning increases the fraction of training runs that converge to an optimal joint policy. While centralized action sampling may be challenging in tasks with many agents due to the exponential growth of the joint action space, we view MA-PROPS as a first step toward understanding and mitigating joint sampling error in multi-agent learning. In summary, our contributions are: 1. We characterize an under-explored failure mode of independent on-policy policy gradient algorithms: even when the expected gradient points toward optimal behavior, joint sampling error can cause convergence to suboptimal solutions. 2. To mitigate this failure mode, we introduce an adaptive sampling method (MA-PROPS) that aims to reduce joint sampling error. 3. We empirically demonstrate that MA-PROPS reduces joint sampling error more efficiently than on-policy sampling and consequently makes independent on-policy policy gradient MARL converge to an optimal joint policy more reliably. Figure 2 : MA-PROPS overview. Rather than collecting data {\mathcal{D}} by sampling actions from each agent’s current policy \pi_{{\bm{\theta}}_{i}} , we collect data with a centralized policy \pi_{\bm{\phi}} that we continually adapt to reduce joint sampling error in {\mathcal{D}} with respect to the joint policy induced by each agent’s policy. 2 Related Work Reducing sampling error via adaptive data collection. Prior work in single-agent RL has shown that adaptive action sampling can reduce sampling error more efficiently than standard on-policy sampling. Zhong et al. 2022 first demonstrated this idea theoretically, and Corrado and Hanna 2023