ACPO: Agent-Chained Policy Optimization for

Multi-Agent Reinforcement Learning · 2026

Read ACPO: Agent-Chained Policy Optimization for by Multi-Agent Reinforcement Learning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30072.txt.

Introduction

This author builds ACPO: agent-chained policy optimization for multi-agent reinforcement learning. The building builds. Their optimizing optimizes agent-chained policy. The learning is multi-agent reinforcement. It is RL. ACPO acpos. The ACPO is RL multi-agent. The policy optimization is agent-chained.

Catalog record

CREATORMulti-Agent Reinforcement Learning
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30072

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning Abstract Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows. † † footnotetext: \dagger Equal advising. 1 Introduction Cooperative multi-agent reinforcement learning (MARL) seeks decentralized policies that jointly maximize a shared return, in domains ranging from autonomous vehicle fleets ( 45 ) and traffic signal control ( 9 ) to fleet management ( 20 ) , power networks ( 39 ) , and Large Language Models ( 42 ; 22 ) . Underlying every cooperative MARL problem is a single-agent Markov Decision Process over the joint action space (Multi-Agent MDP) ( 6 ) , whose optimal policy maximizes the shared return ( 6 ) . Single-agent policy gradient methods ( 19 ; 33 ; 35 ) extend to this MDP in principle, with the joint policy playing the role of a single centralized agent’s policy. In real-world deployments, however, this joint policy must factorize across agents and each agent must act on its own information without inter-agent communication. This is the setting of Centralized Training with Decentralized Execution (CTDE) ( 23 ; 11 ) . Computing the Multi-Agent MDP (MMDP) policy gradient under decentralized execution has proven to be a challenge. Existing CTDE methods either do not handle the non-stationarity incurred by independent policy updates or solve a different problem. Independent policy optimization (MAPPO ( 44 ) , MADDPG ( 23 ) ) trains each decentralized actor against a centralized critic conditioned on the joint action. As actors are trained independently without accounting for the non-stationarity incurred by the other agents’ changing policies, joint improvement guarantees do not hold in general (Appendix A ). Alternating policy optimization (HATRPO/HAPPO ( 18 ; 48 ) ) sidesteps the MMDP by updating each agent against a per-agent reduced MDP with the others’ policies fixed. This monotonically improves the joint return at each step, but the underlying problem is now a sequence of unilateral best-response problems, and the fixed point is a Nash Equilibrium that can be arbitrarily far from the optimal joint policy 1 1 1 In game theory terminology, the optimal joint policy in cooperative MARL is the social optimum that maximizes welfare. (Table 1 ). In this paper, we show that the policy gradient in the MMDP can be computed directly under the decentralized execution constraint, via an exact decomposition into decentralized per-agent terms. The decomposition is enabled by augmenting each agent’s state with a belief over the actions committed by preceding agents in a serialized view of the joint decision. Our key result is the Multi-Agent Policy Gradient Decomposition Theorem (Theorem 4.2 ): the joint policy gradient decomposes into per-agent terms, each involving only a per-agent score function and a per-agent critic, with the updates coordinated through beliefs. Unlike value decomposition methods ( 37 ; 32 ; 40 ; 46 ) , this decomposition is made without any structural assumption on the joint value function. The per-agent critics are determined by a chained Bellman recursion in which each agent bootstraps from the next agent in the chain (Figure 1 ). We instantiate the decomposition as Agent-Chained Policy Optimization (ACPO), a general actor-critic algorithm with on-policy (PPO-based ( 35 ) ) and off-policy (TD3-based ( 13 ) ) variants. Each agent updates its actor against its own decentralized critic, and the independent per-agent updates collectively conduct a single gradient step on the MMDP joint policy. On Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, ACPO outperforms strong baselines, with the gap widening substantially as the number of agents grows. 2 Background 2.1 Multi-Agent MDP We consider a cooperative multi-agent reinforcement learning (MARL) setting with \mathcal{{N}}=\left\{1,\ldots,N\right\} agents, formally modeled as a Multi-Agent Markov Decision Process (MMDP) ( 6 ) . At time step t , each agent i\in\mathcal{N} simultaneously takes action a_{t}^{(i)}\in\mathcal{A}^{(i)} sampled from individual policy \pi^{(i)}(a_{t}^{(i)}\mid s_{t}) where s_{t}\in\mathcal{S} is the state. The state transition is Markovian, i.e. the next state s_{t+1} is given by transition function T(s_{t+1}|s_{t},\vec{a}_{t}) where \vec{a}_{t} is the joint action \vec{a}_{t}=[a_{t}^{(1)},\ldots,a_{t}^{(N)}] . Each agent receives shared reward r_{t} generated by the common reward function R(s_{t},\vec{a}_{t}) . The goal of cooperative MARL is to find a set of agent policies \vec{\pi}=[\pi^{(1)},\ldots,\pi^{(N)}] that maximize the total expected return J(\vec{\pi})=\mathbb{E}_{\vec{\pi}}\left[\sum_{t}\gamma^{t}r_{t}\right] where \gamma is the discount factor. Following standard practice in cooperative MARL, we adopt the Centralized Training with Decentralized Execution (CTDE) setting where policies are trained with shared information, but executed independently 2 2 2 See Appendix B for further discussion of CTDE. . 2.2 Previous CTDE Approaches in Cooperative MARL Independent Policy Optimization One way to solve MMDPs is to apply single-agent RL on the joint action space with independently factorized policies ( 23 ; 44 ) . Each policy is trained via an objective of the form J(\pi^{(i)})=\mathbb{E}_{\pi^{(i)}}\mathbb{E}_{\vec{\pi}^{-i}}[Q(s,a^{(i)},\vec{a}^{-i})] , where Q is the centralized action-value function, and \vec{a}^{-i},\vec{\pi}^{-i} are the joint actions and joint policies excluding agent i . The N agents each maximize J(\pi^{(i)}) independently. MADDPG ( 23 ) instantiates this with deterministic policy gradients via a learned Q ( 19 ) , and MAPPO ( 44 ) uses a centralized state-value function with GAE ( 34 ; 35 ) to compute advantages A(s,a^{(i)},\vec{a}^{-i}) . Both train policies in parallel and scale well in practice. However, there is no general guarantee of joint policy improvement even in tabular settings. As we show in Appendix A , simple counter-examples exist where these methods diverge. Convergence holds only in restricted settings where the optimal action-value decomposes such that independent updates align with joint improvement. Alternating Policy Optimization A second approach defines a reduced MDP for each agent and learns the best response policy, yielding monotonic improvement of the joint policy and convergence to a Nash Equilibrium (NE) ( 5 ) . Each agent i alternates and solves \langle\mathcal{S},\mathcal{A}^{(i)},T^{(i)},R^{(i)}\rangle , where T^{(i)}(s_{t+1}\mid s_{t},a^{(i)}_{t}):=\mathbb{E}_{\vec{\pi}^{-i}}[T(s_{t+1}\mid s_{t},a^{(i)}_{t},\vec{a}^{-i})] and R^{(i)}(s_{t},a^{(i)}_{t}):=\mathbb{E}_{\vec{\pi}^{-i}}[R(s_{t},a^{(i)}_{t},\vec{a}^{-i})] marginalize over the other agents’ actions under their most recent policies \vec{\pi}^{-i} . When \pi^{(i)} is being updated, the previous agent policies \vec{\pi}^{<i} have already been updated. HATRPO ( 18 ) and HAPPO ( 48 ) instantiate alternating best-response updates with trust-region ( 33 ) or clipped-surrogate ( 35 ) optimization. The advantage A^{(i)}(s,a^{(i)}) is well-defined in the

Read the complete text