Co2PO: Coordinated Constrained Policy Optimization for Multi-Agent RL
Shrenik Patel, Christine Truong · 2026
Read Co2PO: Coordinated Constrained Policy Optimization for Multi-Agent RL by Shrenik Patel, Christine Truong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02970.txt.
Introduction
Keeping many learning agents safe while they explore is the aim of Shrenik Patel and 1 co-author in a 2026 paper. In constrained multi-agent reinforcement learning, safety rules limit what agents may do. Standard penalty methods react only after violations occur, which suppresses exploration and makes agents overly conservative in complex settings.
Their framework, Co2PO, adds risk-aware communication between agents. A shared blackboard broadcasts position intent and yield signals. A learned hazard predictor forecasts possible violations ahead of time. Those forecasts enter the optimization objective, so agents can anticipate collective dangers. Tests on multi-agent safety benchmarks gave higher returns than leading constrained baselines. Policies also complied with cost limits at deployment. Ablations confirmed each component mattered.
Catalog record
| CREATOR | Shrenik Patel, Christine Truong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02970
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Co2PO: Coordinated Constrained Policy Optimization for Multi-Agent RL Shrenik Patel, Christine Truong Abstract Constrained multi-agent reinforcement learning (MARL) faces a fundamental tension between exploration and safety-constrained optimization. Existing leading approaches, such as Lagrangian methods, typically rely on global penalties or centralized critics that react to violations after they occur, often suppressing exploration and leading to over-conservatism. We propose Co2PO, a novel MARL communication-augmented framework that enables coordination-driven safety through selective, risk-aware communication. Co2PO introduces a shared blackboard architecture for broadcasting positional intent and yield signals, governed by a learned hazard predictor that proactively forecasts potential violations over an extended temporal horizon. By integrating these forecasts into a constrained optimization objective, Co2PO allows agents to anticipate and navigate collective hazards without the performance trade-offs inherent in traditional reactive constraints. We evaluate Co2PO across a suite of complex multi-agent safety benchmarks, where it achieves higher returns compared to leading constrained baselines while converging to cost-compliant policies at deployment. Ablation studies further validate the necessity of risk-triggered communication, adaptive gating, and shared memory components. † † affiliationnotice: Equal contribution 1 Introduction Constrained multi-agent reinforcement learning (MARL) aims to maximize collective return while satisfying safety constraints, but in cooperative settings this tension is especially sharp: aggressive exploration can incur costly violations, while overly conservative updates can prematurely suppress learning and prevent agents from discovering high-performing joint behavior ( 4 ) . The problem is amplified by partial observability and interaction-driven dynamics. Even when agents share the same reward objective, safety failures often emerges from coordination breakdowns in timing, intent, or mutual awareness, rather than from conflicting incentives ( 17 ; 5 ) . As a result, purely reactive constraint handling can be slow to correct the underlying cause of violations, while heavy-handed penalties can make learning brittle ( 4 ) . We introduce Co2PO, a coordinated constrained policy optimization framework that targets these coordination-driven safety failures through selective, risk-aware coordination . Rather than treating communication as a persistent channel, Co2PO uses it as a sparse control mechanism: agents write to a shared blackboard only when they predict elevated near-term hazard risk. Each write contains compact safety-relevant signals—a positional state summary, an intent vector, and a yield flag—that other agents can retrieve to condition their actions. This design preserves decentralized action selection while providing coordination signals precisely when interaction risk is most acute. Across cooperative multi-agent safety benchmarks, Co2PO improves feasible performance and reaches cost-compliant policies at evaluation under the same training budget. We observe transient violations early in training, when hazard prediction and coordination behavior are still being learned, but final policies satisfy the cost constraint during evaluation. Our contributions are as follows: Contributions. • We introduce Co2PO, enabling selective coordination via a shared blackboard with decentralized actions. • We propose hazard-triggered writes of compact signals (state, intent, yield) with adaptive write-rate control. • We evaluate on cooperative multi-agent safety benchmarks with ablations isolating blackboard and gating. 2 Problem Setup and Preliminaries We study constrained cooperative multi-agent reinforcement learning (MARL) under partial observability. Agents seek high return while satisfying an expected cost budget at evaluation, and we summarize performance with feasible return . 2.1 Constrained Cooperative Markov Game We model the environment as a cooperative constrained Markov game (CMG) with n agents ( 15 ) : \mathcal{G}=\langle\mathcal{N},\mathcal{S},\{\mathcal{O}_{i}\}_{i\in\mathcal{N}},\{\mathcal{A}_{i}\}_{i\in\mathcal{N}},P,r,c,\gamma,\rho_{0},d\rangle. Agent i observes o_{t}^{i}\in\mathcal{O}_{i} and selects a_{t}^{i}\in\mathcal{A}_{i} , yielding joint action a_{t}=(a_{t}^{1},\dots,a_{t}^{n}) and transition s_{t+1}\sim P(\cdot\mid s_{t},a_{t}) from s_{0}\sim\rho_{0} . The environment emits a shared reward r_{t}=r(s_{t},a_{t}) and safety cost c_{t}=c(s_{t},a_{t}) . Policies factorize as \pi(a\mid o)=\prod_{i=1}^{n}\pi_{i}(a^{i}\mid o^{i}) with o=(o^{1},\dots,o^{n}) . The discounted return and cost are \displaystyle J_{R}(\pi) \displaystyle=\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t}\,r(s_{t},a_{t})\right], (1) \displaystyle J_{C}(\pi) \displaystyle=\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t}\,\bar{c}(s_{t},a_{t})\right], (2) where \bar{c} is the team cost used for enforcement. We seek \max_{\pi}\;J_{R}(\pi)\quad\text{s.t.}\quad J_{C}(\pi)\leq d, (3) with cost budget d . 2.2 CTDE Preliminaries We use centralized training with decentralized execution (CTDE): critics may condition on centralized information during training, while each actor \pi_{i} conditions only on its local observation (and any permitted message/memory) at execution ( 16 ; 28 ) . 2.3 Safety Metric: Feasible Return We report feasible return to reflect deployment-time cost constraints: J_{F}(\pi)\;=\;J_{R}(\pi)\cdot\mathbf{1}\!\left[J_{C}(\pi)\leq d\right]. (4) 2.4 Lagrangian Relaxation We optimize a Lagrangian relaxation of Equation 3 , following standard constrained policy optimization formulations ( 1 ) : \max_{\pi}\;\min_{\lambda\geq 0}\;\;\mathcal{L}(\pi,\lambda)\;=\;J_{R}(\pi)\;-\;\lambda\big(J_{C}(\pi)-d\big), (5) with the dual variable \lambda updated online. Actor–critic updates commonly use a hybrid advantage ( 25 ; 20 ; 30 ) A^{\text{hyb}}_{t}\;=\;A^{R}_{t}\;-\;\lambda\,A^{C}_{t}. (6) 2.5 Risk Events and Hazard Labels We construct a proactive hazard label from dense per-step costs. Let c_{t}^{i} be the instantaneous cost for agent i . Define a hazard event z_{t}^{i}\;=\;\mathbf{1}\!\left[c_{t}^{i}>\delta\right], (7) and a lookahead label with horizon H , h_{t}^{i}\;=\;\mathbf{1}\!\left[\max_{\tau\in\{t,\dots,t+H\}}z_{\tau}^{i}=1\right], (8) computed within episode boundaries. We use h_{t}^{i} to supervise the hazard predictor used for selective coordination in Section 3 . 3 Methodology We present Co2PO, a constrained MARL method that augments decentralized policies with selective shared-memory coordination. Each agent predicts near-term hazard risk from its local observation and writes a compact message to a shared blackboard only when predicted risk exceeds a threshold. Agents then read a fixed-size context from the blackboard and condition their actions on the retrieved information. Training uses a constrained actor–critic objective with reward and cost critics, a dual variable for constraint enforcement, supervised hazard forecasting, and a penalty that discourages excessive writing. 3.1 Per-Timestep Interaction Protocol We consider E parallel environment instances and n agents. At timestep t in environment e : (1) write-info each agent computes hazard probability and message contents from its observation; (2) write agents gate writes and update the blackboard; (3) read each agent retrieves up to k other agents’ messages to form a fixed-length context; (4) act each agent selects an action conditioned on its observation and retrieved context. The blackboard is maintained per environment instance and cleared on episode termination. 3.2 Blackboard: State, Write, and Read Stored fields. For each environment e and agent i , the blackboard stores a single current entry B_{t}^{e,i}\;=\;\big(x_{t}^{e,i},\,u_{t}^{e,i},\,y_{t}^{e,i},\,p_{t}^{e,i},\,w_{t}^{e,i}\big), (9) where x_{t}^{e,i}\in\mathbb{R}^{d} is a compact state summary, u_{t}^{e,i}\in\mathbb{R}^{d} is a learned intent vector, y_{t}^{e,i}\in[0,1] is a learned yield flag, p_{t}^{e,i}\in[0,1] is the hazard probability, and w_{t}^{e,i}\in\{0,1\} is the write indicator. Write information from local observation. Each agent computes message fields and a hazard score from its observation: (\ell_{t}^{i},\;u_{t}^{i},\;y_{t}^{i})\;=\;F_{\theta}(o_{t}^{i}),\qquad p_{t}^{i}\;=\;\sigma(\ell_{t}^{i}), (10) where \ell_{t}^{i} is a hazard logit and \sigma(\cdot) is the sigmoid. Event-triggered writes. Writes are gated by a threshold: w_{t}^{e,i}\;=\;\mathbf{1}\!\left[p_{t}^{e,i}>\tau_{t}\right], (11) and only entries with w_{t}^{e,i}=1 are valid for reads. Similarity-based read (top- k ) with fixed-length context. At each timestep, agent i forms a query vector from its own state summary, q_{t}^{e,i}=x_{t}^{e,i} . It then considers the set of active entries written by other agents in the same environment instance, \{\,B_{t}^{e,j}\,:\,j\neq i,\;w_{t}^{e,j}=1\,\} , and scores each candidate j by cosine similarity ( 6 ) : s_{t}^{e}(i,j)=\left\langle\frac{x_{t}^{e,j}}{\lVert x_{t}^{e,j}\rVert_{2}+\varepsilon},\;\frac{q_{t}^{e,i}}{\lVert q_{t}^{e,i}\rVert_{2}+\varepsilon}\right\rangle, where \varepsilon>0 is a small constant for numerical stability. Agent i retrieves the k highest-scoring active entries and, for each retrieved sender j , constructs \psi(B_{t}^{e,j})=\big[x_{t}^{e,j};\,u_{t}^{e,j};\,y_{t}^{e,j};\,p_{t}^{e,j}\big]\in\mathbb{R}^{2d+2}. The memory context is the concatenation of up to k such vectors in ranked order, zero-padded if fewer than k entries are active: m_{t}^{e,i}=\big[\psi(B_{t}^{e,j_{1}});\dots;\psi(B_{t}^{e,j_{k}})\big]\in\mathbb{R}^{k(2d+2)}. (12) 3.3 Threshold Control (Optional) We allow adaptation of \tau_{t} online to target a desired write rate \rho^{\star} . Let \rho_{t} be the observed mean write rate and \bar{\rho}_{t}