A Recipe for Stable Offline Multi-agent Reinforcement Learning
Dongsu Lee, Daehee Lee, Amy Zhang · 2026
Read A Recipe for Stable Offline Multi-agent Reinforcement Learning by Dongsu Lee, Daehee Lee, Amy Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08399.txt.
Introduction
Lee and colleagues offer a recipe for stable offline multi-agent reinforcement learning. Despite single-agent offline reinforcement learning achievements, multi-agent learning has struggled to adopt the paradigm, persisting with on-policy training and self-play. One reason is instability in non-linear value decomposition, which led prior works to avoid complex mixing networks for linear decomposition with value regularization from single-agent setups. The authors analyze instability in non-linear value decomposition, proposing scale-invariant normalization. It stabilizes actor-critic training without altering the Bellman fixed point.
Catalog record
| CREATOR | Dongsu Lee, Daehee Lee, Amy Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51259 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08399
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Recipe for Stable Offline Multi-agent Reinforcement Learning Dongsu Lee, Daehee Lee, Amy Zhang Abstract Despite remarkable achievements in single-agent offline reinforcement learning (RL), multi-agent RL (MARL) has struggled to adopt this paradigm, largely persisting with on-policy training and self-play from scratch. One reason for this gap comes from the instability of non-linear value decomposition, leading prior works to avoid complex mixing networks in favor of linear value decomposition ( e.g. , VDN) with value regularization used in single-agent setups. In this work, we analyze the source of instability in non-linear value decomposition within the offline MARL setting. Our observations confirm that they induce value-scale amplification and unstable optimization. To alleviate this, we propose a simple technique, scale-invariant value normalization ( SVN ), that stabilizes actor-critic training without altering the Bellman fixed point. Empirically, we examine the interaction among key components of offline MARL ( e.g. , value decomposition, value learning, and policy extraction) and derive a practical recipe that unlocks its full potential. † † affiliation: University of Texas at Austin † † affiliation: Sungkyunkwan University † † contribution: Supervision † † date: August 10, 2026 † † Correspondence: Dongsu Lee at [email protected] † † Blog: https://dongsuleetech.github.io/blog/lazy-borrowed-recipe/ 1 Introduction While offline RL has achieved notable success, its extension to multi-agent settings remains relatively underexplored. More importantly, insights that hold in single-agent settings often fail to transfer to MARL. For example, while DDPG+BC (BRAC) ( 15 ) can be even preferable to advantage-weighted regression (AWR) ( 49 ) as a single-agent policy extraction ( 47 ) , we observe that in MARL, even minor deviations can precipitate severe performance degradation ( Figure 1 ). This instability highlights a key challenge to multi-agent systems: even a minor deviation in individual agent actions can cascade into a complete breakdown of the coordination. Figure 1 : Revisiting offline RL insights in MARL . ( Left ) The convex hull denotes the dataset action support, and dots represent actions sampled from the learned policy. BRAC exhibits mode-seeking behavior that extends beyond the dataset support, while AWR remains mode-covering and strictly in-distribution. ( Right ) Although such mode-seeking would be helpful in single-agent RL, even small out-of-distribution actions induced by BRAC lead to severe performance degradation in MARL, highlighting the sensitivity of joint behavior to individual policy deviations. These results are based on TD learning and hold it regardless of the value decomposition methods ( cen tralized, vdn , and dec entralized). Despite these structural challenges, most existing studies have merely extended single-agent value regularization techniques to multi-agent settings ( 71 ; 57 ) with linear value decomposition ( 34 ; 31 ) or centralization ( 71 ) . Although effective in some cases, centralized critics raise scalability concerns, and linear value decomposition can struggle to capture complex coordination structures. That is, achieving limited short-term success but offering little insight into the deeper challenges of offline MARL. This naturally raises a central question motivating this research: Where does the bottleneck in offline MARL come from, and how should we design algorithms to explicitly address it? In this study, we move beyond simple value regularization with linear value decomposition. Specifically, we first focus on non-linear value decomposition ( 53 ) in offline MARL. Then, we empirically investigate the interplay between value decomposition, policy extraction, and value learning to unlock the potential of offline MARL. Our key discoveries and contributions are as follows: Analyses In our pathological observations, the bottleneck of the non-linear method manifests itself as a coupled instability between value learning and policy extraction. Unlike linear decomposition, mixing networks structurally couple per-agent approximation errors through their Jacobian. This coupling breaks the contractivity of the global TD operator and turns value updates expansive rather than contractive. As a result, joint Q-values can grow exponentially even on expert datasets. This value-scale amplification further propagates to policy extraction, where actor gradients become dominated by the absolute magnitude of the value function rather than relative advantages. This dominance leads to poorly calibrated loss and unstable updates. Comprehensively, these effects form a feedback loop that destabilizes learning in non-linear value decomposition. Solutions To alleviate this issue, we propose a simple yet effective normalization technique that stabilizes nonlinear value decomposition without altering the Bellman fixed point. Our method renders both critic and actor updates scale-invariant, thereby directly addressing the amplification mechanism induced by the non-linear value decomposition while preserving theoretical correctness. This normalization restores stable and well-scaled optimization dynamics for actor-critic training with nonlinear value decomposition, enabling the non-linear value decomposition to be used reliably in the offline setting for the first time. Experiments Our empirical results further offer clear guidance for designing offline MARL algorithms. We find that performance is far more sensitive to value decomposition and policy extraction than value learning methods. In particular, non-linear value decomposition and mode-covering policy extraction yield stable and strong performance. Moreover, we demonstrate that nonlinear value decomposition equipped with our solution is practical across both continuous and discrete control, and remains stable when transitioning from offline to online. These findings highlight value decomposition as both a fundamental bottleneck and a promising lever to advance offline MARL. 2 Related work Over the past few years, data-driven approaches have fundamentally reshaped control systems, enabling learning-based methods to tackle real-world problems across several domains, including autonomous driving ( 39 ; 26 ; 27 ; 28 ) and robotics ( 3 ; 6 ) . A major driver of this progress has been the development of off-policy RL algorithms, which train value functions via temporal difference (TD) learning while leveraging static operational logs ( 33 ) . Compared to single-agent RL, offline MARL faces a more severe out-of-distribution challenge. A small deviation in an individual policy can induce joint behaviors that are absent from the dataset. This can lead to unseen coordination patterns even when each agent’s action is individually plausible. To prevent such a failure mode, prior works have largely followed the success of single-agent RL in terms of value regularization and policy extraction, such as conservatism ( 46 ; 57 ; 8 ) , in-sample maximization ( 71 ) , action support constraint ( 76 ; 23 ) , distribution matching ( 78 ; 34 ; 36 ) , convex duality ( 43 ) , and density weighting ( 31 ) . In practice, these approaches rely on behavior-regularized policy gradients( e.g., BRAC ( 15 ; 65 ) ) or AWR ( 49 ) for policy extraction. However, the role of value decomposition remains underexplored. Value decomposition in prior work is limited to simple linear forms ( e.g. , VDN ( 63 ) ) or a fully centralized critic. Although these choices improve stability, they also limit expressivity or introduce scalability challenges as the number of agents grows. More expressive non-linear mixing architectures are often avoided due to long-observed instability ( 58 ; 38 ) . Rather than proposing another algorithmic variant, this work aims to diagnose why non-linear value decomposition becomes unstable. We trace this instability to the structural coupling between per-agent value learning and policy extraction that amplifies joint out-of-distribution errors. Based on this analysis, we introduce a simple normalization method that preserves the Bellman fixed point and enables practical non-linear value decomposition.