Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning
Austin A. Nguyen, Michael P. Wellman · 2026
Read Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning by Austin A. Nguyen, Michael P. Wellman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00374.txt.
Introduction
Nguyen and Wellman tackle game solving from fixed data alone, where only part of the game dynamics may show in the record and a claimed equilibrium cannot even be checked. They rank candidate equilibria by their chance of low regret under the available evidence, extending Policy Space Response Oracles with uncertainty measures and a regret leaning objective. A new meta strategy solver steers search for the offline case. Tests show their COffeE PSRO method finds lower regret answers than leading offline rivals and clarifies how method parts shape game fidelity.
Catalog record
| CREATOR | Austin A. Nguyen, Michael P. Wellman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53123 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning Austin A. Nguyen, Michael P. Wellman Abstract . Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories. We consider the problem in a mixed-motive multiagent setting, where the goal is to solve a game under the offline learning constraint. We first frame this problem in terms of selecting among candidate equilibria. Since datasets may inform only a small fraction of game dynamics, it is generally infeasible in offline game-solving to even verify a proposed solution is a true equilibrium. Therefore, we consider the relative probability of low regret (i.e., closeness to equilibrium) across candidates based on the information available. Specifically, we extend Policy Space Response Oracles (PSRO), an online game-solving approach, by quantifying game dynamics uncertainty and modifying the RL objective to skew towards solutions more likely to have low regret in the true game. We further propose a novel meta-strategy solver, tailored for the offline setting, to guide strategy exploration in PSRO. Our incorporation of C onservatism principles from Off lin e reinforcement learning approaches for strategy E xploration gives our approach its name: \ourapproach . Experiments demonstrate \ourapproach ’s ability to extract lower-regret solutions than state-of-the-art offline approaches and reveal relationships between algorithmic components, empirical game fidelity, and overall performance. \doi XEYY6214 C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 25 AAAI Track 1. Introduction Game-theoretic analysis is a powerful tool when evaluating multiagent systems because it focuses attention on configurations where agents behave rationally relative to each other. In complex games with large strategy spaces, identifying such equilibrium profiles through multiagent reinforcement learning (MARL) requires substantial computation and data . Gathering ample data for game-theoretic analysis can thus be difficult and costly in real-world applications. A capability to perform such analysis offline from pre-generated, fixed-size datasets addresses the extreme case of limits on real-time data collection. Strategy generation using offline reinforcement learning is well-studied in single-agent and cooperative multiagent domains. Many offline learning algorithms adopt the conservatism principle : strategies should stay in dataset-informed regions and assume unknown regions yield low returns. This can be interpreted loosely as policy selection : among policies that seem approximately optimal, favor ones for which the dataset provides strong evidence and thus better relative confidence of high actual (true) return. Analogously, we frame offline game-solving as equilibrium selection , where we choose among profiles apparently in approximate equilibrium, aiming to minimize actual (true-game) regret. Our contribution is in the application and extension of conservatism to offline game-solving . This extension poses a special challenge: consideration of a candidate strategy profile’s potential strategic deviations. Our approach achieves this by combining offline RL algorithmic concepts and Policy Space Response Oracles (PSRO) ( Lanctot et al., 2017 ) , an online game-solving framework that iteratively extends a strategy set using deep RL best-responses to mixed strategies in an empirically estimated game model. Given a dataset of trajectories (sequences of true-game multiagent interactions), we first train an ensemble dynamics model to replace an online simulator and use prediction differences within the ensemble to quantify uncertainty under the dataset. We apply conservatism by introducing a response objective that balances reward, uncertainty given the current response target, and uncertainty of potential strategic deviations. Further, we propose a novel meta-strategy solver that minimizes a heuristic-based, pessimistic estimate of regret. This extension of C onservatism from Off lin e RL approaches to aid strategy E xploration in PSRO gives our approach its name: \ourapproach . In experiments on a sequential bargaining game, we find that \ourapproach tends to output lower-regret solutions than tested baselines. Our results highlight the role of conservatism in improving model fidelity, robustness to dataset changes, and solution regret. 2. Related Work Conservatism is a common theme in offline RL approaches ( Levine et al., 2020 ) . Model-free methods have incorporated conservatism through policy regularization ( Kumar et al., 2019 ; Wu et al., 2019 ) , value penalties ( Kumar et al., 2020 ) , and implicit behavioral constraints ( Nair et al., 2020 ; Peng et al., 2019 ) , while model-based methods have used uncertainty quantification and reward penalties ( Yu et al., 2020 ; Kidambi et al., 2020 ) . Conservative methods in offline MARL have addressed centralized coordination ( Wang et al., 2024 ) and nonstationarity in decentralized optimization ( Jiang and Lu, 2023 ) . Yang and Wang (2021) provide an overview of game-theoretic approaches in MARL. The framework of Policy Space Response Oracles (PSRO) ( Bighashdel et al., 2024 ; Lanctot et al., 2017 ) employs an empirical game modeling approach ( Wellman et al., 2025 ) , iteratively extending the strategy set using deep RL to derive policies maximizing return against an other-player response target defined by a meta-strategy solver (MSS). Substantial work has investigated strategy exploration ( Jordan et al., 2010 ) in PSRO, in particular through MSS selection of response targets ( Balduzzi et al., 2019 ; Marris et al., 2021 ; Muller et al., 2020 ; Smith et al., 2021 ; Wang and Wellman, 2023 ) or through definition of response objectives ( Perez-Nieves et al., 2021 ; Wang and Wellman, 2024 ; Nguyen et al., 2025 ) . In offline game-theoretic MARL, theoretical work has investigated the necessary conditions for equilibrium extraction ( Cui and Du, 2022 ; Zhong et al., 2022 ) , defining unilateral concentration as the property where a dataset covers an equilibrium and all its unilateral strategic deviations. Other approaches focus on offline evaluation of given strategies to determine which are most likely equilibria ( Zhang et al., 2023 ; Zhong et al., 2022 ) . Shao et al. (2025) employ conservative critics and low-rank matrix completions to select equilibria given a fixed dataset of payoff data for select policy profiles. Most relevant here is work on strategy generation for offline equilibrium extraction ( Li et al., 2022 ) , which uses a trained dynamics model in place of a simulator and mixes the final solution with a behavior cloning policy. Chen et al. (2026) adapts fictitious self-play using single-agent offline RL and importance sampling. 3. Notation We model a multiagent environment as a stochastic game defined by \langle\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},Z,O,n,\gamma\rangle . The environment follows trajectories of states s\in\mathcal{S} , influenced by actions a\in\mathcal{A} taken alternately by each of n agents. Each action prompts a transition according to the state transition function \mathcal{T}(s^{\prime}\mid s,a):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1] . Agent rewards are given by \mathcal{R}(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}^{n} , and \gamma\in[0,1) is a discount factor. The environment is partially observable, meaning agents draw observations z\in Z from an observation function O(s):\mathcal{S}\rightarrow Z . We denote an action-observation history by h_{t} , with \mathcal{H}_{t}\equiv(Z\times\mathcal{A})^{t} the space of conceivable t -length histories. A policy or pure strategy \pi_{i} for player i is defined as a mapping from h_{t} to a probability distribution over legal actions \pi_{i}:\mathcal{H}\to\Delta(\mathcal{A}) . We omit subscript i to refer to a joint policy (or strategy profile ) \pi=(\pi_{1},\dotsc,\pi_{n}) . A trajectory \tau describes a rollout of game play and is represented by a