High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning
Qinyu XuSchool of Management and EngineeringNanjing University [email protected] Yuanyang ZhuSchool of Information ManagementNanjing University [email protected], Xuefei WuSchool of Management and EngineeringNanjing University [email protected] Chunlin ChenSchool of Robotics and · 2025
Read High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning by Qinyu XuSchool of Management and EngineeringNanjing University [email protected] Yuanyang ZhuSchool of Information ManagementNanjing University [email protected], Xuefei WuSchool of Management and EngineeringNanjing University [email protected] Chunlin ChenSchool of Robotics and (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20218.txt.
Introduction
Qinyu Xu and three co-authors propose Continued Fraction Q-Learning, a value decomposition method for multi-agent reinforcement learning that can model high-order interactions between agents. Knowing how agents interact is central to coordination, but earlier methods either hit a combinatorial explosion or hid the interactions inside opaque networks. Their framework captures interactions of arbitrary order with cost that grows only linearly in the number of agents.
A variational information bottleneck extracts latent information used to assign credit to each agent. That step filters out noisy interactions, which improves both cooperation and interpretability at once. Extensive experiments show consistently better performance than rivals, and the explanations the method gives line up with the authors theoretical analysis of the interactions.
Catalog record
| CREATOR | Qinyu XuSchool of Management and EngineeringNanjing University [email protected] Yuanyang ZhuSchool of Information ManagementNanjing University [email protected], Xuefei WuSchool of Management and EngineeringNanjing University [email protected] Chunlin ChenSchool of Robotics and |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning Qinyu XuSchool of Management and EngineeringNanjing University [email protected] Yuanyang ZhuSchool of Information ManagementNanjing University [email protected], Xuefei WuSchool of Management and EngineeringNanjing University [email protected] Chunlin ChenSchool of Robotics and AutomationNanjing University [email protected] Abstract The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity 𝒪 ( n ) \mathcal{O}\left({n}\right) in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis. 1 Introduction Cooperative multi-agent reinforcement learning (MARL) has recently drawn much attention oroojlooy2023review , such as autonomous vehicles han2022stable ; zhang2023spatial ; dinneweth2022multi , robotics huang2020multi ; wang2024multi , and autonomous warehouses zhou2021multi ; krnjaic2024scalable . Success in these domains generally depends on a tapestry of local interactions: agents influence one another not only through explicit pairwise effects but also through higher-order dependencies that shape emergent behaviour yang2018mean ; bloembergen2015evolutionary , which is a key factor to make coordination efficient. How to model those interactions among agents and to do so in a way humans can understand remains a challenge. Value decomposition MARL has emerged as a powerful framework with the large capacity of deep neural networks and the centralized training and decentralized execution (CTDE) paradigm, which factorises a team return into individual utilities. Leading value decomposition models—such as the monotonic mixer in QMIX rashid2018qmix —capture interaction structure only implicitly, leaving their decisions opaque. Post-hoc explainers (e.g., Shapley value attribution wang2020shapley and masked-attention visualisation goto2022solving ) offer limited insight and could not recover the underlying temporal or relational dynamics. Another inherent interpretable technique, like decision tree milani2022maviper ; liu2025mixrts ; vos2024optimizing or Shapley-based wang2020shapley , handles only low-order interactions; higher-order terms still explode combinatorially. Here, we argue that modeling high-order collaboration patterns among agents is still crucial for promoting coordination, which is still missing. Continued fractions bender1994continued ; puri2021cofrnets provide a natural remedy: their recursive form captures interactions of any order, and a simple truncation yields a finite and interpretable approximation without combinatorial blow-up. Building on these ideas, we propose Continued Fraction Q-Learning (QCoFr), a novel framework that combines expressive value decomposition with intrinsic interpretability. QCoFr represents the joint action-value function as a weighted sum of continued fraction modules—recursive structures that explicitly approximate arbitrary-order interactions among agents with linear complexity \mathcal{O}(n) in the number of agents. To strengthen credit assignment, QCoFr incorporates a variational information bottleneck (VIB) alemi2017deep to capture task-relevant latent information to bridge local histories to the optimal joint action \bm{u} . The assistive information will be used to estimate the credits together with global state \bm{s} for assisted value factorization, which relieves the spurious correlation between state \bm{s} and Q_{tot} . We also give a rigorous proof that QCoFr takes a linear combination of continued fractions and has the property of universal approximation, even each with finite depth and linear layers. We demonstrate through extensive experiments on LBF, SMAC, and SMACv2 benchmarks that QCoFr not only generally achieves better performances on solving all tasks but also provides interpretability lacking in the state-of-the-art baselines. 2 Background Dec-POMDP. A fully cooperative multi-agent task can be modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) oliehoek2016concise . A Dec-POMDP can be defined as a tuple \left<N,S,U,P,r,Z,O,n,\gamma\right> , where N denotes a set of n agents and s\in S is the global state of the environment. At each time step, agent i selects an action u_{i}\in U , forming a joint action \bm{u}_{t}\in\mathbf{U}\equiv U^{n} . This results in a the next state s^{\prime} according to the state transition function P\left(s^{\prime}\mid s,\bm{u}\right):S\times\mathbf{U}\times S\rightarrow[0,1] . All agents obtain the same joint reward r(s,\bm{u}):S\times U^{n}\rightarrow\mathbb{R} and \gamma\in(0,1] is the discount factor. Due to partial observability, each agent i\in N receives individual observation z_{i}\in Z from observation function o_{i}\in O(s,i) . Each agent maintains an action-observation history \tau_{i}\in T , conditioned by its policy \pi_{i} . The overall objective is to find an optimal joint policy \bm{\pi}=\left\langle\pi_{1},\ldots,\pi_{n}\right\rangle to maximize the joint value function Q^{\bm{\pi}}\left(s_{t},\bm{u}_{t}\right)=\mathbb{E}\left[R_{t}\mid s_{t},\bm{u}_{t}\right] . Value Decomposition. Value decomposition has emerged as a dominant paradigm in multi-agent reinforcement learning (MARL) under the centralized training and decentralized execution (CTDE) paradigm. It seeks to approximate the joint action-value function Q_{tot} by decomposing it into individual utility functions Q_{i} , where each utility depends solely on the agent’s local trajectory \tau_{i} . To ensure that decentralized action selection remains consistent with the globally optimal joint action, the decomposition must satisfy the Individual-Global-Max (IGM) principle rashid2018qmix : \begin{array}[]{r}\arg\max_{\bm{u}}Q_{tot}(\bm{\tau},\bm{u})=\left\{\arg\max_{u_{1}}Q_{1}\left(\tau_{1},u_{1}\right),\cdots,\arg\max_{u_{n}}Q_{n}\left(\tau_{n},u_{n}\right)\right\}.\end{array} (1) Among representative methods, VDN sunehag2017value implements a simple additive decomposition, treating all agents’ contributions equally by summing their utilities. To capture more complex coordination while preserving IGM, QMIX rashid2018qmix introduces a mixing network that combines utilities into the joint value through a monotonic function. During training, the mixing network and agent networks are jointly optimized by minimizing the temporal-difference (TD) loss of Q_{tot} , while during execution, agents act independently using their local policies derived from the learned utility functions. The introduction of representative related works for the above formulation can be referred to in Appendix A . 3 High-order Interactions Modeling for Decomposition In a large multi-agent task, agents are usually decomposed into several coalitions, each consisting of multiple agents that cooperate to accomplish the common goal. Besides, each agent could belong to different coalitions at different time steps. This coalition organization is general and can characterize most coordination patterns among agents. Thus, it is essential to model complex interactions among agents and estimate their credits for understanding their coordination patterns. In this section, we provide an informal comparative analysis of using the continued fraction network (CFN) architecture against widely adopted value decomposition methods: VDN sunehag2017value , QMIX rashid2018qmix , and NA 2 Q liu2023n . From the view of value factorization, VDN provides a simple yet highly interpretable approach by representing the joint Q-value simply as the sum of individual agent Q-values, Q_{tot}(\bm{\tau},\bm{u})=\sum_{i\in N}Q_{i}\left(\tau_{i},u_{i}\right) . VDN inherently models only first-order interactions, which could limit its ability to capture richer and group-level coordination. QMIX enriches the functional class of factorisation than that of VDN by introducing a state-conditioned monotonic mixer Q_{tot}(s,\bm{\tau},\bm{u})=f_{\mathbf{QMIX}}\left(\bm{s},[Q_{1}\left(\tau_{1},u_{1}\right),\ldots,Q_{n}\left(\tau_{n},u_{n}\right)]\right) , where f_{\mathbf{QMIX}} is constrained to be monotonic in each argument. This implicit higher-order modeling increases representational capacity, but at the expense of interpretability— f_{\mathbf{QMIX}} behaves as a black box, obscuring which interactions drive performance. NA 2 Q tries to expand the joint value via a Taylor-like decomposition across all agents up to order n : \begin{array}[]{r}Q_{tot}=f_{0}+\sum_{i=1}^{n}\alpha_{i}\underbrace{f_{i}\left(Q_{i}\right)}_{\text{order-1 }}+\cdots+\sum_{k\in\mathcal{D}_{h}}\alpha_{k}\underbrace{f_{k}\left(Q_{k}\right)}_{\text{order-$h$}}+\cdots+\alpha_{1\ldots n}\underbrace{f_{1\ldots n}\left(Q_{1},\ldots,Q_{n}\right)}_{\text{order-$n$}},\end{array} (2) where f_{k}\in\left\{f_{1},\cdots,f_{1\ldots n}\right\}^{m} is modeled with neural additive model agarwal2021neural and \mathcal{D}_{h} is the set of all non-empty subsets of h\in\{1,...,n\} with order-