MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning
Haohan Yu, Lu Wang, Jinmiao Cong, Shengzhi Wang, Chanjuan Liu · 2026
Read MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning by Haohan Yu, Lu Wang, Jinmiao Cong, Shengzhi Wang, Chanjuan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01805.txt.
Introduction
Yu and four colleagues design a learning signal that helps teams of AI agents coordinate. In multi-agent reinforcement learning, several agents learn together toward a shared goal, but each one only sees its own view of the world. A helpful move may pay off for a teammate later, yet the shared team reward gives no direct credit for it. Their framework, MAGIC, estimates how one agent's action shapes its teammates' futures over several steps, then turns only the beneficial effects into extra rewards.
Counterfactual comparisons pit what actually happened against what would have happened under a different action, and an advantage gate steers exploration toward moves aligned with the task. On particle-world tasks and StarCraft micromanagement benchmarks, MAGIC beat leading prior methods, with average relative gains of about twenty-seven percent and ten percent. The result suggests that crediting cross-agent influence step by step builds steadier teamwork.
Catalog record
| CREATOR | Haohan Yu, Lu Wang, Jinmiao Cong, Shengzhi Wang, Chanjuan Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01805
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning Haohan Yu, Lu Wang, Jinmiao Cong, Shengzhi Wang, Chanjuan Liu Abstract A key challenge in multi-agent reinforcement learning (MARL) lies in designing learning signals that effectively promote coordination among agents. Designing such signals requires estimating how one agent’s current action affects its teammates over future interaction steps. To address this, we introduce Multi-step Advantage-Gated Interventional Causal MARL (MAGIC), a framework that estimates multi-step action effects between agents and selectively converts them into intrinsic rewards. MAGIC uses counterfactual action interventions to compare teammate futures under factual and counterfactual branches, and introduces a gate based on advantage to direct exploration toward beneficial behaviors aligned with the task goal. Experiments on Multi-Agent Particle Environments (MPE) and StarCraft micromanagement benchmarks (SMAC and SMACv2) show that MAGIC consistently outperforms leading prior methods, with average relative final performance improvements of 26.9% and 10.1%, respectively. 1 Introduction Cooperative multi-agent reinforcement learning (MARL) studies how multiple agents learn policies to optimize a shared team objective ( 11 ; 14 ; 18 ) . In this setting, each agent acts from its local observation, but the value of its action often depends on how it affects other agents and the future team outcome. This makes coordination difficult when team rewards are sparse or delayed, because an action that helps a teammate may not receive immediate feedback from the environment ( 12 ; 9 ; 1 ; 5 ) . Centralized training with decentralized execution (CTDE) is widely used to alleviate this difficulty ( 11 ; 14 ; 18 ) . It allows the learner to use centralized information during training while keeping each agent’s policy decentralized during execution. However, the feedback provided by the team reward remains indirect for each individual action. The shared return can evaluate the outcome of a joint behavior, but it does not directly indicate whether one agent’s current local action will help its teammates coordinate in the future. This motivates an additional training signal that can make the cooperative value of local actions more explicit. One common approach is to use inter-agent influence as an intrinsic reward ( 6 ; 12 ; 8 ; 2 ) . Early influence-based methods encourage coordination by measuring how much one agent affects others through immediate action responses, trajectory dependence, or mutual information ( 6 ; 8 ; 7 ; 10 ) . These signals show that influence can be useful for cooperative learning, but they also leave two key issues. First, useful influence may be delayed. A current action may have little effect on a teammate in the next transition, but may change the teammate’s future position, route, or attack timing after several interaction steps. Second, strong influence is not necessarily useful. An agent can strongly affect its teammates while still reducing the team return. Thus, a useful coordination signal should capture delayed influence and should also be aligned with task improvement ( 8 ; 5 ; 13 ) . Figure 1 : Motivating predator–prey example. Left: a delayed give-way action helps teammate b ; right: a high-influence action disrupts b and hurts team return. Figure 1 illustrates delayed useful influence and harmful influence in a cooperative predator-prey task. In the left panel, agent a ’s give-way action does not immediately bring it closer to the prey, but helps agent b reach a better future capture position. In the right panel, agent a strongly changes agent b ’s route, but this interference moves the team away from a better capture outcome. These examples make the desired training signal more specific. It should look beyond immediate action effects and encourage such effects only when they support the team objective. Based on this principle, we propose Multi-step Advantage-Gated Interventional Causal MARL (MAGIC). MAGIC augments a standard CTDE learner with a causal and task-aligned intrinsic reward computed only during training. To estimate whether one agent’s current action affects its teammates’ future states, MAGIC constructs factual and counterfactual branches at each sampled time step. The factual branch keeps the realized action of the source agent, while the counterfactual branches replace only this source action with valid alternatives. All branches share the same current state and the same actions of the other agents. MAGIC then compares the teammate futures produced by these branches over multiple rollout steps. If replacing the source agent’s current action leads to clearly different teammate futures, this indicates that the realized action has a strong counterfactual action effect in the current situation. This comparison is different from measuring statistical dependence between actions and future states in collected trajectories. It directly asks whether the teammates’ future states would change if only the source agent’s action were replaced in the same situation. To compare the teammate futures produced by these branches over multiple rollout steps, MAGIC uses a learned forward model to roll out the factual and counterfactual branches from the same sampled decision context. In this design, the forward model is not used as an exact long-term simulator. Its role is to preserve the differences between branches that are relevant to teammate future states within a finite horizon. When the rollout keeps the relative strength of factual and counterfactual effects separable, MAGIC can estimate action effects reliably even if the absolute predicted states are imperfect. After the action effect score is obtained, MAGIC aligns it with the task objective through an extrinsic advantage gate. The action effect score is agent-specific because it measures the effect of each agent’s own realized action on its teammates’ futures. The gate uses an extrinsic advantage computed at the team level to estimate whether the realized team transition improves expected task return. This gate keeps more of the action effect signal when the transition benefits the task and suppresses it when the transition is harmful. As a result, MAGIC encourages agents to produce effects on teammates only when such effects are supported by task improvement. Our main contributions are as follows. • We propose a multi-step counterfactual action effect estimator. It measures how replacing one agent’s current action changes its teammates’ future states over multiple rollout steps, which allows the learning signal to capture delayed coordination effects. • We introduce an advantage-gated intrinsic reward that converts action effects into rewards only when they are supported by task improvement. The gate retains more intrinsic reward for beneficial team transitions and suppresses high-effect behaviors that reduce the expected task return. • We evaluate MAGIC across multiple MARL benchmarks and task families, including Multi-Agent Particle Environments (MPE), StarCraft Multi-agent Challenge (SMAC), and SMACv2. MAGIC outperforms representative methods from the two related lines studied in this paper, including influence-based intrinsic-reward methods and task-aligned coordination methods. Beyond final performance, we use ablations to identify the role of each component and diagnostic studies to examine the reliability of the proposed method. 2 Related Work Our work studies training signals for improving coordination in cooperative MARL. Related efforts include influence-based intrinsic rewards and task-aligned coordination signals. Influence-based intrinsic rewards. Intrinsic rewards are widely used to encourage exploration and coordination under sparse or delayed rewards ( 3 ; 12 ; 9 ; 1 ; 5 ) . In MARL, recent social-influence and coordinated-exploration methods encourage agents to affect each other through mutual information,