Multi-agent In-context Coordination via Decentralized Memory Retrieval
Tao Jiang, Zichuan Lin, Lihe Li, Yi-Chen Li, Cong Guan, Lei Yuan, Zongzhang Zhang, Yang Yu, Deheng Ye · 2025
Read Multi-agent In-context Coordination via Decentralized Memory Retrieval by Tao Jiang, Zichuan Lin, Lihe Li, Yi-Chen Li, Cong Guan, Lei Yuan, Zongzhang Zhang, Yang Yu, Deheng Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10030.txt.
Introduction
Tao Jiang and eight co-authors present MAICC, a method for fast coordination among reinforcement learning agents that must cooperate on unseen tasks. Large transformers adapt to new tasks from context alone, but in multi-agent settings each agent deploys separately, so task alignment and credit for rewards can mismatch. MAICC trains a centralized embedding model that captures fine-grained trajectory patterns, then distills it into decentralized copies.
At execution time, agents retrieve relevant past trajectories as context and combine them with their current sub-trajectories to decide actions. A new memory mechanism balances fresh online data against stored memory, and a hybrid utility score mixes individual and team returns for fairer credit assignment. On foraging and StarCraft multi-agent benchmarks, adaptation to new tasks was faster than with existing methods.
Catalog record
| CREATOR | Tao Jiang, Zichuan Lin, Lihe Li, Yi-Chen Li, Cong Guan, Lei Yuan, Zongzhang Zhang, Yang Yu, Deheng Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10030
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-agent In-context Coordination via Decentralized Memory Retrieval Tao Jiang, Zichuan Lin, Lihe Li, Yi-Chen Li, Cong Guan, Lei Yuan, Zongzhang Zhang, Yang Yu, Deheng Ye Abstract Large transformer models, trained on diverse datasets, have demonstrated impressive few-shot performance on previously unseen tasks without requiring parameter updates. This capability has also been explored in Reinforcement Learning (RL), where agents interact with the environment to retrieve context and maximize cumulative rewards, showcasing strong adaptability in complex settings. However, in cooperative Multi-Agent Reinforcement Learning (MARL), where agents must coordinate toward a shared goal, decentralized policy deployment can lead to mismatches in task alignment and reward assignment, limiting the efficiency of policy adaptation. To address this challenge, we introduce M ulti- A gent I n- C ontext C oordination via Decentralized Memory Retrieval (MAICC), a novel approach designed to enhance coordination by fast adaptation. Our method involves training a centralized embedding model to capture fine-grained trajectory representations, followed by decentralized models that approximate the centralized one to obtain team-level task information. Based on the learned embeddings, relevant trajectories are retrieved as context, which, combined with the agents’ current sub-trajectories, inform decision-making. During decentralized execution, we introduce a novel memory mechanism that effectively balances test-time online data with offline memory. Based on the constructed memory, we propose a hybrid utility score that incorporates both individual- and team-level returns, ensuring credit assignment across agents. Extensive experiments on cooperative MARL benchmarks, including Level-Based Foraging (LBF) and SMAC (v1/v2), show that MAICC enables faster adaptation to unseen tasks compared to existing methods. Code is available at https://github.com/LAMDA-RL/MAICC . 1 Introduction In-Context Learning (ICL) has emerged as a compelling paradigm for few-shot generalization, enabling models to tackle novel tasks by interpreting contextual cues without explicit retraining [ 4 ] . This approach is epitomized by large language models, whose remarkable in-context abilities, unlocked through pretraining on vast web-scale corpora, have set new standards in natural language processing [ 6 ] . The success of this paradigm has catalyzed a parallel pursuit within Reinforcement Learning (RL) to instill agents with similar on-the-fly policy adaptation capabilities [ 35 ] . To this end, the prevailing strategy reformulates RL as a sequence modeling problem [ 5 ] : agents are trained on diverse trajectory datasets to internalize learning algorithms, allowing them to adapt to novel downstream tasks by conditioning on a few contextual examples [ 20 , 21 ] . This burgeoning field of In-Context Reinforcement Learning (ICRL) has shown notable promise, but its success has mainly been demonstrated in structured environments such as single-agent grid worlds and game-based tasks. ICRL has demonstrated strong capabilities for fast adaptation in single-agent environments. Typically, these methods condition on in-context trajectories and maintain a memory that is continuously updated with new online experiences to inform decision-making. Despite its notable success, extending this paradigm to cooperative Multi-Agent Reinforcement Learning (MARL) scenarios presents significant challenges. Unlike the single-agent setting, where an agent aims to maximize its individual cumulative rewards, cooperative MARL requires multiple agents to collaborate towards a shared objective [ 58 ] . This collaborative nature introduces distinct challenges, particularly when deployed in a decentralized manner [ 19 ] . Firstly, decentralized execution confines each agent to its local observations, often leading to a biased or incomplete understanding of the overall task characteristics. Secondly, agents typically receive only a shared team-level reward, making it difficult to assess individual contributions. This ambiguity in credit assignment can lead to the “lazy agent" problem [ 50 ] , where certain agents fail to learn effective policies and contribute meaningfully to the team’s success. These twin challenges of partial observability and credit assignment critically undermine the efficacy of conventional ICRL approaches in the MARL setting. Therefore, given the proven adaptive capabilities of ICL, a method that enables efficient adaptation to unseen cooperative tasks in decentralized multi-agent settings is urgently needed. To address the above objective, we propose M ulti- A gent I n- C ontext C oordination via Decentralized Memory Retrieval (MAICC), a framework designed for rapid team coordination under Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [ 38 ] . Specifically, we train a single centralized embedding model to extract fine-grained trajectory representations, and multiple decentralized embedding models for decentralized execution that approximate the centralized model to obtain team-level task information. With these pretrained models, we retrieve relevant multi-agent trajectories to serve as in-context examples. These retrieved trajectories, combined with agents’ current sub-trajectories, are used to guide and improve the decision-making process. During test time, we introduce a novel memory mechanism that efficiently balances an online replay buffer with a multi-task offline dataset for trajectory retrieval. Building upon this memory, we design a hybrid utility score that integrates both individual- and team-level returns, thereby enabling more accurate credit assignment across agents. We evaluate our method on several standard cooperative MARL benchmarks, including Level-Based Foraging (LBF) [ 40 ] and the StarCraft Multi-Agent Challenge (SMAC) v1 and v2 [ 45 , 9 ] . Experimental results show that MAICC, equipped with efficient trajectory retrieval for ICL, enables significantly faster adaptation to unseen tasks compared to existing ICRL and multi-task MARL baselines. Additionally, visualizations of the learned trajectory embeddings verify the effectiveness of our embedding model design, capturing both individual- and team-level behavior patterns. Ablation studies further isolate and confirm the contribution of each key component in our framework. Together, these findings demonstrate MAICC’s strong empirical performance, its ability to address the limitations of prior ICRL approaches, and its potential for broader deployment in complex multi-agent scenarios. 2 Related Work In-context RL. By framing RL as a sequence modeling problem, Decision Transformer (DT) [ 5 ] can make decisions based on provided prompts [ 56 ] . Subsequent studies scaled up model size and training data, enabling agents to exhibit ICL capabilities [ 22 , 43 ] . Algorithm Distillation [ 20 ] takes a significant step towards ICRL by utilizing historical trajectories. This enables agents to automatically improve their performance through trial and error, without updating their parameters. Agentic Transformer (AT) [ 29 ] further demonstrates that cross-episodic contexts can help agents leverage hindsight, thus enabling performance improvement at test time [ 13 ] . Decision-Pretrained Transformer (DPT) [ 21 ] explores an alternative approach by predicting the optimal action given random historical trajectories and the current state. Subsequently, Retrieval-Augmented Decision Transformer (RADT) [ 46 ] introduces retrieval augmentation into ICRL, utilizing a DT-based embedding model to select relevant historical trajectories and thereby further aid action prediction. However, these methods have only demonstrated effectiveness on single-agent tasks with simple interactions [ 49 ] and perform poorly on more complex decentralized cooperative tasks. In contrast, our approach achieves efficient trajectory retrieval tailored to the characteristics of Multi-Agent Systems (MASs), thereby facilitating collaborative adaptation to unseen tasks. To the best of our knowledge, our method is the first ICRL approach for Dec-POMDPs [ 38 ] . Cooperative multi-agent RL. Many real-world problems are large-scale and complex, rendering single-agent modeling inefficient and often impractical [ 10 ] . These challenges are more effectively addressed within the MAS setting [ 7 ] , where MARL provides a robust framework for solution [ 58 ] . In cooperative MARL, where agents pursue shared objectives, significant