Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

Yuheng Jing, Kai Li, Ziwen Zhang, Jiajun Zhang, Zeyao Ma, Jiaxi Yang, Lei Zhang, Zhe Wu, Jinmin He, Junliang Xing, Jian Cheng · 2026

Read Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork by Yuheng Jing, Kai Li, Ziwen Zhang, Jiajun Zhang, Zeyao Ma, Jiaxi Yang, Lei Zhang, Zhe Wu, Jinmin He, Junliang Xing, Jian Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24423.txt.

Introduction

Jing and ten co-authors introduce ICRL4AHT, a benchmark for in-context reinforcement learning in teamwork. In-context reinforcement learning lets agents adapt to new tasks, but its teamwork value with unknown partners was unexplored. The benchmark offers a teammate suite plus a pipeline for teammate generation, history collection, and online evaluation. Tests of history-conditioned methods including Algorithm Distillation and Decision-Pretrained Transformer reveal limits. Unlike in single-agent domains, the baselines fail to adapt at test time, often trailing random baselines on unseen teammates and layouts.

Catalog record

CREATORYuheng Jing, Kai Li, Ziwen Zhang, Jiajun Zhang, Zeyao Ma, Jiaxi Yang, Lei Zhang, Zhe Wu, Jinmin He, Junliang Xing, Jian Cheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122398
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24423

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork Yuheng Jing, Kai Li, Ziwen Zhang, Jiajun Zhang, Zeyao Ma, Jiaxi Yang, Lei Zhang, Zhe Wu, Jinmin He, Junliang Xing, Jian Cheng Abstract In-Context Reinforcement Learning (ICRL) has enabled foundation agents to adapt instantaneously to novel tasks, yet its efficacy in Ad-Hoc Teamwork (AHT)—where coordination with unknown partners is required—remains unexplored. To rigorously evaluate this, we introduce a large-scale benchmark ICRL4AHT , built upon a high-throughput JAX implementation of Overcooked-V2. Our benchmark includes a large, diverse teammate suite spanning both RL and heuristic policies, enabling controlled train-test shifts, and provides a reproducible end-to-end pipeline for teammate generation, learning-history collection, dataset construction, and online multi-episode evaluation. We evaluate representative history-conditioned ICRL algorithms, including Algorithm Distillation (AD) and Decision-Pretrained Transformer (DPT), across millions of transitions. Results reveal notable limitations: contrary to their success in single-agent domains, these baselines fail to exhibit robust test-time adaptation in multi-agent settings. Specifically, these methods frequently underperform random baselines across both unseen teammate and unseen layout tracks, with no clear in-context improvement over long horizons. These findings highlight the challenges of strategic inference under partial observability within the OvercookedV2 AHT protocol, establishing our benchmark as a critical testbed for next-generation coordination algorithms. 1 Introduction The paradigm of In-Context Reinforcement Learning (ICRL) has fundamentally shifted how agents adapt to novel environments ( 36 ) . By treating an agent’s interaction history as a prompt for a sequence model, ICRL enables rapid, few-shot adaptation without the need for gradient updates or explicit fine-tuning. Recent foundational works ( 31 ; 62 ) have demonstrated that algorithms such as Algorithm Distillation (AD) ( 26 ) and Decision-Pretrained Transformer (DPT) ( 27 ) can achieve promising performance across diverse single-agent tasks, effectively ‘learning to learn’ from context. This success suggests a promising path toward foundation agents capable of solving open-ended problems in dynamic environments. However, the complexity of real-world deployment inherently involves interaction with other autonomous entities rather than static environments. This challenge is formalized as Ad-Hoc Teamwork (AHT) , where an agent must collaborate with novel partners without prior coordination or communication protocols ( 23 ) . Theoretically, ICRL is an ideal candidate for AHT: an agent could infer a partner’s hidden intent or policy purely from the interaction history—treating the teammate as a dynamic part of the environment context. Yet, unlike the existing literature which often relies on Population-Based Training ( 29 ; 30 ) or idealized self-play, the capability of general-purpose ICRL agents to adapt to truly unseen, heterogeneous teammates remains largely unverified. Progress in this direction is currently stifled by the lack of adequate evaluation infrastructure . While benchmarks like XLand-100B ( 38 ) provide the scale necessary for single-agent ICRL, the multi-agent domain lacks a counterpart that offers both scale (millions of transitions) and strategic diversity . Existing AHT benchmarks typically rely on small sets of homogeneous partners or limited task variants ( 6 ; 2 ) , failing to impose the severe distribution shifts required to rigorously test in-context generalization. Absent controlled variation in partner behavior across fixed tasks, observed performance conflates task competence with partner-specific overfitting , precluding any claim of genuine coordination. Figure 1 : Benchmark Pipeline Overview ( ICRL4AHT ). (1) A benchmark manifest specifies layouts, tasks, and other properties, as well as defines two evaluation tracks that disentangle generalization over teammates and layouts. (2) A diverse teammate policy pool is constructed from both RL-trained policies and heuristic policies, where teammate policies \pi^{-i} can be sampled for either training or testing. (3) Using OvercookedV2 layouts, interactions between an RL ego agent and a sampled teammate policy are rolled out and recorded as learning histories. (4) The collected data are packaged into a chunked HDF5 dataset with an episode / task index, fast metadata queries, and prefetch / cache support, enabling efficient context-window batch sampling for ICRL training. (5) ICRL learners are trained on the dataset and then (6) evaluated online for test-time adaptation under the two tracks, reporting metrics such as adaptation curves. To bridge this gap, we introduce ICRL4AHT , a large-scale benchmark with a reproducible dataset-generation and evaluation pipeline specifically designed to probe the limits of ICRL in cooperative settings . Built upon a high-performance JAX implementation of OvercookedV2 ( 14 ) , our framework features a highly efficient generation pipeline that curates a Teammate Suite of unprecedented diversity , comprising both diverse RL policies and distinct heuristic policies. We structure the benchmark around two rigorous evaluation tracks: (1) Teammate Generalization , testing adaptation to unseen behaviors within familiar layouts, and (2) Layout Generalization , testing robustness to novel environmental dynamics. This design ensures that high performance requires genuine in-context reasoning rather than statistical overfitting. We utilize ICRL4AHT to conduct the first large-scale empirical analysis of representative history-conditioned ICRL methods (AD and DPT) in the context of AHT. Our results reveal a notable limitation in current sequence-model ICRL baselines: across both generalization tracks, existing ICRL methods often underperform simple random baselines . Furthermore, we observe flat learning curves during evaluation, indicating a near-absence of the expected in-context improvement, even as the interaction horizon expands. Further ablations on context length and teammate-action conditioning suggest that the gap is not merely an implementation detail or an easily tuned hyperparameter, but reflects a substantive limitation of the studied ICRL baselines when faced with realistic partner diversity and environment shifts. We release the full benchmark, including the teammate suite, dataset generator, and evaluation pipeline , to facilitate reproducibility and future research. Our repository is available at https://github.com/AHT-Hub/ICRL4AHT . Our contributions are threefold: • A Scalable Benchmark and Reproducible Pipeline : We provide a reproducible, JAX-based pipeline for teammate generation, learning-history collection, dataset construction, and standardized evaluation of cooperative multi-agent ICRL methods with controllable diversity. • A Diverse Teammate Suite : We construct a heterogeneous library of teammates that strictly separates training (RL-based) and testing (heuristic-based) distributions , enforcing a rigorous test of generalization. • Benchmarking the Limits : We demonstrate that representative history-conditioned sequence-model ICRL baselines, while successful in single-agent settings, struggle with the strategic inference required for partner adaptation under the OvercookedV2 AHT protocol studied here, highlighting a critical new frontier for the community. 2 Background 2.1 In-Context Reinforcement Learning Transformers have become a standard backbone for decision-making by reframing RL as sequence modeling ( 7 ; 22 ; 5 ; 66 ) and enabling scalable conditioning and multitask pretraining ( 13 ; 41 ; 59 ; 28 ; 46 ) . When pretrained with appropriate contextual structure, these models can exhibit ICRL— gradient-free test-time adaptation driven purely by conditioning on interaction histories—closely related to recurrent or meta-RL formulations ( 61 ; 10 ) ; see 3 for a comprehensive survey of meta-RL. Recent advances include noise-distillation-based emergence of ICRL ( 70 ) , scalable in-context agents such as AMAGO ( 15 ) and Vintix ( 42 ) , and analyses of when Q-learning can outperform model-based alternatives in this regime ( 56 ) . In AHT, this capability is especially desirable because an agent must infer and respond to unseen partner behaviors from limited online interaction; ICRL4AHT instantiates this evaluation setting with standardized online protocols and tracks that disentangle generalization across teammates and

Read the complete text