CRAFT : Grounded Multi-Agent Coordination Under Partial Information
Abhijnan Nath, Hannah VanderHoeven, Nikhil Krishnaswamy · 2026
Read CRAFT : Grounded Multi-Agent Coordination Under Partial Information by Abhijnan Nath, Hannah VanderHoeven, Nikhil Krishnaswamy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25268.txt.
Introduction
Nath, VanderHoeven, and Krishnaswamy test whether language models can build together while each sees only part of the job. In CRAFT, several agents hold partial views of a 3D build. They must talk their way to one shared structure none can fully see. The benchmark sorts failures into grounding, belief, and communication faults. Fifteen models were tried, from small open ones to top frontier systems. Stronger solo reasoning did not mean better teamwork. Small open models often matched the giants, which marks coordination as an open problem.
Catalog record
| CREATOR | Abhijnan Nath, Hannah VanderHoeven, Nikhil Krishnaswamy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104694 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.25268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CRAFT : Grounded Multi-Agent Coordination Under Partial Information Thanks: https://www.signallab.ai Abhijnan Nath, Hannah VanderHoeven, Nikhil Krishnaswamy Abstract We introduce CRAFT , a multi-agent benchmark for evaluating pragmatic communication in large language models under strict partial information. In this setting, multiple agents with complementary but incomplete views must coordinate through natural language to construct a shared 3D structure that no single agent can fully observe. We formalize this problem as a multi-sender pragmatic reasoning task and provide a diagnostic framework that decomposes failures into spatial grounding, belief modeling, and pragmatic communication errors—including a taxonomy of behavioral failure profiles in both frontier and open-weight models. Across a diverse set of models—including 8 open-weight and 7 frontier models including reasoning models—we find that stronger reasoning ability does not reliably translate to better coordination: smaller open-weight models often match or outperform frontier systems, and improved individual communication does not guarantee successful collaboration. These results suggest that multi-agent coordination remains a fundamentally unsolved challenge for current language models. Our code can be found at https://github.com/csu-signal/CRAFT 1 Introduction Large language models (LLMs) have evolved from single-turn assistants into components of multi-agent systems, powering coding workflows, scientific discovery pipelines, and autonomous task execution environments where multiple agents must coordinate toward shared goals ( 34 ; 80 ; 8 ; 11 ) . In such settings, agents—often built on different models—must communicate and collaborate effectively, making coordination a central challenge for both research and deployment. However, recent work shows that coordination between LLM agents remains fragile even in simple settings ( 42 ; 1 ; 64 ; 13 ; 52 ) . Models struggle with partner modeling and joint decision-making even when interacting with identical copies of themselves. This challenge becomes substantially harder in realistic scenarios where agents operate under partial observability , hold complementary private information, and must communicate through open-ended natural language to achieve a shared goal. We argue that the missing capability is pragmatic communication —deciding what to say, how much to say, and when to say it based on other agents’ knowledge and needs ( 21 ; 15 ) . While frameworks such as Rational Speech Acts (RSA) ( 20 ) and the Bounded Pragmatic Speaker (BPS) model ( 55 ) formalize this behavior, existing evaluations largely focus on single-agent reasoning or offline interpretation ( 29 ; 83 ) . This misses out on a key gap: LLMs may exhibit strong internal reasoning (formal competence) but fail to use it effectively in interaction (functional competence) ( 45 ; 11 ) . Prior work has studied pragmatic understanding in controlled reference games, typically in dyadic settings ( 35 ; 41 ; 26 ) . In contrast, we focus on pragmatic generation in multi-agent environments with partial information. Although 55 suggests RLHF-trained LLMs can be interpreted as bounded pragmatic speakers ( 65 ) , it remains unclear whether this extends to settings requiring sustained coordination under information asymmetry. This gap is most evident in tasks such as the Distributed Partial Information Puzzle (DPIP; 84 ), where agents must iteratively establish common ground when given complementary but distinct information. Humans succeed through sustained pragmatic repair across multiple turns 1 1 1 Table 5 shows a snippet from a DPIP “game” where humans take \approx 30s and 23 turns (of dialogic exchanges) involving pragmatic repair to resolve an action. , but current LLM evaluations rely on offline analysis or dyadic-agent tasks ( 72 ) for evaluating LLMs. Such works do not test whether models can produce effective communication in multi-agent, partially observable settings. Figure 1: CRAFT framework overview. A structure generator creates a target 3D object and three private 2D views for directors, enforcing information asymmetry. At each turn, directors produce instructions from their partial views, which a builder executes via PLACE , REMOVE , or CLARIFY actions in the CRAFT engine. The system logs task progress and evaluates communication using LLM judges for spatial grounding, mind modeling, and pragmatic sufficiency. To address this, we introduce CRAFT (Collaborative Reasoning Agents For Construction Tasks) , a multi-agent benchmark for evaluating pragmatic communication in grounded, partially observable environments. In CRAFT , three director agents, each with a private view of a 3D target structure, must coordinate via natural language to guide a builder toward the correct configuration. Crucially, success depends not on individual reasoning alone, but on how effectively agents communicate complementary information to support joint action via the builder. We provide novel theory on CRAFT as a “multi-sender” BPS problem, where agents must consider integrating others’ privately-held goal state information via pragmatic “communication” across a horizon of conversation turns in order to successfully complete the task. This provides a principled look at disentangling core communicative failure modes that LLMs—both frontier as well as smaller open-weight models—face in effective communication. To measure this, we propose a novel LLM “grader” evaluation framework that scales efficiently to thousands of dialogue turns. Key results. Our evaluation reveals several surprising patterns. First, frontier and proprietary models do not reliably outperform smaller open-weight models: models such as Mistral-7B and Qwen-2.5-7B Instruct outperform the majority of frontier systems on the task. Second, higher-quality individual communication does not translate to better overall collaboration—LLM-Judge metrics of communication quality are often negatively correlated with task progress. Third, we identify a consistent behavioral failure mode we term correction spirals , where agents over-correct perceived errors (e.g., excessive removal actions) after failing to overcome the information asymmetry via communication, leading to wasted turns and degraded performance. Together, these results show that capabilities that improve individual reasoning—such as perspective-taking—is necessary but not sufficient for collective task success in CRAFT ’s multi-sender setting. We make three contributions: 1. CRAFT benchmark. A multi-agent, grounded construction benchmark with procedurally generated structures across three complexity levels, a physics-constrained game engine, and fine-grained trajectory logging. We introduce an oracle-assisted builder interface to disentangle builder capability from director communication, enabling proper credit assignment in multi-agent systems ( 27 ) . 2. Multi-sender BPS framework. A theoretical formalization of directors in CRAFT as bounded pragmatic speakers ( 15 ) , grounded in Rational Speech Acts, providing a principled model of collaborative communication under partial observability and a framework for diagnosing communicative failures. 3. Empirical evaluation. A large-scale evaluation across open-weight and proprietary models using a suite of LLM “graders” that decompose failures into spatial grounding, reasoning, and pragmatic communication. Our analysis provides interpretable, per-turn diagnostics and reveals how model scale and communication behavior impact collaborative performance. 2 Related Works Spatial reasoning and multi-agent coordination under partial observability. Recent work has made significant progress in spatial reasoning for LLMs and multimodal models, including viewpoint consistency, 3D reasoning, and object localization ( 40 ; 33 ; 12 ; 37 ; 44 ; 79 ; 76 ; 74 ) , with extensions to dialogue, navigation, and spatio-temporal understanding ( 5 ; 18 ; 77 ; 24 ; 81 ; 46 ) . In parallel, recent work studies LLM coordination in symmetric games, zero-shot partner matching, and agentic collaboration pipelines ( 42 ; 1 ; 64 ; 22 ; 47 ; 7 ; 31 ; 23 ; 49 ; 50 ; 51 ) . However, these two lines of work largely evolve independently: spatial reasoning benchmarks are typically framed as single-agent perception or