SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems

Hikaru Shindo, Hanzhao Lin, Lukas Helff, Patrick Schramowski, Kristian Kersting · 2026

Read SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems by Hikaru Shindo, Hanzhao Lin, Lukas Helff, Patrick Schramowski, Kristian Kersting (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16022.txt.

Introduction

Shindo and four colleagues present SocialGrid, an embodied multi-agent world inspired by Among Us for testing planning, tasks, and social reasoning. Even the strongest open model, GPT-OSS-120B, scores below 60 percent on tasks and planning, getting stuck or failing movement. A Planning Oracle splits social skills from planning gaps, and planning help lifts scores. Social skills stay weak: agents spot deception at near-random chance, leaning on shallow cues over gathered evidence. SocialGrid adds auto failure analysis, fine metrics, and an Elo leaderboard from league play.

Catalog record

CREATORHikaru Shindo, Hanzhao Lin, Lukas Helff, Patrick Schramowski, Kristian Kersting
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75932
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems Hikaru Shindo, Hanzhao Lin, Lukas Helff, Patrick Schramowski, Kristian Kersting Abstract As Large Language Models (LLMs) transition from text processors to autonomous agents, evaluating their social reasoning in embodied multi-agent settings becomes critical. We introduce SocialGrid , an embodied multi-agent environment inspired by Among Us that evaluates LLM agents on planning, task execution, and social reasoning. Our evaluations reveal that even the strongest open model (GPT-OSS-120B ( OpenAI 2025 ) ) achieves below 60 % 60\% accuracy in task completion and planning, with agents getting stuck in repetitive behaviors or failing to navigate basic obstacles. Since poor navigation confounds evaluation of social intelligence, SocialGrid offers an optional Planning Oracle to isolate social reasoning from planning deficits. While planning assistance improves task completion, social reasoning remains a bottleneck: agents fail to detect deception at near-random chance regardless of scale, relying on shallow heuristics rather than accumulating behavioral evidence. SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents. We also establish a competitive leaderboard using Elo ratings from adversarial league play. † † affiliationnotice: Equal contribution 1 Introduction Artificial Intelligence is shifting toward a paradigm where Large Language Models (LLMs) are no longer passive text processors, but autonomous agents in dynamic environments ( Yehudai et al. 2025 ; Mohammadi et al. 2025 ; Xu et al. 2024a ; Park et al. 2023 ; Li et al. 2023 ; Guo et al. 2024 ; Wang et al. 2025b ; Chen et al. 2025 ; Wang et al. 2025a ; Cui et al. 2025 ; Light et al. 2023 ; Sun et al. 2025 ) . While textual reasoning has progressed rapidly, deploying these models as embodied agents fundamentally redefines intelligent behavior. An intelligent agent must navigate physical environments and act on observations, constructing internal mental models for social reasoning—such as identifying suspicious actors from their behavior. This convergence of spatial planning and social reasoning represents the next frontier of embodied intelligence. Success depends on an agent’s ability to unify these skills rather than exercise them in isolation ( Yehudai et al. 2025 ; Mohammadi et al. 2025 ) . However, current evaluation frameworks have not kept pace with this progress. Existing benchmarks typically assess these capabilities in silos, evaluating planning within single-agent gridworlds ( Martorell 2025 ) or social reasoning via disembodied text simulations ( Chi et al. 2024 ; Sarkar et al. 2025 ) . To address this gap, we introduce SocialGrid , an embodied multi-agent benchmark designed to evaluate agents along three distinct axes: spatial planning , task execution , and adversarial social reasoning . Inspired by the mechanics of the social deduction game Among Us , SocialGrid places agents in a gridworld environment where ”Crewmates” must navigate the environment, complete tasks, all while identifying hidden ”Impostors” who seek to sabotage the mission. This setup creates a rich, multi-objective environment with multiple agents competing, which requires complex integration of spatial memory and theory-of-mind. SocialGrid provides a highly modular benchmarking framework—adjustable by grid dimensions, spatial layout, and agent density—integrated with automated diagnostics that yield granular, actionable insights. Figure 1 : SocialGrid Overview. Inspired by Among Us , SocialGrid is a controllable, embodied benchmark evaluating LLM agents in multi-agent, multi-objective environments. (Left) User Input: Enables systematic control of environmental complexity (e.g., map area, room count) and agent configuration. (Center) Environment: Agents operate under physical constraints; an optional Planning Oracle isolates social reasoning from navigation. (Right) Evaluation & Feedback: Provides multi-dimensional metrics across spatial, task, and social axes, with automated failure analysis and league-based Elo rankings. See Appendix A for visualization screenshots. Our systematic evaluation of state-of-the-art models, ranging from 14B to 120B parameters, reveals two fundamental bottlenecks. First, we identify a severe deficit in spatial planning . In non-adversarial baselines, even the strongest open models (e.g., GPT-OSS-120B ( OpenAI 2025 ) ) achieve success rates below 60% in basic navigation and task completion, frequently getting stuck in repetitive behaviors or failing to navigate basic obstacles. This suggests that the raw reasoning power of LLMs does not automatically translate into effective embodied action. Since these planning failures would confound any evaluation of higher-level social intelligence, SocialGrid offers Planning Oracle , a symbolic module akin to a digital navigation system that guides the LLM’s physical behavior by providing automated pathfinding and action assistance. While this assistant successfully boosts navigation and task execution, it unmasks a second, more critical failure mode: social reasoning . Even with the burden of navigation and task execution lifted, agents consistently fail to detect deceptive behavior, performing near random chance in identifying impostors. We make the following contributions: (1) We introduce SocialGrid 1 1 1 Code available at github.com/ml-research/SocialGrid , a controllable, embodied multi-agent environment unifying spatial planning, task execution, and adversarial social reasoning into a single evaluation framework. (2) We quantify a significant gap in spatial reasoning, demonstrating that models still struggle with basic navigation and task execution in embodied environments. (3) We identify a fundamental social reasoning deficiency, with qualitative analysis revealing that LLMs rely on shallow heuristics rather than accumulating behavioral evidence. (4) We establish a standardized leaderboard based on adversarial league play and Elo ratings, providing the community with a rigorous metric for tracking progress in embodied intelligence. 2 SocialGrid SocialGrid offers agent developers a controllable and embodied benchmark to evaluate the planning, reasoning, and psychological inference capabilities of LLM agents. To this end, we adopt the GridWorld environment as the underlying framework ( Chevalier-Boisvert et al. 2023 ) , which offers a highly customizable and multi-agent environment. Figure 1 shows the overview of SocialGrid . Users specify environment parameters (map size, room layout, etc.) and agent configurations (left), which instantiate a gridworld where agents navigate, complete tasks, and engage in social reasoning under physical constraints (center). The framework outputs multi-dimensional metrics across spatial, task, and social reasoning axes, with automated failure analysis and Elo-based rankings (right). Embodiment in SocialGrid. We use embodied in the classical RL sense ( Sutton & Barto 1998 ) : agents perceive local observations, execute physical actions that modify world state, and receive feedback through closed-loop interaction. Unlike prompt engineering, actions in SocialGrid have spatial consequences—movement changes grid coordinates, doors block paths, and agents navigate shared space with collision constraints. A key goal of this work is to evaluate spatial planning and social reasoning jointly, as real-world deployment requires agents to handle both simultaneously. Contrary to in-context simulations ( Chi et al. 2024 ; Sarkar et al. 2025 ) , SocialGrid provides a grounded environment where agents must actively observe state changes and execute physical actions. This shifts the challenge from textual social deduction to embodied social reasoning under the joint constraints of space, action, and theory of mind. Now we devise the detailed design of SocialGrid . 2.1 Observations and Actions SocialGrid provides a customizable observation space. Limited Field of View: Each agent explores the minigrid environment with a limited field of view. The observation is the list of objects in the sight. This setting challenges the agents to explore the environment, collect information, and build a mental model of the whole map to perform planning and reasoning. Global Observation: Each agent

Read the complete text