ESI-Bench : Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

Yining Hong* Jiageng Liu* Han Yin Manling Li Leonidas Guibas Li Fei-Fei Jiajun Wu Yejin Choi · 2026

Read ESI-Bench : Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop by Yining Hong* Jiageng Liu* Han Yin Manling Li Leonidas Guibas Li Fei-Fei Jiajun Wu Yejin Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18746.txt.

Introduction

Yining Hong and seven co-authors introduce ESI-Bench, a 2026 benchmark for spatial intelligence. Agents act to gain views and reason about how they change with action, uncovering occluded structure and dynamics. Spanning 10 categories and 29 subcategories on OmniGibson, it asks agents to sequence perception and manipulation. Active exploration beats passive sensing, with agents finding strategies. Yet most failures come from action blindness: poor actions yield poor views and errors. Models also commit early with high confidence regardless of evidence, a gap better perception and interaction cannot close.

Catalog record

CREATORYining Hong* Jiageng Liu* Han Yin Manling Li Leonidas Guibas Li Fei-Fei Jiajun Wu Yejin Choi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE116291
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ESI-Bench : Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop Yining Hong* Jiageng Liu* Han Yin Manling Li Leonidas Guibas Li Fei-Fei Jiajun Wu Yejin Choi Abstract Spatial intelligence unfolds through a perception–action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen , they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-Bench , a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke’s core knowledge systems. Agents must decide what abilities to deploy — perception, locomotion, and manipulation — and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close. All data, dataset construction codes, and evaluation scripts are publicly available at https://esi-bench.github.io/ . Figure 1 : ESI-Bench is a comprehensive benchmark for embodied spatial intelligence, spanning 10 task categories and 29 subcategories organized around Spelke’s four core knowledge systems ( 19 ) : object representation , layout and geometry , number representation , and agents and goal-directed actions . 1 Introduction Perception is often characterized in cognitive science as perceptually guided action ( 20 ; 9 ) : a perception–action loop where knowing how observations change as a function of action ( 18 ) , and knowing which actions elicit informative sensing, are often more challenging than sensing itself. This is especially critical in spatial intelligence, which concerns not only what is seen , but also what is unseen . Latent physical properties such as occluded structure, dynamics, containment, and functionality that are inaccessible to passive sensing must be actively revealed through interaction, making spatial intelligence inherently embodied. We move beyond prior formulations of spatial intelligence that assume passive oracle observations ( 11 ; 27 ; 30 ) by recasting the observer as an actor. Our work contrasts with prior works in three key ways: (1) from spatial sensing to spatial competence , where agents are evaluated not only on what they can perceive, but on whether they know which embodied abilities to deploy to solve spatial tasks; (2) selective sensing , where agents must determine which observations are worth acquiring, prioritizing task-relevant information over redundant or uninformative inputs; and (3) resolving perceptual ambiguities , where agents must reason through misleading observations to infer hidden spatial structures and underlying physical constraints beyond what is directly observed. We introduce ESI-Bench , a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories, 29 subcategories and 3,081 task instances, addressing the critical perception–action gap by focusing on questions that cannot be answered from passive observation alone. Our category design follows Spelke’s core knowledge systems ( 19 ) , which identify four faculties of spatial intelligence: object representation, layout and geometry, number representation, and agents & goal-driven actions. Building on these theoretical foundations, we conduct human surveys to identify the most challenging spatial tasks that require embodied interaction and manipulation within each faculty, which are distilled into a structured taxonomy spanning diverse forms of spatial reasoning, as illustrated in Figure 1 . These tasks only become meaningful when an agent has a body, a belief state, and physical stakes in the outcome: the agent must determine what abilities to deploy (perception, locomotion, manipulation), which actions to take (where to move, what to probe, how to manipulate), and how to execute them in the right order to successfully answer the questions. We conduct extensive experiments on state-of-the-art MLLMs across three paradigms: passive single-view, passive random multi-view, and active exploration, alongside a ground-truth oracle that separates perception errors from action errors. Our experiments reveal multiple key insights: (1) active exploration unlocks emergent spatial strategies : without explicit instruction, active agents spontaneously discover diverse action compositions, driving substantial gains over passive counterparts while passive random multi-view, despite consuming far more images, often adds noise rather than signal; (2) action blindness dominates perceptual blindness : for most tasks the bottleneck is not perception but action selection, as models given oracle viewpoints succeed largely, yet certain tasks expose a hard perceptual ceiling that no action can overcome; and (3) failures cascade and compound : suboptimal actions produce uninformative views, which trigger worse subsequent actions, creating a compounding chain of errors that cannot be recovered within the step budget. We further investigate whether explicit 3D representations can help. We find that while explicit 3D grounding stabilizes reasoning on depth-sensitive tasks by recovering information that 2D projections fundamentally lose, imperfect reconstructions prove more harmful than 2D baselines, as geometric artifacts actively distort fine-grained spatial relations and mislead downstream reasoning. Finally, human studies expose a critical gap in epistemic calibration ( i.e. , a model’s confidence matches the quality and uncertainty of its evidence). We observe that unlike humans who actively seek falsifying viewpoints, explore orthogonal angles, and revise their beliefs when contradicted, models commit prematurely with uniformly high confidence regardless of evidence quality, anchoring to first impressions and ignoring contradictory observations, a metacognitive failure that neither better perception nor more embodied interaction alone can close. 2 Related Works Table 1 : Comparison with spatial reasoning and embodied benchmarks. Here, embodied means that an agent is situated in an embodied environment and acts through an explicit action space in the environment. Active? : whether the benchmark supports active perception. Action Space : L = locomotion, P = perception, M = manipulation. Core Knowledge : Obj = object, Geom = geometry, Num = number, Phys = physics, Agt = agent. Hidden : whether the benchmark targets latent properties such as occlusion, containment, dynamics that are inaccessible from passive observation. Benchmark Type Active? Action Hidden Core VSR ( 11 ) Spatial ✗ P ✗ Geom BLINK ( 8 ) Spatial ✗ P ✗ Obj, Geom, Num 3DSRBench ( 16 ) Spatial ✗ P ✗ Obj, Geom VSI-Bench ( 27 ) Spatial ✗ P ✗ Obj, Geom, Num MMSI-Bench ( 30 ) Spatial ✗ P ✗ Obj, Geom MindCube ( 21 ) Spatial ✗ P ✓ Geom, Phys PhysBench ( 5 ) Spatial ✗ P ✓ Obj, Geom, Phys CausalSpatial ( 15 ) Spatial ✗ P ✓ Obj, Phys EmbSpatial-Bench ( 6 ) Spatial ✗ P ✗ Obj, Geom OpenEQA ( 17 ) Embodied ✓ L, P ✗ Obj, Geom EmbodiedBench ( 28 ) Embodied ✓ L,

Read the complete text