REM : Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories

Jacob Thompson, Emiliano Garcia-Lopez & Yonatan Bisk · 2025

Read REM : Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories by Jacob Thompson, Emiliano Garcia-Lopez & Yonatan Bisk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00736.txt.

Introduction

Jacob Thompson, Emiliano Garcia-Lopez, and Yonatan Bisk test how well AI models reason about space in a 2025 paper. People build mental maps that stay stable as they move around. Those maps let us track objects and how they relate, even from a new viewpoint.

The authors argue that large multimodal models, despite heavy video training, lack this basic skill. That gap matters for embodied uses, like robots that act in the world. Their REM evaluation uses multi-frame trajectories, sequences of views along a path, to probe the weakness. A clear test is the first step toward closing the gap.

Catalog record

CREATORJacob Thompson, Emiliano Garcia-Lopez & Yonatan Bisk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00736

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REM : Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories Thanks: Dataset and Code: github.com/jbt-cmu/REM Jacob Thompson, Emiliano Garcia-Lopez & Yonatan Bisk Abstract Humans build viewpoint-independent cognitive maps through navigation, enabling intuitive reasoning about object permanence and spatial relations. We argue that multimodal large language models (MLLMs), despite extensive video training, lack this fundamental spatial reasoning capability, a critical limitation for embodied applications. To demonstrate these limitations and drive research, we introduce REM (Reasoning over Embodied Multi-Frame Trajectories), a benchmark using controllable 3D environments for long-horizon embodied spatial reasoning. REM systematically evaluates key aspects like object permanence/distinction, spatial relationships, and numerical tracking across dynamic embodied viewpoints. Our evaluation shows that the best-performing current models exhibit promising overall performance, but become increasingly unreliable at even moderate complexity levels easily handled by humans. These findings highlight challenges MLLMs face in developing robust spatial representations from sequential visual input. Consequently, REM provides targeted metrics and diagnostics to foster improved spatial understanding in future models. 1 Introduction When navigating a familiar campus, park, or neighborhood, we rarely consult external maps or need explicit directions. Instead, we rely on sophisticated mental representations built through experience. These three-dimensional cognitive maps ( 28 ; 25 ) function as accessible spatial databases that persist independently of our current viewpoint. Our mental maps encode precise spatial relationships (”standing near the main gate, the statue is to the left of the library”) and allow us to plan optimal routes through areas currently out of sight (”cutting through the science building will save time”). This remarkable ability to maintain and interrogate comprehensive spatial representations across changing viewpoints even enables us to answer difficult questions like ”How many blue recycling bins are on campus?” through explicit mental enumeration, drawing on our persistent cognitive model rather than immediate visual perception. This fundamental spatial reasoning capability, which humans develop naturally, represents a critical challenge for current AI systems attempting to understand embodied environments. Multimodal Large Language Models (MLLMs) have emerged as powerful tools, demonstrating impressive capabilities on static image and video understanding tasks. Their ability to process and reason about visual and textual information makes them promising candidates for robotics and embodied AI applications. However, success in complex, dynamic environments requires more than recognizing objects or actions in isolated frames; it demands robust embodied spatial reasoning. This entails building and maintaining persistent internal models of 3D scenes—akin to human cognitive maps—that accurately encode spatial relationships and object identities even as they move in and out of view during navigation. To investigate the extent to which current MLLMs possess these capabilities, we introduce REM ( Reasoning over Embodied Multi-Frame Trajectories ). REM is a novel benchmark specifically designed to evaluate embodied spatial reasoning using multi-frame visual sequences simulating egocentric navigation within synthetic 3D environments of controlled complexity. Unlike benchmarks focusing solely on static images ( 10 ; 11 ; 19 , e.g.,) or complex, uncontrolled video scenarios ( 22 ; 15 , e.g.,) , REM allows for systematic probing of fundamental spatial understanding across changing viewpoints. It assesses critical capacities including object permanence (tested via numerical counting across frames), spatial relationships (tracking left/right positioning), temporal ordering (understanding appearance sequence), and the ability to integrate visual context with movement history for object identification. Our evaluation using REM reveals significant shortcomings in state-of-the-art MLLMs. While newer models show impressive capability in reasoning over simpler scenes, as shown in our ’baseline’ dataset, performance degrades rapidly with increased complexity. Furthermore, models fundamentally struggle with object permanence and distinction, particularly under challenging viewpoint shifts, as demonstrated in our ’Full Rotation’ experiments where performance typically collapses. These findings underscore the specific, deep challenges MLLMs face in developing robust spatial representations from sequential visual input, highlighting the benchmark’s utility in driving targeted improvements for embodied AI. Figure 1: REM at a glance . Left: top-down plot showing object distribution and camera trajectory. Center: egocentric views from selected frames, simulating an agent’s perception during navigation. Right: example question-answer pairs that test different aspects of spatial reasoning: counting, comparison, temporal ordering, and left/right relative positioning. 2 REM : Reasoning over Embodied Multi-Frame Trajectories Summarized in Table 1 , we introduce REM , a set of three Blender-generated egocentric datasets designed to quantitatively evaluate multimodal large language models’ abilities in embodied visuospatial reasoning, specifically focusing on object permanence and distinction, spatial relationships, and accuracy in counting and numerical object comparisons. Each dataset consists of trajectories composed of sequential egocentric image frames. Crucially, the discrete camera action taken between consecutive frames (e.g., ‘move forward 1m’, ‘rotate right 15^{\circ} ’) is explicitly provided alongside the image sequence as input to the model, simulating an agent aware of its own movements. These trajectories occur within simple synthetic environments containing objects of visually distinct shapes ( cuboids, spheres, cones ) and colors ( brown, yellow, red, green, blue, purple, black, orange ). These environments are intentionally made to be simple, with easily distinguishable object shapes and colors, as REM aims to diagnostically pinpoint failures and performance scaling behavior in visuospatial reasoning and memory, not object detection or visual distinction. REM consists of three datasets— Baseline , Single Frame , and Full Rotation . The Baseline dataset benchmarks general capabilities across carefully varied task complexities; Single Frame isolates single-image counting ability as a control; and Full Rotation specifically probes object permanence/distinction and contextual reasoning under challenging viewpoint changes. 2.1 Baseline dataset We introduce our baseline dataset as the foundation of REM , designed to comprehensively evaluate MLLM performance across four visuospatial reasoning tasks. It contains nearly 50,000 question-answer pairs distributed across over 3,000 embodied image-action trajectories. These trajectories systematically vary three key dimensions: • Trajectory Length : Sequences contain either 2, 4, 8, 16, 32, or 64 image frames with associated actions, to evaluate how performance scales with increasing context length and scene views. • Scene Congestion : Environments contain varying object densities, with either 8, 16, 24, 36, or 48 total objects to test performance under different visual complexity conditions. • Object Duplication : Scenes range from all-unique objects to mostly duplicated objects (as few as two distinct types), testing identical object individuation and tracking. Trajectory lengths are distributed uniformly, while trajectory object densities and total duplicate counts can be found in Figure 10 of the Appendix. The baseline dataset includes four primary question categories, meant to probe different aspects of visuospatial reasoning: • Object Counting : “How many blue objects are there?” • Comparison : “Are there more red cones, spheres, or equal?” • Relative Positioning : ‘‘Is the green sphere to the left or right of the blue cuboid?’’ 1 1 1 Importantly, this question is only asked in trajectories where objects maintain their left/right positioning throughout all appearances in the trajectory. • Temporal Ordering : “Did we see the purple cone before, after, or same time as the orange cuboid?” Each trajectory is automatically generated to include a variety of movement patterns, with randomized sequences of both forward movements and rotations. We vary both the number of consecutive forward movements before rotating (randomly left or right) and how many consecutive 15^{\circ} rotations occur in a single direction, ranging from single 15^{\circ} turns to complete 180^{\circ} (or greater) viewpoint changes, as permitted by the remaining trajectory length.

Read the complete text