Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures
Risham Sidhu, Julia Hockenmaier · 2026
Read Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures by Risham Sidhu, Julia Hockenmaier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17333.txt.
Introduction
Sidhu and Hockenmaier build GSU, a text-only dataset for testing how well language models reason about space on grids. The tasks cover navigation, finding objects, and building structures from coordinate lists. Models handle basic grid ideas well, they report. They struggle with points of view tied to an embodied agent and with naming 3D shapes from coordinates. Fine-tuning a small model can match large frontier models, they find.
Catalog record
| CREATOR | Risham Sidhu, Julia Hockenmaier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87458 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures Risham Sidhu, Julia Hockenmaier Abstract We introduce GSU, a text-only grid dataset to evaluate the spatial reasoning capabilities of LLMs over 3 core tasks: navigation, object localization, and structure composition. By forgoing visual inputs, isolating spatial reasoning from perception, we show that while most models grasp basic grid concepts, they struggle with frames of reference relative to an embodied agent and identifying 3D shapes from coordinate lists. We also find that exposure to a visual modality does not provide a generalizable understanding of 3D space that VLMs are able to utilize for these tasks. Finally, we show that while the very latest frontier models can solve the provided tasks (though harder variants may still stump them), fully fine-tuning a small LM or LORA fine-tuning a small LLM show potential to match frontier model performance, suggesting an avenue for specialized embodied agents. 1 Introduction With the advent of more powerful LLMs, their applications have broaden from text generation to complex problems ranging from advanced mathematical reasoning to multimodal agents. Many of these tasks require the ability to reason about space, whether to solve geometry problems ( Zhao et al. 2025 ; Gao et al. 2025 ) or to understand verbal commands about their surroundings as an embodied agent ( Zhang et al. 2024 ; Feng et al. 2025 ) , to complete tasks while interacting with objects in complex environments ( Padmakumar et al. 2021 ; Shridhar et al. 2020 ) , to describe visual scenes ( Fu et al. 2024a ) , or to reason over 2- and 3-D images ( Cheng et al. 2024 ; Chen et al. 2024 ) . Approaches to such tasks incorporate additional modalities through vision encoders, projections of point cloud data, or more specialized representations, but using these effectively necessitates updates to base LLM architecture or additional training. Furthermore, even with such additions, it is unclear if fundamentally non-physical models that have been trained primarily on unpaired text can truly understand 3D space. Despite recent advances on various spatial reasoning tasks, the internal reasoning processes of LLMs, VLMs, and variations thereof are often not interpretable, meaning that merely measuring performance is distinct from evaluating understanding. And even when performance is strong, further investigations into how additional modalities, like vision, are used in spatial tasks reveal that multi-modal LLMs receive little additional information from images when provided with rich text representations ( Wang et al. 2024a ) . In that case, textual descriptions of physical space may serve as a convenient proxy for complex visual inputs when evaluating spatial reasoning. By removing visual perceptions, they offer a controlled method of probing whether LLMs posses robust internal representations of space, and if such representations can be manipulated with textual descriptions alone. Given that grids are an established representation for 3D space, they are a natural choice to express 3D layouts and movements. In this work, we present GSU, a dataset composed of 3 simple text-only grid spatial reasoning tasks aimed at applications in embodied domains, to serve as a diagnostic for the inherent spatial reasoning abilities of existing models and identify where further work is needed to induce the spacial capabilities embodied agents require. • The Navigation Task requires instruction generation or following on 2D and 3D grids with two potential spatial frames of reference: Cardinal (i.e. fixed), testing recognition of 1D directional differences in coordiantes, and Egocentric (i.e. moving with an embodied agent), testing understanding of rotating/updating spatial reference frames. • The Object Localization task asks where targets are relative to a viewer ( Egocentric ), testing 2D and 3D direction differences between coordinates, or relative to a separate spatial reference ( Allocentric ), testing spatial reference in collaborative settings. • The Structure Composition task needs models to translate sets of coordinates into basic shapes (e.g. rows, columns, cubes), testing ability to link coordinates to real-world shapes and process large numbers of coordinates. Figure 1: The GSU Dataset tasks and settings visualized for clarity (note that the models do NOT receive these visuals and instead receive a textual version of the environment shown in Appendix E ). The grey arrows indicate the heading, i.e. direction that the embodied agent is facing. In the Navigation task, they remain aligned with +Y in the Cardinal setting and rotate to reflect the last direction of travel in the Egocentric setting, which affects the direction of the second step and the final coordinates. For the Object Localization task, we show how different headings affect the spatial relations between the target and the reference. For the Structure Composition task, we show the 3 structure categories that models may be asked to describe. 2 The GSU Dataset We now expand on each of these relatively simple spatial tasks and the specific facets of textual coordinate reasoning they test. For more details on generation and dataset statistics, please refer to Appendix A . 2.1 The Navigation Task The first task measures models’ understanding of textual coordinates as a representation of physical space. In this task, models must correlate natural language directions (i.e. "left", "down", "backward") with grid traversals across coordinates differing in 1 dimension at a time, i.e. equating moving from (0, 0) to (2, 0), to (2, 1) with moving "right 2" and "forward 1". 2.1.1 Instructor vs. Follower VLN tasks are generally a conversation between two counterparts: an instructor and a follower. While the instructor is often a person and the follower an agent, this is not necessarily always the case (e.g. a collaborative system with multiple agents, Narayan-Chen et al. 2019 ’s MDC Architect task, voice assistants, etc). Models should thus be proficient in both directions of this mapping between verbal spatial directions and coordinate traversals. In the Follower setting, models receive a set of natural language instructions explaining how to traverse the grid (e.g. "move right 2 steps and then forward 1" ) and must output a coordinate reflecting their final location (e.g. (2,1) ). Whereas in the Instructor setting, models receive a grid traversal expressed as a series of intermediate coordinates (e.g. (0,0),(2,0),(2,1) ) and must output the natural language instructions that correlate to that path (e.g. "move right 2 and forward 1" ). 2.1.2 Spatial Frames of Reference We consider two frames of reference for the spatial directions of the grid. In the Cardinal setting, the spatial directions are fixed to the axes of the grid and never change, i.e. to move forward always means to increase the y value, while to move left always means to decrease the x value. While many grid based tasks use this fixed perspective of spatial relations (e.g. Aghzal et al. 2025 ), it does not transfer well to practical collaborative settings where an embodied agent or a human it is instructing have their own frame of reference that may not align with a fixed grid. The Egocentric formulation aligns spatial directions with the perspective of an embodied subject navigating the grid, orienting the spatial directions with their body, and so rotating where they are relative to the fixed axes of the grid. For example, in the Egocentric setting to go right 2 steps