RieMind: Geometry-Grounded Spatial Agent for Scene Understanding

Fernando Ropero, Erkin Turkoz, Daniel Matos, Junqing Du, Antonio Ruiz, Yanfeng Zhang, Lu Liu, Mingwei Sun, Yongliang Wang · 2026

Read RieMind: Geometry-Grounded Spatial Agent for Scene Understanding by Fernando Ropero, Erkin Turkoz, Daniel Matos, Junqing Du, Antonio Ruiz, Yanfeng Zhang, Lu Liu, Mingwei Sun, Yongliang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15386.txt.

Introduction

Ropero and co-authors present RieMind, an agent that reasons about indoor scenes through an explicit three-dimensional scene graph. Vision-language models still stumble on metric and spatial questions. RieMind splits perception from reasoning: a module builds the graph, and the agent queries it with geometric tools. Tools report sizes, distances, poses, and relations between objects. With graphs built from ground truth, scores on VSI-Bench beat prior work by up to 16 percent. The agent also topped base vision models by wide margins, with no task-specific tuning.

Catalog record

CREATORFernando Ropero, Erkin Turkoz, Daniel Matos, Junqing Du, Antonio Ruiz, Yanfeng Zhang, Lu Liu, Mingwei Sun, Yongliang Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30477
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15386

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RieMind: Geometry-Grounded Spatial Agent for Scene Understanding Fernando Ropero, Erkin Turkoz, Daniel Matos, Junqing Du, Antonio Ruiz, Yanfeng Zhang, Lu Liu, Mingwei Sun, Yongliang Wang Abstract Visual Language Models (VLMs) have increasingly become the main paradigm for understanding indoor scenes, but they still struggle with metric and spatial reasoning. Current approaches rely on end-to-end video understanding or large-scale spatial question answering fine-tuning, inherently coupling perception and reasoning. In this paper, we investigate whether decoupling perception and reasoning leads to improved spatial reasoning. We propose an agentic framework for static 3D indoor scene reasoning that grounds an LLM in an explicit 3D scene graph (3DSG). Rather than ingesting videos directly, each scene is represented as a persistent 3DSG constructed by a dedicated perception module. To isolate reasoning performance, we instantiate the 3DSG from ground-truth annotations. The agent interacts with the scene exclusively through structured geometric tools that expose fundamental properties such as object dimensions, distances, poses, and spatial relationships. The results we obtain on the static split of VSI-Bench provide an upper bound under ideal perceptual conditions on the spatial reasoning performance, and we find that it is significantly higher than previous works, by up to 16%, without task specific fine-tuning. Compared to base VLMs, our agentic variant achieves significantly better performance, with average improvements between 33% to 50%. These findings indicate that explicit geometric grounding substantially improves spatial reasoning performance, and suggest that structured representations offer a compelling alternative to purely end-to-end visual reasoning. Figure 1 : RieMind: Geometry-grounded framework for spatial agentic scene understanding, and in this figure, it is reasoning over the relative direction of objects in a scene, while performing allocentric-egocentric frame transformations. 1 Introduction The rapid rise of large language models (LLMs) and their linguistic and reasoning capabilities has left behind a reduced focus on spatial intelligence and understanding. This gap has then been filled by the development of visual language models (VLMs), that are able to concurrently understand visual information, and natural language. This multimodality has seen an ever increasing focus these past few years [ 29 ] , though some challenges remain concerning their spatial reasoning and understanding capabilities [ 4 , 19 ] . These challenges become more severe when long-term visual and spatial context is required, such as long-form videos, showing a degradation in performance, leading to hallucinations [ 22 , 18 ] . Current VLMs lack robust, and grounded spatial understanding. Spatial understanding is currently often gauged by question answering (QA) on either image or video benchmarks [ 5 , 27 ] . These benchmarks range from 2D image relational spatial understanding [ 5 ] , to 3D indoor spatial reasoning requiring metric information, such as distances, sizes, and geometries, usually utilising videos as inputs [ 27 ] . The two main approaches to indoor spatial understanding rely on, either large-scale spatial instruction fine-tuning of VLMs [ 9 , 16 , 14 , 26 , 21 , 30 ] , or tool-augmented reasoning [ 25 , 7 ] , where VLMs can reason by utilising geometric tools, such as depth, frames, bounding boxes, and others. However, both these approaches rely on a tight coupling between perception and implicit model representations, which can limit spatial metric consistency and interpretability. This is specifically important for 3D indoor environments, where reasoning should be grounded in persistent geometric representations. This paper focuses on 3D indoor spatial understanding. While related works often frame this problem as a video understanding one, i.e . using a VLM to ingest videos and then apply them to QA benchmarks, we take a different conceptual approach. Indoor environments are often static in nature, hence we consider the input videos as a data ingestion and perception modality and not the basis for spatial reasoning. Therefore, we represent each scene using a 3D scene graph (3DSG) that is able to unify semantic, topological, and metric scene representations, providing a unified understanding of inter-object and intra-scene relationships [ 13 , 23 , 20 ] . These 3DSGs are often used in robotics where precise geometric and spatial information are required for task planning, exploration, and object manipulation [ 17 , 3 , 12 ] . Building on this representation, we propose RieMind, an agentic framework that decouples perception from reasoning. Instead of using a VLM, we employ an LLM that invokes structured geometric tools and queries the 3DSG directly, retrieving explicit geometric information such as locations, distances, poses, dimensions, areas, volumes, and semantic relationships. The 3DSG is typically built from a separate perception module that performs instance segmentation followed by scene graph construction. In this paper, we assess an agent’s spatial reasoning capabilities under ideal perceptual conditions by constructing the 3DSG from ground-truth annotations, eliminating errors introduced by upstream segmentation. This allows our framework to be agnostic to the upstream perception pipeline, as the scene information is abstracted and accessible exclusively through tool calls. We show that explicit geometric grounding yields improved spatial reasoning performance on VSI-Bench [ 27 ] when compared to previous approaches. The contributions of this paper are as follows: • We cast 3D indoor spatial understanding as a reasoning process over explicit geometric scene representations, by building an agentic, geometric, 3DSG-grounded framework that allows for interpretable metric and spatial reasoning for LLMs. • We demonstrate state-of-the-art results on VSI-Bench, with an average increase of 16% over competing fine-tuned spatial understanding models, and average improvements between 33% and 50% when comparing a base VLM with our agentic version. 2 Related Work VLMs for Indoor Spatial QA. Indoor spatial understanding through QA has seen a surge of datasets and benchmarks, the most popular of which has been VSI-Bench [ 27 ] . Several works have improved the performance of base VLMs through spatial instruction QA fine-tuning or reinforcement learning, e.g . ViCA [ 9 ] , SpaceR [ 16 ] , and others [ 14 , 26 , 21 , 30 ] . Other research directions include adding geometric priors or auxiliary representations into VLMs [ 31 , 24 ] . Our work is somewhat complementary to this, as it is still focused on the same type of spatial understanding, but instead of the end-to-end implicit learning and reasoning, we allow for an LLM to ground its reasoning through tools that query an explicit geometric representation of a scene in the form of a 3DSG. Tool Integrated Spatial Reasoning. A complementary research line focuses on augmenting VLMs with geometric tools to improve spatial reasoning, rather than fine-tuning directly, in order to reduce reliance on implicit priors and improve explanability of the reasoning process. This has been investigated in [ 25 , 11 , 7 ] , and these works demonstrate that spatial understanding of VLMs benefits from access to geometric tools. The SpatialScore benchmark [ 25 ] introduces a tool-using agent, named SpatialAgent, with access to geometric and spatial tools. TIGeR [ 11 ] applies a similar tool based agent but for robotic manipulation instead. MM-Spatial [ 7 ] also introduces tools, specifically depth. When it comes to tooling, these approaches have one thing in common, they rely on geometric estimation, as these tools estimate depth, pose, bounding boxes, and other quantities. This is where our works diverge, as we ground tool

Read the complete text