ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models Tingshu Mou, Jiabo He, Renying Wang, Ce Liu, Hao Yang, Tiehua Zhang, Jingjing Chen, Xingjun Ma Abstract Recent advances in Multi-modal Large Language Models (MLLMs) target 3D spatial intelligence, yet the progress has been largely driven by post-training on curated benchmarks, leaving the inference-time approach relatively underexplored. In this paper, we take a training-free perspective and introduce ViSRA , a human-aligned Vi deo-based S patial R easoning A gent, as a framework to probe the spatial reasoning mechanism of MLLMs. ViSRA elicits spatial reasoning in a modular and extensible manner by leveraging explicit spatial information from expert models, enabling a plug-and-play flexible paradigm. ViSRA offers two key advantages: (1) human-aligned and transferable 3D understanding rather than task-specific overfitting; and (2) no post-training computational cost along with heavy manual curation of spatial reasoning datasets. Experimental results demonstrate consistent improvement across a set of MLLMs on both existing benchmarks and unseen 3D spatial reasoning tasks, with ViSRA outperforming baselines by up to a 15.6 % 15.6\% and 28.9 % 28.9\% absolute margin respectively. † † footnotetext: ∗ Equal contribution. † † footnotetext: † The work was completed during Tingshu’s internship at BCAI. † † footnotetext: ‡ Corresponding author (xingjunma@fudan.edu.cn). 1 Introduction Multi-modal Large Language Models (MLLMs) have recently shown impressive progress in following instructions grounded in visual inputs [ 4 , 46 , 10 , 25 , 30 ] , which naturally raises a broader question: can these models develop genuine 3D-centric spatial intelligence? Spatial reasoning in 3D scenes—e.g., understanding relative directions and distances among objects, requires more than visual perception and semantic awareness [ 41 ] . It demands building and manipulating an intermediate representation of space that is consistent across viewpoints and time. This is particularly critical for real-world 3D scenes (e.g., videos), where the model must maintain object permanence and track evolving relations under camera motion with human-aligned perceptual routines. Despite impressive performance on 2D tasks, current MLLMs struggle to essentially understand the 3D space and perform poorly on existing benchmarks. Figure 1: Comparison of three paradigms for 3D spatial reasoning. We evaluate ViSRA against the pre-trained base MLLM (e.g., Qwen-2.5VL) and the post-training method (e.g., Spatial-MLLM). Left: three paradigms for 3D spatial reasoning. Middle: qualitative examples showing that the base model can fail on both established and unseen questions, the post-trained model can succeed on established but fail on unseen questions, while ViSRA succeeds on both. Right: quantitative comparison across established and unseen tasks, where ViSRA achieves the best overall performance. A common strategy to address the above problem is to post-train MLLMs on spatial datasets via supervised fine-tuning, architectural modifications, instruction tuning, or preference-based optimization [ 3 , 43 , 5 , 23 , 39 , 13 ] . While this line of works can boost accuracy on curated spatial suites, it also introduces several concerns. Firstly, the post-training pipeline involves manual collection and curation of large-scale video reasoning datasets and comes with high computational cost. More fundamentally, reliance on existing spatial datasets can encourage benchmark-specific overfitting, producing apparent improvements that may not transfer to out-of-distribution (OOD) problems [ 44 , 27 ] , as shown in Figure 1 (middle). In real-world scenarios which are often far more complex than curated benchmark tasks, such methods may neither reliably solve the problem nor provide cues that align with human reasoning. To address these issues, we propose ViSRA , an inference-time, human-aligned Vi deo-based S patial R easoning A gent that enhances MLLMs’ spatial reasoning via the dynamic, modular integration of state-of-the-art visual perception tools and a multi-role framework. Specifically, we design seven types of tools that leverage domain-expert models to extract 2D and 3D object information, as well as scene geometry. ViSRA then adopts a four-role agent framework to exploit spatial information through planning, a reflection–execution loop, and final summarization. This design yields two main benefits: (i) it directly inherits continual advances in perception models without additional post-training; and (ii) it is potentially a generalizable and human-aligned agent framework, where intermediate tracked objects, reconstructed geometry, and derived relations serve as explicit spatial cues for answering questions. We evaluate on VSI-Bench [ 41 ] across multiple base MLLMs and observe consistent gains by up to 15.6\% in spatial intelligence. We further introduce VSI-Bench-Extra to assess generalization to unseen questions, where ViSRA consistently surpasses benchmark-specific post-trained models and outperforms base MLLMs by up to 28.9\% . To conclude, our contributions are as follows: • We empirically show that current MLLMs often lack reliable 3D understanding and struggle with robust and generalizable 3D spatial reasoning, particularly under distribution shifts. • We introduce ViSRA, an inference-time video-based spatial reasoning agent that enhances the spatial intelligence of existing MLLMs. ViSRA dynamically leverages continually improving perception models to construct explicit spatio-temporal evidence in a human-aligned manner, achieving strong performance without post-training on spatial benchmarks. • We further observe that ViSRA performs superiorly on unseen spatial tasks, indicating promising generalization potential to OOD challenges and real-world corner cases. 2 Related Work Figure 2: Performance comparison across problem types on VSI-Bench (a subset of 779 questions). Qwen2.5-VL-7B yields a drop on six question types given ground-truth(GT) cognitive maps. Figure 3: A comparison example. Qwen3-VL-8B succeeds to answer a spatial question with the source video but outputs the wrong answer with the summarized cognitive map. 2.1 MLLMs for Video Understanding MLLMs have made substantial progress in video understanding, exhibiting strong capability in modeling high-level semantics and temporal dynamics from video inputs. Early video MLLMs [ 47 , 14 , 31 , 26 ] typically extended image-based vision and language models to videos by combining a pre-trained visual encoder with a language model, and by scaling video and text alignment as well as instruction tuning on large datasets. Subsequent works further improved training and inference strategies, leveraging emerging techniques such as structured reasoning and reinforcement learning. For instance, Video-R1 [ 15 ] adapted reinforcement learning to video understanding and proposed T-GRPO to better exploit temporal information and enhance reasoning over long videos. MotionEpic [ 14 ] incorporated a spatio-temporal Scene Graph (STSG) as structured input and output and introduced a Video-of-Thought inference framework, enabling fine-grained video understanding and grounding. VideoAgent [ 12 ] proposed a multi-modal agentic framework that improved explainability and generalization by automatically invoking pre-defined tools. Despite these advances, existing video MLLMs remained primarily optimized for semantic video understanding and often underperformed on video-based spatial reasoning tasks. 2.2 Spatial Reasoning in MLLMs We have witnessed growing interest in spatial reasoning for MLLMs in recent years, accompanied by the emergence of benchmarks that systematically evaluated video-based spatial intelligence [ 41 , 24 , 40 , 27 ] . To our best knowledge, VSI-Bench was first proposed as a video-based visual-spatial intelligence benchmark to probe MLLMs’ perceptual, linguistic, and temporal capabilities on spatial reasoning tasks [ 41 ] . Following benchmarks, such as STI-Bench [ 24 ] and MMSI-Video-Bench [ 27 ] , were also designed to evaluate MLLMs’ spatio-temporal understanding through challenging tasks, revealing MLLMs’ limitations in real-world spatio-temporal understanding, ranging from spatial construction and motion understanding to planning, estimation, prediction, and cross-video reasoning. To endow MLLMs with stronger spatial intelligence, a bunch of works adopted post-training with explicit spatial encoders [ 16 , 39 , 7 , 13 , 48 ] and large-scale spatially grounded data [ 43 , 5 , 23 ] . Spatial-MLLM [ 39 ] integrated VGGT [ 37 ] into a dual-encoder design, exploiting the spatial priors provided by geometry foundation models. GS-Reasoner [ 7 ] targeted 3D visual grounding and spatial understanding via a dedicated fusion mechanism for geometric and semantic features, together with a Grounded Chain-of-Thought (GCoT) dataset that provided spatial reasoning traces. On the data side, VST-P and VST-R [ 43 ] scaled spatial supervision with a million-level dataset for enhancing spatial perception and a 135K-sample dataset for spatial reasoning instruction. Cai et al. [ 5 ] further scaled spatial training data to eight million diverse samples and reported strong performance across a broad range of spatial intelligence benchmarks. In addition to post-training, training-free methods [ 29 , 32 ] enhanced spatial reasoning by injecting signals from spatial expert models. GPT4Scene [ 32 ] reconstructed scenes and rendered Bird’s Eye View (BEV) images as inputs to facilitate spatial reasoning. Coarse Correspondences [ 29 ] leveraged tracking models (e.g., Tracking Anything [ 42 ] ) to associate objects across frames and fed object-marked frames to MLLMs, improving spatio-temporal reasoning without task-specific fine-tuning. While there have been a few training-free works, a human-aligned agent that improves MLLMs’ spatial reasoning in a generalizable and explicit way is still underexplored. Figure 4: Overview of ViSRA. The left panel summarizes spatial tools that produce accurate intermediate predictions, while the right panel illustrates the multi-role agent framework that orchestrates these tools to solve spatial queries. 3 Proposed Approach In this section, we first present observations and analyses of current MLLMs on spatial reasoning in Section 3.1 , and then introduce our approach from Section 3.2 to Section 3.4 . 3.1 Limitations of Current Approaches At the beginning of our exploration, we wondered what could enhance the spatial intelligence of MLLMs. 41 suggested that cognitive maps represented internal layouts of environments, serving as a potentially interpretable scaffold for improving spatial reasoning. We thus conducted an experiment on a subset of VSI-Bench [ 41 ] by providing each scene’s ground-truth (GT) cognitive map (generation details in Appendix F ) as an additional input. However, results in Figure 2 show that cognitive maps were largely ineffective in practice across most task types. More surprisingly, performance even dropped on tasks that humans would typically regard as well supported by a map, such as inferring the relative direction among multiple objects. A qualitative example in Figure 3 highlights this limitation: the model answered correctly only with the original video, but failed once the cognitive map was provided. This indicates that existing MLLMs can miss basic spatial relations that are straightforward for humans, thus failing to own human-aligned capability. As mentioned in Section 2 , post-trained models (e.g., Spatial-MLLM [ 39 ] ) achieved strong performance on VSI-Bench, but they were unable to produce human-aligned spatial reasoning cues. This raised the concern that such improvements might reflect benchmark-specific adaptation rather than transferable spatial understanding. To probe generalization, we constructed VSI-Bench-Extra by extending VSI-Bench with the same video sources and introducing three additional question types to evaluate MLLMs on out-of-distribution (OOD) spatial questions. As shown in Figure 1 (bottom right), post-trained models did not outperform the baselines, revealing that current post-training methods did not deliver generalizable 3D spatial reasoning. Overall, it suggests two takeaways: (i) existing MLLMs still lack human-aligned capability of spatial reasoning; and (ii) post-training methods primarily boost in-benchmark performance, but yield negligible OOD generalization. 3.2 Overview of Our Agentic Approach Motivated by the above explorations and observations, we propose ViSRA , an inference-time Vi deo-based S patial R easoning A gent framework in a human-aligned manner. As shown in Figure 4 , ViSRA consists of (i) a suite of spatial tools and (ii) a multi-role agent architecture. Spatial tools leverage expert models from relevant domains to extract visual and geometric cues. Given video and question prompts, ViSRA produces human-aligned spatial reasoning by orchestrating tool calls and aggregating their results in four roles. We describe relevant spatial tools in Section 3.3 and the agent design in Section 3.4 . 3.3 Spatial Tools As illustrated in Figure 4 , we implement a suite of spatial tools that can be invoked by ViSRA. These tools expose object-centric temporal dynamics and scene-centric geometric structures, enabling ViSRA to retrieve where an object appears over time and spatial relationships among objects in a consistent 3D space. Concretely, the tool suite includes 2D object detection, cross-frame object tracking, 3D object detection, scene modeling, knowledge retrieval, video/image query, and other utility tools for spatial reasoning. Figure 5: An inference example. ViSRA answers a relative-distance question by using four roles and invoking multiple spatial tools. 2D object detection. When answering video-based spatial questions about objects, a natural first step is to identify key frames in which the target objects appear and localize their image regions. This tool performs per-frame object detection on key frames, treating each detection as a view of the target object. Given a set of object queries from a question, the MLLM first performs frame filtering by prompting each sampled frame with “Does this frame contain the following objects: …?” to select candidate key frames. An image detection model (e.g., Rex-omni [ 19 ] ) is then applied to the selected frames to generate bounding boxes (bboxes) for target objects. The tool returns the indices of the selected frames along with corresponding bboxes. Cross-frame object tracking. Humans track an object across time using appearance cues such as shape and color. Following this intuition, we apply a tracking model (e.g., Segment Anything Model (SAM) [ 6 ] ) to propagate 2D bboxes from the detection tool across frames. This produces tracklets that associate multiple views of the same object instance throughout the video. 3D object detection. Beyond appearance cues, humans also rely on geometric consistency to recognize object instances and distinguish their locations. To provide such spatial cues, we use a foundational geometric model (e.g., VGGT [ 37 ] ) for object-level 3D detection, mapping 2D pixels inside each detected bbox to 3D points, and computing the 3D center of each view as the centroid of these 3D points. We then cluster different views of the same object instances based on Euclidean distances between their 3D centers. Details of our proposed clustering algorithm are provided in Appendix C.1 . Camera information for each frame can also be calculated for better 3D understanding. Scene modeling. Humans form a coherent scene modeling mentally while watching a video. Although the foundational geometric model reconstructs geometry as point clouds, it does not explicitly provide a stable ground-plane reference. We therefore estimate the ground plane via Random Sample Consensus (RANSAC) [ 17 ] . Concretely, we detect “floor” regions using the 2D detection tool and lift the corresponding pixels to 3D using the 3D detection tool. We then fit a plane to these 3D points via RANSAC and average the estimated plane parameters across frames to obtain a robust ground-plane estimate. The plane normal defines the vertical axis, and we set the positive direction (i.e., “up”) as the half-space containing the majority of reconstructed points. This yields a real-world-aligned scene coordinate frame, enabling view transformations such as the bird’s-eye-view (BEV) rendering. Knowledge retrieval. Some estimation questions require spatial priors beyond visual evidence. To address this, we introduce a knowledge retrieval procedure that provides object- and scene-level size statistics. We compile a knowledge file with more than 500 entries of common objects and rooms from real-world sources, each with size statistics and a brief description. Given the question, the MLLM retrieves the five most relevant entries and uses them as additional priors to answer otherwise ambiguous estimation queries. This human-aligned design also enables us to understand the role of spatial priors in estimation tasks. Video/Image query. The agent can also directly query the video (or selected key frames) via the MLLM. This tool allows the agent to use specific prompts to either answer the question directly or acquire complementary cues that are not explicitly exposed by other tools. Other tools. To facilitate downstream reasoning over the extracted cues and relationships, we implement multiple lightweight utility function tools that the agent can call to compute distances, relative directions, heights, and obstructions, etc, with details available in Appendix C.3 . 3.4 Multi-Role Agent Framework: ViSRA Algorithm 1 Multi-role agent inference with spatial tools 1: Input: Question q ; video v ; tool set \mathcal{T} ; tool schemas \Sigma ; call budget B 2: Output: Final answer a 3: Notation: t\in\mathcal{T} denotes a selected tool; \mathbf{x} denotes tool arguments under \Sigma ; c=(t,\mathbf{x}) denotes a tool call; r denotes the raw tool output; e denotes its natural-language interpretation; and \mathcal{C} denotes the call chain. 4: Roles: Planner reads \Sigma and q to produce a tool-selection plan \pi . Reflector verifies \mathcal{C} and decides whether to stop or call. Executor executes c by invoking t on v and interprets raw outputs. Summarizer consolidates \mathcal{C} to produce the final a . 5: \pi\leftarrow\textbf{Planner}(q,\mathcal{T},\Sigma) 6: n\leftarrow 0;\ \mathcal{C}\leftarrow\emptyset 7: while n\varepsilon then 29: break 30: end if 31: Let C_{i} and C_{j} be the clusters in \mathcal{C} containing p_{i} and p_{j} , respectively 32: if C_{i}=C_{j} then 33: continue \triangleright already in the same cluster 34: end if 35: \mathcal{F}(C_{i})\leftarrow\bigcup_{p\in C_{i}}\mathcal{F}(p) and \mathcal{F}(C_{j})\leftarrow\bigcup_{p\in C_{j}}\mathcal{F}(p) 36: if \mathcal{F}(C_{i})\cap\mathcal{F}(C_{j})\neq\emptyset then 37: continue \triangleright no shared frames within one instance 38: end if 39: C\leftarrow C_{i}\cup C_{j} 40: \mathcal{C}\leftarrow(\mathcal{C}\setminus\{C_{i},C_{j}\})\cup\{C\} 41: end for 42: Convert clusters to instances over original views. 43: \mathcal{I}\leftarrow\{\bigcup_{p\in C}\mathcal{U}(p)\mid C\in\mathcal{C}\} 44: return \mathcal{I} Setting (Frames) Tool-call (s) Agent (s) Total (s) Avg. (%) 64 19.96 20.18 40.14 52.2 32 11.06 19.61 30.67 50.4 16 6.02 20.68 26.70 46.7 Table 6: Latency and Accuracy per Question for Qwen2.5-VL-7B on a Single A100. Tool-call indicates amortized tool execution time; agent denotes framework overhead. Avg. is the VSI-Bench accuracy (%). Appendix A Limitations Limitations. Despite our initial exploration of the inference-time and human-aligned agent for enhancing the spatial intelligence of MLLMs, substantial opportunities remain for future work. In our experiments, the tool set is fixed and relatively limited; a natural next step is to systematically study how tool scale, diversity, and capability trade-offs influence performance and generalization. Moreover, compared with direct question answering, agent-driven tool invocation can introduce nontrivial latency, underscoring the need for more efficient agent designs. We will show the inference latency of our method statistically in Section C.2 . Negative societal impacts. Improved video-based spatial reasoning may raise privacy and safety concerns. The ability to infer object locations and spatial relations from videos could be misused for surveillance or tracking. Moreover, in embodied or safety-critical applications, incorrect spatial reasoning caused by imperfect perception tools may lead to unsafe decisions. Careful validation, privacy safeguards, and human oversight are therefore necessary before real-world deployment. Appendix B Existing Assets and Licenses We use publicly available benchmarks, models, and tool components only for research evaluation. All existing assets are credited through their original papers or official repositories, including VSI-Bench, ViewSpatial-Bench, OST-Bench, MMSI-Video-Bench, Qwen2.5-VL, InternVL, LLaVA-OneVision, Spatial-MLLM, Qwen3-VL, VLM-3R, GS-Reasoner, Rex-Omni, Grounding-DINO, and SAM series models. We follow the corresponding licenses and terms of use of these assets, and do not redistribute their data, model weights, or code unless permitted by their original licenses. Appendix C Additional Method Details C.1 Constrained Greedy Clustering We detail the Constrained Greedy (CG) clustering algorithm used in our 3D object detection tool. A key constraint is that an object category cannot yield two distinct views from the same frame within one instance; therefore, we enforce a hard frame-disjoint constraint during clustering. Meanwhile, we assume that views of the same physical object tend to have smaller 3D-center distances than views from different objects. Based on both the key constraint and the solid assumption, we adopt a CG clustering procedure (Algorithm 2 ): we enumerate all point pairs, sort them by Euclidean distance between their 3D centers pairs, and greedily merge the two clusters associated with a pair if (i) the distance is below \varepsilon , (ii) the two points are not already in the same cluster, and (iii) the two clusters have no overlapping frames; otherwise, the pair is skipped. We further incorporate an optional tracking prior \mathcal{T} as an initialization of \mathcal{V} . If \mathcal{T} partitions \mathcal{V} into several track groups, we pre-merge each group into a single point whose 3D center is the mean of member 3D centers, while its frames and 2D bounding boxes (bboxes) are stored as sets; the CG then proceeds over these merged points. The above algorithm is illustrated for one object category, and we run it across all categories in a loop. C.2 Inference Latency Table 6 reports the average per-question inference latency under different frame-sampling configurations. As shown, a moderate increase in total latency, from 26.70 s to 40.14 s, leads to a substantial improvement in accuracy, from 46.7\% to 52.2\% . These results indicate that this acceptable trade-off enables our system to achieve training-free adaptability, provide explicit spatial evidence, and seamlessly benefit from advances in stronger perception tools without requiring retraining. C.3 Details of ViSRA Agent prompts. For each agent role in ViSRA—planner, reflector, executor, and summarizer—we craft a dedicated prompt to guide the MLLM to perform its designated function. The prompts are listed below: Prompt for Planner Role. You are a Planner agent that answers video-based spatial questions. Input. You will receive only: (i) video_path , (ii) question , and (iii) available tool JSON schemas. Core requirements. • Decide which tools are needed based on the schemas; do not decide concrete numeric hyperparameters, and focus only on the tool chain. • Return a JSON object with three fields: plan (a list of tool names in execution order), tool_chain (a list of objects of the form {"tool": "tool_name"} with no arguments), and information (a list of required information types). Decision rules (evaluated in order). • Appearance / temporal questions: use 2D detection and sort by frame index. Set information=["2d_views","appearance_sorting"] and plan=["tool_2d_object_detection"] . • Object attributes: for object size, dimensions, or material, use knowledge retrieval. Set information=["visual_description","retrieved_knowledge"] and plan=["tool_knowledge_retrieval"] . • Numerical scene estimation (critical priority): if the question asks for room area or absolute distance with explicit units (e.g., meters or square meters), use the direct video query tool. Set information=["visual_estimation_response"] and plan=["tool_video_image_query"] . Override rule: even if the wording says “Measuring from the closest point”, if a physical unit is requested, use this tool rather than the 3D calculation tool. • General visual VQA: for general description, color, action, or atmosphere, use tool_video_image_query . • 3D spatial reasoning / comparisons: if the question involves spatial layout, counting, or relative comparison (and is not captured by the previous rule), include ["2d_views","3d_center","3d_bbox"] . For relative-distance comparisons, use ["tool_2d_object_detection", "tool_object_3d_detection", "tool_calculate_distance"] with information=["closest_point_distance"] ; for relative direction, use ["tool_2d_object_detection", "tool_object_3d_detection", "tool_calculate_direction"] with information=["relative_direction"] ; otherwise (counting or general layout), use ["tool_2d_object_detection", "tool_object_3d_detection"] . Output. Return JSON only. Prompt for Reflector (Step-by-step Controller) Role. You are an execution agent that iterates step-by-step. Input. A specific instruction for the current step, only the JSON schema for that tool, a PLAN (high-level reference), a HISTORY list of prior tool calls/outputs, and the current video_path and question . Task. • Analyze the previous tool output (if any) in the analysis field. • Execute the current-step instruction. If instructed to call a tool, output action="call_tool" and set tool to the exact name given. Strict rules. • Use only arguments explicitly defined in the provided TOOL_SCHEMA . • Strictly follow behavioral hints, logic, and constraints described in the tool docstring/description. • Do not invent arguments that are not in the schema, and do not reuse arguments from previous steps if they do not belong to the current tool. • For video_path , use the exact provided string. • For output_2d or tool_3d_output , do not repeat the full heavy content from history; instead pass an empty object {} or empty list [] , which the system will inject automatically. • If instructed to finalize, output action="final" . Output format (strict). Return exactly one JSON object with the following keys: { "analysis": "...", "action": "call_tool" | "final", "tool": "...", // if action="call_tool" "args": {...}, // if action="call_tool" "final_answer": "..." // if action="final" } Prompt for Executor (Tool Result Interpreter) Role. You are an evidence interpreter (Executor) for a video spatial reasoning agent. Input. You will be given the tool JSON schema, tool input arguments, tool output result, and the original question (context only). Task. • Generate exactly one natural-language paragraph that describes only the tool output. • Focus on the video content in the returned result (objects, frames, locations, 3D centers). • Cite concrete evidence using exact keys/values from the output (e.g., frames=[...] , bbox entries, 3d_center values). • Do not answer the original question. Output format (strict). Return exactly one JSON object and no extra text: { "result_description": "..." } Prompt for Summarizer (Final Answer Integrator) Role. You are a Summarizer agent. Task. • Analyze evidence, do not narrate process: ignore the history of which tools were called; focus strictly on tool_output and result_description ; extract actual values such as 3D coordinates, calculated distances, direction strings, and object counts; then perform explicit logical comparisons over these values to reach the conclusion. • Extract the final answer: for multiple-choice questions, final_answer must contain only the option letter; for numerical questions, it must contain only the digit(s). Do not include explanation, punctuation, or extra words in final_answer . Output format (strict). Return exactly one JSON object and no extra text: { "summary": "...", "final_answer": "..." } Description of the tools. Tool descriptions are an important part for constructing tool schema, as they guide the agent to select the appropriate tool at each step. We specify each tool’s input, output, and the information it provides. Detailed descriptions are given below: Tool Descriptions 2D Object Detection Tool ( tool_2d_object_detection ) IMPORTANT: • The returned detections are category-level views collected across frames and are not instance-aware; do not use this tool for counting. • When generating the objects argument, the agent must perform noun-phrase extraction over the question and must include the reference object (anchor object) together with the target objects. Use this tool for the following question types: • Appr. Order. (first-time appearance order) • Rough 2D position questions (left/right in the image) • As the mandatory prerequisite for tool_object_3d_detection Special object-parsing examples: • “Which object is closest to the TV?” \rightarrow objects="chair, sofa, stool, tv" . • “Is the tv to the left of the sofa?” \rightarrow objects="tv, sofa" . Inputs: • video_path : str . Local video file path (prefer absolute). • objects : str . Comma-separated object category names. Include both the target objects and the reference object. Output: • A dictionary whose keys are object categories and whose values contain the collected views: { "chair": { "views": [...] }, "table": { "views": [...] } } 3D Object Detection Tool ( tool_object_3d_detection ) IMPORTANT: • The output is a list of physical instances ; unlike the 2D tool, this tool can be used for counting. • Instances are sorted by visibility ( member_count ); index _1 denotes the most prominent instance of a category. Usage order: • Use after tool_2d_object_detection . • using_tracking=True is required for object counting, because tracking and clustering are used to prevent over-counting caused by camera motion or view changes. • aligned_scene=True should be enabled for horizontal/vertical relationships, height, front/back/left/right direction, or absolute layout. Use this tool for the following question types: • Object counting • Relative distance, using 3d_center for Euclidean comparison • Relative direction, using 3d_center together with aligned_scene=True • Route planning Inputs: • video_path : str . Local video file path (prefer absolute). • output_2d : the exact dictionary returned by tool_2d_object_detection(...) . • using_tracking : bool . Default False . Set to True for counting. • aligned_scene : bool . Default False . Set to True for direction/relation tasks that require a real-world frame. Output: • A list of instance dictionaries in the following format: [ { "instance_id": "chair_1", "category": "chair", "3d_center": [x, y, z], "bbox_3d": [x1, y1, z1, x2, y2, z2] }, { "instance_id": "chair_2", "category": "chair", "3d_center": [...], "bbox_3d": [...] } ] Knowledge Retrieval Tool ( tool_knowledge_retrieval ) Purpose: • Retrieve object-/room-level size statistics and descriptions as spatial priors for estimation tasks. When to use: • For underdetermined estimation questions (e.g., object size estimation, room size estimation) where priors beyond visual evidence are needed. Inputs: • query (string): question text or distilled caption. • top_k (int, default: 5): number of entries to return. Output: • Top- k retrieved entries (e.g., k=5 ), where each entry is a single str containing size statistics and a short description, in the following format: { "query": "...", "top_k": 5, "entries": [ "entry string 1", "entry string 2", ... ] } Video/Image Query Tool ( tool_video_image_query ) Purpose: • Directly query the video (or a selected key frame) via the MLLM, returning a natural-language response . • Acquire complementary cues that are not explicitly exposed by other tools. When to use: • For general description, visual attributes, action recognition, and estimation-style questions that need direct semantic or metric cues beyond structured 2D/3D outputs. Inputs: • video_path (string): local video file path (prefer absolute). • prompt (string): the specific query prompt. • query_type (string, default: "video" ): one of {"video","image"} . • frame_idx (int, default: -1 ): required when query_type="image" , specifying the queried frame index. Critical prompt templates. For room-size and absolute-distance estimation, the tool description in tool.py requires the agent to use the following exact prompt templates (replacing {question} with the original question): 1. Room size / area estimation prompt = ( "You are an expert at estimating room size (area) from videos.\n" "Use the visual information in the video to answer the user’s question.\n" "Return ONLY a single best numerical estimate (integer or decimal) in square meters.\n" "Output format (STRICT): NUMBER\n" "- Do NOT output units.\n" "- Do NOT output a range.\n" "- Do NOT output any explanation.\n\n" f"QUESTION: {question}\n" ) 2. Distance estimation (between two objects) prompt = ( "You are an expert at estimating REAL-WORLD distance between two objects from videos.\n" "Use the visual information in the video to answer the user’s question.\n" "Return ONLY a single best numerical estimate (integer or decimal) in meters.\n" "Output format (STRICT): NUMBER\n" "- Do NOT output units.\n" "- Do NOT output a range.\n" "- Do NOT output any explanation.\n\n" f"QUESTION: {question}\n" ) An analogous pair of templates is used in tool_multi_image_query , with “from videos” replaced by “from multiple images”. Output: • A dict containing a direct natural-language response : { "video_path": "...", "query_type": "video" | "image", "frame_idx": , "prompt": "...", "response": "natural language text" } Distance Computation Tool ( tool_calculate_distance ) Purpose: • Calculate Euclidean distance between the 3D centers of object instances. • The resulting distances are relative to the scene scale rather than guaranteed metric distances. Inputs: • tool_3d_output : the list returned by tool_object_3d_detection(...) . • video_path : absolute path to the video. • reference_instance : str . It must be a specific instance ID, such as tv_1 , rather than a category name. • target_instances : str . A comma-separated list of specific instance IDs. Output: { "video_path": "...", "result": { "reference_instance": "tv_1", "target_instances": ["chair_1", "stool_1", "sofa_1", "stove_1"], "distances": {"chair_1": 0.27, ...}, "unit": "relative" } } Relative Direction Computation Tool ( tool_calculate_direction ) Purpose: • Compute the relative direction of a target object under an egocentric reference frame defined by a standing location ( stand ) and a facing direction ( face ). When to use: • For relative-direction questions once 3D centers are available (optionally stabilized by scene modeling). Inputs: • tool_3d_output : the list returned by tool_object_3d_detection(...) . • video_path : absolute path to the video. • stand_instance : str . A specific instance ID, e.g., stove_1 . If multiple same-category objects exist, the implementation prefers the most prominent instance (suffix _1 ) unless the question implies otherwise. • face_instance : str . A specific instance ID, e.g., sofa_1 . • target_instance : str . A specific instance ID, e.g., tv_1 . Output: { "video_path": "...", "result": { "stand_instance": "stove_1", "face_instance": "sofa_1", "target_instance": "tv_1", "direction": "front-left|front-right|back-left|back-right", "evidence": {...} } } Height Comparison Tool ( tool_compare_height ) Purpose: • Compare heights of two objects by comparing their z -axis values. IMPORTANT: • Requires an aligned scene : 3D outputs must be produced after calling tool_scene_modeling so that the z -axis corresponds to real-world vertical direction. Inputs: • tool_3d_output : the list returned by tool_object_3d_detection(...) with alignment applied. • video_path : must match tool_3d_output["video_path"] . • instance_a : str . MUST be an INSTANCE NAME . • instance_b : str . MUST be an INSTANCE NAME . Output: { "video_path": "...", "result": { "instance_a": "stool_1", "instance_b": "table_1", "z_a": 1.23, "z_b": 0.87, "relation": "a_higher|b_higher|equal" } } Object Obstruction Checking Tool ( tool_calculate_obstruction ) Purpose: • Check whether an obstruction object lies on the route (straight-line segment) from a source object to a destination object, using 3D centers. Inputs: • tool_3d_output : the list returned by tool_object_3d_detection(...) . • video_path : must match tool_3d_output["video_path"] . • source_instance : str . MUST be an INSTANCE NAME . • destination_instance : str . MUST be an INSTANCE NAME . • obstruction_instance : str . MUST be an INSTANCE NAME . Output: { "video_path": "...", "result": { "source_instance": "table_1", "destination_instance": "sofa_1", "obstruction_instance": "stool_1", "is_obstruction": true, "evidence": { "source_center": {"x":..., "y":..., "z":...}, "destination_center": {"x":..., "y":..., "z":...}, "obstruction_center": {"x":..., "y":..., "z":...}, "t": 0.42, "closest_point": {"x":..., "y":..., "z":...}, "distance_to_segment": 0.18, "threshold": 0.25 } } } Appendix D Details of VSI-Bench-Extra As mentioned in Section 3.1 and 4.3 , we design three new question types to evaluate MLLMs’ out-of-distribution (OOD) spatial reasoning capability: relative direction backward , object obstruction , and relative distance farthest . Relative direction backward modifies VSI-Bench relative-direction prompt from ‘facing …” to ‘with my back to …,” and correspondingly flips the ground-truth (GT) answer. Object obstruction asks whether (and which) an object obstructs the route from object A to object B. Relative distance farthest asks for the farthest object, in contrast to the nearest-object formulation used in VSI-Bench. For relative direction backward , we directly reuse the medium and hard subsets of VSI-Bench relative-direction questions, rewrite the questions, and reverse the answers as ground truth, resulting in 751 questions. For object obstruction and relative distance farthest , we follow the VSI-Bench construction pipeline: we leverage GT bounding boxes from the scan datasets ARKitScenes, ScanNet, and ScanNet++, filter out ambiguous annotations, and generate questions using templates with human verification. This yields 400 object obstruction questions and 444 relative distance farthest questions. The templates used to generate these three types are as follows: Relative Direction Backward If I am standing by the {positioning object} and with my back to the {orienting object} (facing directly away from it), is the {querying object} to my front-left, front-right, back-left, or back-right? Directions refer to the quadrants of a Cartesian plane (assuming I am standing at the origin and facing the positive y-axis). Object Obstruction If I am standing by the {positioning object} and facing the {orienting object} , which object is there as an obstruction when I walk straight to the {orienting object} ? Relative Distance Farthest Measuring from the closest point of each object, which of these objects ( {choice a} , {choice b} , {choice c} , {choice d} ) is the farthest from the {category} ? Appendix E Qualitative Experimental Results Here we present additional qualitative results, including both successful and failed cases, together with their explicit intermediate spatial representations shown from Figure 6 to 9 . ViSRA is able to produce correct answers while generating human-interpretable intermediate reasoning outputs by invoking spatial tools. However, it may also fail due to limitations of the underlying expert models (e.g., detectors), as illustrated in Figure 9 . Figure 6: An example of ViSRA solving an object-counting question correctly. Figure 7: An example of ViSRA solving an appearance-order question correctly. Figure 8: An example of ViSRA solving a relative-direction question correctly. Figure 9: An example of ViSRA solving an object-counting question incorrectly due to misdetecting a stool as a table. Appendix F Generation of Ground-truth Cognitive Maps In Section 3.1 , we constructed experiments to evaluate the ineffectiveness of cognitive maps for spatial intelligence. Specifically, we used GT 3D object centers and bboxes in 3D datasets to generate 81 cognitive maps for a subset of 779 questions. We employed a 10×10 grid with normalized scene length while maintaining the x–y aspect ratio for a scene, with the JSON format adhering to the VSI-Bench specification [ 41 ] . In the visualization, objects mapped to the same coordinate were displayed with lateral offsets to prevent overlap (Figure 10 ). Figure 10: Visualized and textual examples of ground-truth cognitive maps generated from 3D annotations. The three on the left panel are from ARKitScenes, and the three on the right panel are from ScanNet. Figure 11: Two examples in VSI-Bench where ambiguous referring expressions lead to ambiguous answers. Appendix G Limitations of Existing Benchmarks Existing benchmarks can contain unsure or wrong answers in their datasets. For example, the problem 3565 in VSI-Bench asks "If I am standing by the backpack and facing the door, … ", which is unclear from the video (The upper half of Figure 11 ) as a person can stand by either the left or right of the backpack. Therefore, both B and C appear plausible, but the GT answer is C. Another example is that the problem 1343 involves multi-instance categories (e.g., multiple sofas and tables), leading to different answers depending on which instance is selected. (The lower half of Figure 11 ). In conclusion, we believe that most established spatial reasoning benchmarks can have similar quality issues unless each question-answer pair together with the video are manually inspected very carefully.