LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li · 2026

Read LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models by Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09712.txt.

Introduction

Tian and seven colleagues present LAST, a framework that sharpens spatial reasoning in multimodal language models. Data-driven scaling struggles to instill geometric priors, and reading diverse tool outputs poses a second hurdle. LAST-Box, an interactive sandbox, turns varied tool calls into atomic instructions and reusable spatial skills, giving back annotated images and text. A three-stage training path moves from understanding tool outputs to adaptive invocation. On four datasets, LAST-7B gains around 20 percent over its backbone and beats strong proprietary models.

Catalog record

CREATORShi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44505
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09712

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li Abstract . Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: the difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks. 1. Introduction Spatial reasoning ability ( 2 ; 38 ) is a foundational capability for intelligent systems to perceive, plan, and execute tasks in the physical world. Whether in robot navigation ( 13 ) , embodied AI interaction ( 24 ) , or complex 3D scene understanding ( 19 ) , models are required not only to recognize objects but also to accurately parse their geometric layouts and topological relationships. Recently, multimodal large language models (MLLMs) have made significant strides in general visual understanding and semantic reasoning ( 11 ; 1 ) . However, these models often struggle with tasks involving precise spatial layouts, scale estimation, and fine-grained relationship reasoning. Existing research indicates that MLLMs frequently suffer from hallucinations regarding spatial relations ( 17 ) or fail to comprehend complex perspective and depth information ( 4 ) , which severely limits their potential in real-world physical scenarios and practical applications. To mitigate these limitations, existing studies predominantly rely on data-driven scaling, i.e., fine-tuning models with increasing amounts of spatially related data ( 18 ; 2 ; 17 ) . However, for spatial reasoning, these strategies often fail to enable models to truly internalize structured geometric priors and spatial constraints. In contrast, tool-augmented learning presents a more compelling alternative ( 42 ; 27 ) . Given that many fundamental spatial perception skills, such as depth estimation ( 39 ) and precise object localization ( 23 ) , have already been effectively addressed by specialized vision models, a natural paradigm shift is to relieve MLLMs from relearning these mature capabilities and instead equip them with the ability to invoke expert tools. By integrating these highly optimized visual perception models, MLLMs can directly compensate for their perceptual deficiencies in complex spatial reasoning tasks. (a) Comparison of localization accuracy between MLLMs and specialized models. Light bars denote [email protected] and dark bars denote [email protected]. Bar chart comparing [email protected] and [email protected] localization accuracy of Qwen2.5-VL-3B, Qwen2.5-VL-7B, and GroundingDINO. GroundingDINO achieves the highest accuracy under both metrics. (b) Effects of different categories of tool hints across diverse tasks. Grouped bar chart showing performance changes from baseline when using text hints, image hints, or both hints across four spatial task types. Figure 1 . Visualization of preliminary experimental results for problem analysis. However, in the domain of spatial reasoning, this paradigm encounters two distinctive challenges. Firstly, spatial perception tools are heterogeneous and parameter-rich, requiring careful configuration at each step. Errors in tool invocation can easily accumulate over long reasoning horizons, making robust multi-step tool usage by MLLMs difficult. Secondly, perception tools produce diverse low-level outputs, such as segmentation masks or depth maps, which are not readily interpretable by MLLMs. As a result, models struggle to understand these outputs and convert them into high-level spatial cues required for complex reasoning. To address these challenges, we propose LAST ( L everaging Tools as Hin t s), a unified framework for tool-augmented spatial reasoning. LAST introduces an extensible interactive sandbox, the LAST-Box, which standardizes heterogeneous vision models into atomic instructions that can be composed into higher-level spatial skills. These skills shorten the tool invocation horizon and encapsulate low-level tool outputs into model-interpretable textual and visual cues, enabling MLLMs to acquire perceptual capabilities in an intuitive and controlled manner. To support effective tool interaction, we adopt a progressive curriculum-based training framework that guides models from understanding tool outputs to proficient tool invocation. Experiments on four datasets show that LAST-7B trained under this framework achieves around 20% performance gains over the backbone and outperforms strong proprietary closed-source LLMs on complex spatial reasoning tasks. Our main contributions are summarized as follows: • Atomic-to-Skill Tool Abstraction. We introduce a skill abstraction framework that composes atomic tool instructions into reusable spatial skills, substantially shortening long-horizon tool-calling trajectories. • Progressive Tool-Aware Training. We propose a progressive training paradigm with a tool-aware warm-up stage, enabling models to better internalize tool semantics before learning complex instruction following. • Open-Source 7B Model with Strong Performance. We release an open-source 7B multimodal model that integrates our sandbox and training paradigm, achieving state-of-the-art performance on multiple spatial reasoning benchmarks. 2. Preliminary Analysis Observation 1: MLLMs still fall short of specialized models in fine-grained perception. To examine the differences between general-purpose MLLMs and specialized vision systems in foundational spatial reasoning, we conduct a comparative study on the grounding subtask. We sample 200 instances from the SpatialLadder training set and benchmark the open-source Qwen2.5-VL series against the proprietary GroundingDINO. As shown in Fig. 1(a) , although scaling MLLMs leads to gradual improvements under both relaxed ([email protected]) and strict ([email protected]) metrics, their performance remains notably inferior to GroundingDINO, especially at higher localization precision. Under the strict IoU threshold of 0.7, GroundingDINO achieves 44.2% accuracy, whereas Qwen2.5-VL-7B attains only 33.0%, revealing an 11.2% performance gap. Moreover, the limited gain from scaling the model from 3B to 7B (only +1.5% in [email protected]) indicates that parameter scaling alone is insufficient for MLLMs to acquire precise geometric grounding. These results highlight the persistent advantage of specialized vision models in fine-grained spatial perception and suggest the potential advantages of a tool-augmented paradigm that leverages their strengths. Observation 2: Even with tool-based prompts, MLLMs struggle to effectively recognize and exploit such guidance across diverse spatial tasks. Given the distinct perceptual advantages of specialized models, we further investigated whether directly incorporating their outputs as “hints” could consistently enhance the spatial reasoning capabilities of MLLMs. To explore this, we sampled 100 instances from each subset of SPbench, covering four distinct task categories: two involving multiple-choice reasoning (Relative Direction and Relative Distance) and two requiring numerical estimation (Absolute Distance and Size Estimation). We utilized expert models, specifically GroundingDINO and SAM, to preprocess the images, formatting their predictions into either textual descriptions (text-hint) or visual overlays (image-hint). These tool-augmented inputs were then fed into the model alongside the original images for evaluation. Figure 2 .

Read the complete text