GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning

Deshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang · 2026

Read GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning by Deshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22558.txt.

Introduction

Miao and others argue that vision language models need shape detail matched to each image piece, not one shared shape signal. Their GeoWeaver system builds a layered bank of shape facts from a frozen shape encoder. Each visual token then pulls the shape facts that fit its own spatial role. The chosen facts join the token through a small add on step before language work starts. Tests on spatial reasoning tasks show steady gains while broad multimodal skills stay intact.

Catalog record

CREATORDeshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54547
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning Deshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang Abstract Spatio-temporal reasoning in vision-language models requires visual representations that preserve physical geometry rather than merely semantic appearance. Recent multimodal models incorporate geometric information through structural branches, 3D-aware supervision, reasoning-stage fusion, or long-horizon memory. While these approaches demonstrate the importance of geometry for spatial intelligence, they typically treat geometric cues as a shared signal across all visual tokens. We note that this overlooks a finer-grained challenge: different visual tokens require different geometric evidence depending on their spatial roles. To address this limitation, we introduce GeoWeaver , a pre-reasoning geometric grounding framework that treats geometry as a representational prerequisite for spatio-temporal reasoning. GeoWeaver constructs a multi-level geometry bank from a frozen geometry encoder and performs token-adaptive geometric evidence allocation, enabling each visual token to retrieve the most relevant geometric abstractions. The selected evidence is incorporated into visual tokens via a residual grounding operation prior to language modeling, yielding geometry-grounded representations for downstream reasoning. Extensive evaluations on spatial reasoning benchmarks demonstrate that GeoWeaver consistently enhances geometry-aware reasoning while retaining general multimodal capabilities. This indicates that geometric information yields the greatest benefit not as a late-fusion auxiliary signal but as a fundamental prerequisite that shapes the representational foundation on which large language models perform reasoning. All source code and models will be released at https://github.com/yahooo-m/GeoWeaver . (a) Comparison between pre-fusion methods, LLM-fusion methods, and our pre-reasoning grounding paradigm. (b) Visualization of VGGT feature responses from different layers. Figure 1 : Motivation and paradigm comparison of GeoWeaver. Top: Existing geometry-enhanced VLMs mainly introduce geometry through pre-fusion or LLM-side fusion, while GeoWeaver grounds visual tokens before language reasoning. Bottom: VGGT feature maps from different layers exhibit heterogeneous spatial responses, indicating that multi-layer geometry does not provide a uniform signal. This motivates our design of treating geometry features as a multi-level evidence bank. 1 Introduction Understanding and reasoning about physical space is a fundamental capability for intelligent systems that must perceive, communicate, and act in the real world. Recent advances in large vision-language models (VLMs) significantly improve general visual understanding, enabling strong performance in image and video question answering, instruction follow-up, and multimodal reasoning. However, reliable spatio-temporal reasoning remains a major challenge. Tasks involving relative distance estimation, object layout understanding, temporal tracking of spatial relations, or grounding motion and direction in dynamic scenes continue to expose systematic failures that cannot be explained by language reasoning alone. To address this limitation, recent studies increasingly incorporate spatial information into multimodal models, progressing from perception-side enhancement toward reasoning-oriented spatial modeling. As shown in Figure 1 , early efforts strengthen visual representation by introducing explicit spatial or structural branches, allowing geometry-aware cues to complement semantics-dominated visual features 49 ; 62 . Building on this direction, representation-level approaches further adopt 3D-aware supervision or geometry-oriented objectives to improve the structural sensitivity of visual tokens, suggesting that spatial reasoning depends not only on input design but also on the geometry encoded in intermediate representations. More recent geometry-enhanced MLLMs investigate how structural cues interact with language reasoning, either by injecting geometric features into the multimodal reasoning stack or by aligning spatial signals with hierarchical reasoning stages 11 ; 25 . Beyond single-instance reasoning, long-horizon spatial cognition extends the problem to continuous video streams, where predictive sensing, memory organization, and event-centric modeling support temporally persistent spatial understanding 56 . Despite these advances, existing approaches still largely treat geometry as an auxiliary signal introduced at specific stages of the pipeline, rather than as a prerequisite that shapes visual representations before language reasoning begins. This perspective leaves the representational role of geometry underexplored. We note that spatio-temporal reasoning depends not only on the availability of geometric cues, but also on whether visual tokens are geometry-grounded before entering the language model. Without such grounding, the decoder must jointly resolve visual geometry alignment and high-level relational reasoning. This entanglement pushes a perceptual grounding problem into the reasoning stack, reducing the reliability of spatial reasoning and blurring the division between perception and cognition. Motivated by this observation, we formulate spatio-temporal reasoning as a problem of pre-reasoning geometric grounding . Instead of injecting geometry after visual tokens are formed, we use geometry to shape representations before language decoding. Specifically, a frozen geometry encoder constructs a multi-level geometry bank that serves as a reservoir of structural evidence. Each visual token queries this bank and retrieves the geometric abstractions most relevant to its spatial role. The retrieved evidence is then injected into the visual representation, producing geometry-grounded visual tokens for the language model. This formulation shifts visual–geometric alignment from the reasoning stack to the perceptual interface, allowing the decoder to focus on relational and temporal reasoning over geometry-grounded representations. We instantiate this formulation through a token-adaptive evidence allocation mechanism. The mechanism aligns multi-level geometry tokens with visual tokens, predicts routing weights over geometry layers for each visual token, and applies sparse routing to select compact geometric evidence. As a result, the model avoids uniform exposure to all geometry layers and reduces interference from redundant structural signals. Extensive experiments show that GeoWeaver performs favorably against strong spatial reasoning models across multiple benchmarks while preserving general multimodal capability. Ablation studies further validate the importance of reasoning-relevant geometry layers and sparse evidence allocation, supporting the effectiveness of geometry-grounded visual representations for spatio-temporal reasoning. The main contributions of this work are: • We revisit spatio-temporal reasoning from a representational perspective and identify the lack of geometry-grounded visual tokens as a key limitation of current VLMs. • We propose GeoWeaver , a geometric grounding framework that allocates token-specific geometric evidence from a multi-level geometry bank prior to language decoding. • We demonstrate consistent improvements on geometry-sensitive spatial reasoning benchmarks and show through ablations that both geometry-bank construction and evidence allocation are critical to the proposed grounding process. 2 Related Work Spatial Intelligence in MLLMs. Recent large multimodal models (MLLMs) 28 ; 2 ; 46 ; 27 show strong general-purpose visual understanding, but spatial reasoning in physical environments remains challenging. Recent benchmarks, including VSI-Bench 53 , SPAR 59 , BLINK 14 , and Cambrian-S 56 , evaluate this limitation through tasks involving depth, viewpoint, topology, and spatiotemporal relations 37 ; 15 ; 64 . These studies indicate that spatial intelligence is not merely a difficult subset of generic multimodal reasoning. Instead, it requires visual representations that preserve geometry-sensitive evidence beyond semantics-dominated appearance cues. Our work follows this diagnosis, but focuses on the representation before reasoning: whether visual tokens are geometrically grounded when they enter the language model. Data Scaling and Spatial Reasoning Supervision. Much effort improves spatial reasoning through data construction and specialized optimization. SpatialVLM 6 , SPAR 59 , and Cambrian-S 56 enlarge the supervision space with synthetic 3D scenes, physically grounded tasks, and long-horizon spatial cognition settings. Furthermore, VST 55 , SpaceR 39 , MindCube 57 , SpatialLadder 25 , and Spatial-SSRL 33 exploit structured tuning, reinforcement learning, and verifiable rewards to strengthen spatial reasoning behaviors. SenseNova-SI 4 further suggests that scaling text-based chain-of-thought alone brings limited gains, highlighting that spatial reasoning depends on more than language-side deliberation. These methods provide stronger supervision and optimization signals, while our work addresses a complementary question: how the

Read the complete text