Beyond Pixels: Introducing Geometric-Semantic World Priors for Video-based Embodied Models via Spatio-temporal Alignment
Jinzhou Tang, Jusheng Zhang, Sidi Liu, Waikit Xiu, Qinhan Lv, Xiying Li · 2025
Read Beyond Pixels: Introducing Geometric-Semantic World Priors for Video-based Embodied Models via Spatio-temporal Alignment by Jinzhou Tang, Jusheng Zhang, Sidi Liu, Waikit Xiu, Qinhan Lv, Xiying Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00210.txt.
Introduction
Jinzhou Tang and five co-authors present VEME in 2025, a method to give video-based embodied models a richer sense of the physical world. Vision-language models read still scenes well, but they reason poorly over time and space in moving, open settings. Tasks such as task-oriented navigation and embodied question answering expose that gap.
VEME aligns objects, spatial representations, and visual semantics with spatio-temporal cues in one cross-modal frame. A dynamic, implicit cognitive map, centered on the agent's own viewpoint and past experience, supports the reasoning. The world model aims to generalize to unseen scenes. Better space-time grounding could help embodied agents act sensibly in new homes and work sites.
Catalog record
| CREATOR | Jinzhou Tang, Jusheng Zhang, Sidi Liu, Waikit Xiu, Qinhan Lv, Xiying Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00210
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Pixels: Introducing Geometric-Semantic World Priors for Video-based Embodied Models via Spatio-temporal Alignment Jinzhou Tang, Jusheng Zhang, Sidi Liu, Waikit Xiu, Qinhan Lv, Xiying Li Abstract Achieving human-like reasoning in deep learning models for complex tasks in unknown environments remains a critical challenge in embodied intelligence. While advanced vision-language models (VLMs) excel in static scene understanding, their limitations in spatio-temporal reasoning and adaptation to dynamic, open-set tasks like task-oriented navigation and embodied question answering (EQA) persist due to inadequate modeling of fine-grained spatio-temporal cues and physical world comprehension. To address this, we propose VEME, a novel cross-modal alignment method that enhances generalization in unseen scenes by learning an ego-centric, experience-centered world model. Our framework integrates three key components: (1) a cross-modal alignment framework bridging objects, spatial representations, and visual semantics with spatio-temporal cues to enhance VLM in-context learning; (2) a dynamic, implicit cognitive map activated by world embedding to enable task-relevant geometric-semantic memory recall; and (3) an instruction-based navigation and reasoning framework leveraging embodied priors for long-term planning and efficient exploration. By embedding geometry-aware spatio-temporal episodic experiences, our method significantly improves reasoning and planning in dynamic environments. Experimental results on VSI-Bench and VLN-CE demonstrate 3%-6% accuracy and exploration efficiency improvement compared to traditional approaches. 1 Introduction Embodied agents navigating unknown environments face a fundamental challenge: how to develop spatial understanding and make navigation decisions with limited visual observations ( Zou et al. 2025 ) , much like humans leverage episodic memories and spatial cognition ( Bermudez-Contreras, Clark, and Wilber 2020 ; Coppolino and Migliore 2023 ; Epstein et al. 2017 ) . While vision-language models (VLMs) have achieved remarkable success in static visual understanding ( Sarch et al. 2023 ) , their direct application to embodied navigation tasks reveals critical limitations in spatio-temporal reasoning and the integration of long-term memory ( Liu et al. 2025 ) . Figure 1: An illustrative comparison of navigation behaviors. (a) Our agent, equipped with a cognitively inspired dual-memory framework, constructs an implicit cognitive map from world embedding, observation history, and past trajectory. This allows it to ground the complex spatial instruction (“kitchen behind the sofa ”) in the environment, devise an efficient path, and successfully reach the goal. (b) In contrast, a conventional agent lacking this spatio-temporal memory and reasoning capability fails to comprehend the spatial relationships, leading to incorrect exploration and task failure. The core challenge lies in the embodied reasoning gap: while VLMs demonstrate strong visual understanding ( Liu et al. 2023 ; Ma et al. 2023 ) , they lack the spatial grounding necessary to translate visual observations into effective linguistic reasoning and navigation decisions ( Shahria et al. 2022 ) . This manifests as difficulties in understanding 3D spatial relationships, building persistent spatial representations, and connecting visual semantics with navigational affordances ( brian ichter et al. 2022 ; Liu et al. 2021 ; Zheng, Huang, and Wang 2025 ) . This limitation manifests in two critical ways: first, agents cannot effectively recall and utilize relevant past experiences when encountering similar spatial contexts ( Park et al. 2023 ) ; second, they struggle to align visual semantics with geometric spatial relationships, resulting in poor spatial reasoning in complex environments ( Yang et al. 2025 ; Yin et al. 2025 ) . Existing approaches attempt to address these limitations through different paradigms. SLAM-based methods construct explicit spatial maps with semantic annotations, but they lack the high-level reasoning capabilities necessary for instruction-following and common-sense spatial reasoning ( Ren et al. 2024 ; Chen et al. 2025 ) . LLM/VLM-based approaches leverage powerful language reasoning but fail to capture fine-grained spatio-temporal dependencies. Recent multimodal approaches show promise but remain limited by inadequate cross-modal alignment between visual observations and spatial representations, preventing effective episodic memory formation and recall ( Huang et al. 2024 ; Cheng et al. 2025b ) . Our key insight is drawn from cognitive neuroscience: human spatial intelligence succeeds through the complementary interaction between episodic memory (specific, spatio-temporal experiences) and semantic memory (general spatial knowledge) ( Moscovitch et al. 2005 ; Nadel and Hardt 2011 ) . We propose that embodied agents can achieve similar capabilities by learning to align visual experiences with spatial semantic representations, enabling both episodic memory formation and context-relevant recall. To address the limitations of VLMs in navigation and embodied Q&A, we propose a cognitively inspired dual-memory framework VEME that integrates spatial semantics and episodic experience through three key components: a spatio-temporal encoder that captures geometry-aware visual observations and action histories fused with a learnable world embedding; a cross-modal alignment mechanism that bridges 2D visual semantics and 3D spatial representations via bidirectional attention and contrastive learning; and a VLM-based decision module that leverages structured input tokens incorporating current perception, spatial priors, episodic cues, and language instructions. This design enables the agent to model long-horizon spatio-temporal contexts and generalize navigation and reasoning behavior across novel environments without requiring explicit mapping. Our framework demonstrates considerable improvements over existing methods on standard benchmarks, including VSI-Bench and VLN-CE, particularly in zero-shot transfer scenarios. The main contributions include: (1) introducing a cognitively-inspired dual memory framework for spatial intelligence; (2) developing effective cross-modal alignment techniques for spatio-temporal reasoning; and (3) validating the approach’s effectiveness in reducing exploration costs while improving task completion rates in unknown environments. 2 Related Work Embodied and Vision-Language Navigation Early embodied agents learned navigation using reinforcement learning (RL) ( Chen et al. 2020 ; Liang et al. 2023 ; Zhang et al. 2025 ; Sheng 2025 ) , but their reliance on extensive environment interactions for policy optimization limited their ability to generalize to new scenes. The subsequent paradigm of Vision-Language Navigation (VLN) ( Anderson et al. 2018 ) enabled agents to follow natural language instructions, offering more flexible task specification. However, a common limitation of these approaches is the lack of a persistent spatial memory, which hinders performance on long and complex routes that require recalling previously visited locations. Memory in Embodied Agents To succeed in long-horizon tasks, an agent requires robust memory ( Zheng et al. 2025 ) . One line of work focuses on explicit memory, using techniques like SLAM to build precise but computationally expensive 3D geometric maps ( Jia et al. 2022 ) . An alternative is implicit memory, where models like RNNs encode history into a compact state vector; this approach is efficient but can struggle to retain critical long-term details ( Hong et al. 2021 ) . Our work addresses the remaining challenge of how to efficiently retrieve the most task-relevant information from an agent’s past experiences. Large Models for Embodied Control Recently, Large Language Models (LLMs) and Vision-Language Models (VLMs) have been leveraged for high-level planning in robotics, capitalizing on their vast common-sense knowledge ( Yu, Kasaei, and Cao 2023 ; Li et al. 2025 ) . Despite their powerful reasoning capabilities, these models often exhibit an “embodied reasoning gap” ( Li et al. 2024b ) . Because they are trained on disembodied internet data, their generated plans can be disconnected from an agent’s physical capabilities and the constraints of a 3D environment. Our work aims to bridge this gap by embedding geometry-aware priors directly into the model’s reasoning process. Cross-Modal Alignment for Embodied Reasoning Effective embodied reasoning requires robust cross-modal alignment