LASAR: Towards Spatio-temporal Reasoning with Latent Cognitive Map
Jinzhou Tang, Sidi Liu, Waikit Xiu, Weixing Chen, Keze Wang · 2026
Read LASAR: Towards Spatio-temporal Reasoning with Latent Cognitive Map by Jinzhou Tang, Sidi Liu, Waikit Xiu, Weixing Chen, Keze Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16899.txt.
Introduction
Tang's team asks whether agents build spatial models or merely mimic expert paths. Action and thinking tasks share a flaw: no learning signal forces coding of spatial links over broken runs. Their LASAR design uses a dual memory holding episode runs and a meaning map. A contrast goal, Spatio-temporal Contextual Representation Learning, builds sample pairs from queries made through labeled time context in sim. Tests show 2 to 3.5% gains in zero-shot transfer on VLN-CE and VSI-Bench, plus high self-match of the map.
Catalog record
| CREATOR | Jinzhou Tang, Sidi Liu, Waikit Xiu, Weixing Chen, Keze Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65782 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16899
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LASAR: Towards Spatio-temporal Reasoning with Latent Cognitive Map Jinzhou Tang, Sidi Liu, Waikit Xiu, Weixing Chen, Keze Wang Abstract A fundamental challenge in embodied AI is verifying if agents build internal models of spatial structure or merely learn to mimic task-specific expert trajectories. This is critical as foundational approaches rooted in action-centric tasks (e.g., VLN) and reasoning-centric tasks (e.g., EQA) often share a common limitation: they lack a learning signal that forces them to encode fine-grained spatial relationships (like topology or distance) over long-range, fragmented experiences. To address this, we first propose LASAR, an architecture featuring a dual-memory system designed to maintain both episodic experiences and a semantic cognitive map. We then introduce Spatio-temporal Contextual Representation Learning (ST-CRL), a contrastive objective designed to train this architecture. ST-CRL leverages spatio-temporal cues from cognitive queries generated through annotated spatio-temporal context in simulation to build sample pairs, thereby forming the internal cognitive map from the agent’s experiences. Experiments demonstrate that our method achieves 2%-3.5% gains in both zero-shot generalization on standard VLN-CE and VSI-Bench benchmarks. We also demonstrate that our proposed cognitive map has high self-consistency. 1 Introduction A fundamental challenge in Embodied AI 30 is to build agents that move beyond mimicking superficial policies and instead develop a robust, generalizable understanding of their 3D world 62 ; 4 ; 10 ; 13 ; 59 . Current research has progressed along a spectrum spanning action-centric tasks and reasoning-centric tasks. At one end, Vision-Language Navigation (VLN) 2 challenges an agent to execute a sequence of actions to reach a goal based on instructions, thus prioritizing decision-making and action. At the other, Embodied Question Answering (EQA) 11 requires an agent to answer complex questions about an environment, thereby focusing on understanding and reasoning 33 ; 61 ; 16 ; 48 . However, the traditional distinction between “action” and “reasoning” highlights critical challenges. On the action-centric side, it remains unclear whether agents trained on massive vision-language-action pairs 12 truly understand space or merely overfit to superficial statistical biases within expert trajectories 20 . Many studies confirm that imitation learning alone is insufficient for this deep understanding 27 . Similarly, on the reasoning-centric side, techniques ( e.g. , CoT 44 ) that rely on large language models’ linguistic priors also fail at complex spatial tasks 50 , as their abstract reasoning is detached from a grounded world model, which formal work demonstrates is essential for robust understanding and prediction 14 . These discoveries highlight the need for a spatial-oriented world model to comprehend the environment. While recent work has largely focused on bridging this gap by combining VLN and EQA tasks 60 ; 58 ; 53 , it remains largely unexplored how this combination can be leveraged to form temporally consistent, high-level spatial representations ( e.g. , topology, distance, and relations) that enable robust generalization across novel environments and instructions. To this end, we argue that learning a cognitive map 6 ; 46 , which acts as the module that converts raw experience (i.e., a fragmented stream of { observation, action } pairs) into a queriable world model 3 ; 42 , is a promising approach to building the more robust, high-level spatial logic required for spatio-temporal reasoning. To address this, we introduce LASAR ( LA ten S p A tial R easoner), an LLM-based agent designed to maintain two distinct forms of memory: (1) an episodic memory that stores the sampled temporal sequence of visual-spatial features from the agent’s past observations, and (2) a semantic memory that forms a structured latent cognitive map by querying a learnable codebook of world primitives (termed ‘Spatial Semantic Atlas’) using the agent’s episodic memory. These two memory systems interact dynamically: the episodic memory provides temporal context, while the semantic memory provides spatial relations. A unified LLM backbone reasons over both memories to integrate task queries, ground task-oriented objects, and generate linguistic responses for downstream tasks. To ensure this latent map encodes robust spatial logic, we propose our core innovation, Spatio-temporal Contextual Representation Learning (ST-CRL), to train it. ST-CRL is a contrastive objective that shapes the semantic memory by requiring the agent to answer challenging cognitive queries. These queries are enabled by our proposed MindCraft task, which introduces Q&A concurrently with navigation. We generate the corresponding data via a procedural pipeline built on VLN-CE that injects retrospective, introspective, and prospective queries. ST-CRL then leverages these queries and the simulator’s privileged information to build high-level spatial-semantic contrastive sample pairs, enabling the training of a generalizable cognitive map for downstream tasks. Experimental results show a 3.5% absolute improvement on the VSI-Bench zero-shot reasoning benchmark and improve SPL, SR on VLN-CE by 2%-5% over SOTA baselines. Our main contributions are threefold: (1) We propose LASAR, a novel LLM-based agent reasons in space through a latent cognitive map. (2) We propose ST-CRL, an objective to sculpt this cognitive map, distilling spatio-temporal logical structure from fragmented embodied experiences. (3) We introduce MindCraft, a task framework and related dataset that unifies action and reasoning by injecting a concurrent stream of cognitive queries into navigation. 2 Related Work Embodied Navigation and Reasoning Early embodied agents leveraged reinforcement learning (RL) 43 ; 7 ; 24 to acquire navigation skills, with policy optimization driven by interactions with the environment. The emergence of Visual-Language Navigation (VLN) 26 ; 57 represents a significant advancement, enabling agents to follow natural language instructions, thereby providing greater flexibility and generalization in task descriptions. With the development of large multimodal models, the field of Embodied Question Answering (EQA) 28 ; 51 ; 29 has emerged in parallel, aiming to assess an agent’s understanding of its environment, particularly focusing on spatial perception and reasoning abilities. However, for a long time, the technical solutions for both areas have relied on specialized model architectures and optimization pathways, failing to form effective synergies. To address this issue, integrating the long-term action planning of VLN with the deep reasoning capabilities of EQA can create a framework that simultaneously supports instruction navigation and environmental understanding reasoning, ultimately breaking down technical barriers and achieving a deep fusion of both capabilities. Memory and Internal Representations To succeed in long-horizon tasks, an agent requires robust memory. One line of work focuses on explicit memory, using techniques like Simultaneous Localization and Mapping (SLAM) to build precise 3D geometric maps of the environment 22 ; 21 ; 34 . An alternative is implicit memory 41 ; 19 , where recurrent or transformer-based models encode an agent’s history into a compact state vector 15 . More recently, research 45 has begun to explore the implicit cognitive maps that emerge within large multimodal models. A persistent challenge across these paradigms is the difficulty of forming a globally consistent representation from a stream of egocentric views. Indeed, recent benchmarks have quantified this issue, revealing that even powerful models excel at local spatial awareness but struggle with long-range spatial relationships, which suggests a need for novel methods to better guide the construction of these internal maps. Figure 1 : Overview of LASAR. Observations are processed by 2D and 3D Encoders and merged via Geometric-Semantic Fusion for better spatial understanding. These features populate a dual-memory system: (1) a high-fidelity Episodic Memory storing the temporal visual history, and (2) a structured Latent Cognitive