Brain-inspired spatial intelligence for embodied agents Authors: Shouwei Ruan, Liyuan Wang, Caixin Kang, Qihui Zhu, Songming Liu, Xingxing Wei, Hang Su Published: 2026 ABSTRACT AbstractSpatial cognition enables adaptive goal-directed behavior through structured internal models of space. Robust biological systems consolidate spatial knowledge into three interconnected forms: landmarks for salient cues, route knowledge for movement trajectories, and survey knowledge for map-like representations. While recent advances in multi-modal large language models (MLLMs) have enabled visual-language reasoning in embodied agents, these efforts lack structured spatial memory and instead operate reactively, limiting their generalization and adaptability in complex real-world environments. Here we present Brain-inspired Spatial Cognition for Navigation (BSC-Nav), a unified framework for constructing and leveraging structured spatial memory in embodied agents. BSC-Nav builds allocentric cognitive maps from egocentric trajectories and contextual cues, and dynamically retrieves spatial knowledge aligned with semantic goals. Integrated with powerful MLLMs, BSC-Nav achieves state-of-the-art efficacy and efficiency across diverse embodied navigation tasks (e.g., improving success weighted by path length from 17.6% to 44.9% in instance-level navigation and from 42.7% to 53.1% in zero-shot long-horizon instruction-following), while also supporting versatile embodied behaviors in the real physical world. These results highlight a scalable path toward general-purpose spatial intelligence.Subject terms: Computer science, Cognitive neuroscience Introduction Spatial cognition, the ability to acquire, organize, exploit, and update knowledge about external space, is fundamental to both human beings and artificial intelligence (AI)1,2. It underlies not only sensorimotor skills such as navigation and manipulation but also supports higher-level cognitive functions, including abstraction, planning, and reasoning. In humans, generalizable spatial representations support interpretation of sensory input, anticipation of future events, and flexible adaptation to changing environments, from brewing coffee in a familiar kitchen to navigating an unfamiliar city3–5. Given its broad relevance in embodied interaction with the real physical world, spatial cognition has emerged as a central theme across disciplines, driving advances in robotics6, urban simulation7, and planetary-scale modeling8, while increasingly recognized as a foundational component of artificial general intelligence (AGI)3,9,10. Despite the rapid progress in multi-modal large language models (MLLMs) and growing efforts to equip embodied agents with visual-language reasoning11–13, current artificial systems remain fundamentally limited in large-scale spatial cognition14, especially in tasks requiring long-horizon navigation and mobile manipulation. A central bottleneck lies in the lack of structured spatial memory15–17, a mechanism for persistently encoding, organizing, and retrieving spatial knowledge about the environment. Most existing methods, whether based on end-to-end reinforcement learning18–20 or modular pipelines with powerful MLLMs21,22, process observations in a reactive and stateless manner. Without durable internal models of external space, agents struggle to consolidate coherent spatial representations or perform reasoning beyond immediate stimuli, resulting in fragmented knowledge, short-sighted planning, and poor generalization. Addressing these limitations requires a paradigm shift from reactive processing to memory-centric spatial cognition, which supports persistent representations and compositional reasoning over time and space. Compared to artificial systems, biological spatial cognition offers a robust and compelling template. Decades of neuroscience research revealed that organisms consolidate spatial knowledge into three distinct yet interconnected forms16,17 (Fig. 1a): landmarks, which encode stable associations of salient environmental cues to support localization and contextual understanding23,24; route knowledge, which captures egocentric movement trajectories between landmarks for habitual navigation and path integration25; and survey knowledge, which integrates multiple routes into allocentric, map-like representations that support flexible inference, shortcut discovery, and detour planning26. These spatial representations are accessed and coordinated via working memory27, especially visual-spatial working memory28,29, enabling adaptive retrieval, composition, and generalization based on task demands and environmental familiarity. The BSC-Nav framework for cognitive spatial intelligence. a Structured spatial memory in biological brains, comprising landmarks, route knowledge, and survey knowledge. b The BSC-Nav framework instantiates structured spatial memory in embodied agents. Environment observations (RGB-D images and agent poses) are processed by (i) a landmark memory module encodes and retrieves durable associations of multi-modal environmental cues as the landmarks; and (ii) a cognitive map module accumulates and organizes movement trajectories as the route knowledge into allocentric, map-like representations as the survey knowledge. Upon task invocation, (iii) a working memory module dynamically composes relevant spatial knowledge for adaptive planning and reasoning. c Structured spatial memory enables not only universal navigation but also higher-level spatial-aware skills. The human and brain illustrations in (a) were adapted (with added layers and annotations) from SciDraw artwork, licensed under CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Sources: 10.5281/zenodo.3925989,10.5281/zenodo.3926189. Motivated by these biological principles, we introduce Brain-inspired Spatial Cognition for Navigation (BSC-Nav), a unified framework that instantiates cognitive spatial intelligence in embodied agents via structured spatial memory (Fig. 1b). BSC-Nav explicitly constructs spatial knowledge via two synergistic branches: a landmark memory module, which encodes durable associations between salient environmental cues and 3D spatial locations to instantiate landmarks; and a cognitive map module, which accumulates egocentric observations along movement trajectories as route knowledge and then organizes the voxelized trajectories into allocentric, map-like representations as survey knowledge. Compared to previous embodied navigation frameworks confined to localized modeling with a single granularity, such as purely dense grids30,31 or sparse topologies32,33, the hierarchical spatial architecture of BSC-Nav achieves the decoupling of spatial knowledge while incorporating targeted adaptive update mechanisms. This allows spatial knowledge to continuously evolve as the agent explores external environments, overcoming the limitations of singular, static spatial representations. BSC-Nav further incorporates a working memory module that dynamically retrieves and combines spatial representations from landmark memory and the cognitive map, thereby aligning semantic goals with grounded spatial actions. Each component interfaces seamlessly with large-scale foundation models: visual foundation models like DINOv234,35 provide perceptual grounding of environmental cues, while MLLMs like GPT-4V21 guide high-level semantic interpretation and goal-conditioned reasoning. Unlike previous end-to-end action-planning navigation frameworks11,30,36, which are highly susceptible to data distribution shifts between training and testing environments, BSC-Nav leverages the explicit retrieval mechanism of its working memory to achieve a complete decoupling of high-level semantic goal planning from low-level physical motion control, thereby paving the way toward zero-shot, general-purpose embodied navigation. By integrating MLLM-based embodied agents with structured spatial memory, BSC-Nav achieves robust spatial cognition capabilities, supporting long-horizon reasoning, experience reuse, and flexible transitions between local and global policies. BSC-Nav achieves state-of-the-art performance across a broad spectrum of navigation tasks, including object-goal, open-vocabulary, and instance-level navigation, while also demonstrating strong spatial generalization in instruction following, embodied question answering, and mobile manipulation. These results position BSC-Nav as a scalable path that complements MLLMs’ vision-language reasoning with cognitive spatial intelligence, enabling more capable, adaptable, and cognitively informed AI operating in the real physical world. Results We begin by presenting the BSC-Nav framework, which instantiates brain-inspired structured spatial memory in embodied agents. We then systematically evaluate its performance across both simulated and real-world scenarios, highlighting: (i) universal navigation capabilities in foundational tasks; (ii) higher-level spatially aware skills in instruction-driven visual-language navigation and active embodied question answering; and (iii) real-world efficacy in navigation and mobile manipulation. Construction and exploitation of structured spatial memory in embodied agents Structured spatial memory is essential for guiding goal-directed behavior in complex real-world environments3–5, yet remains largely absent from current embodied agents. Drawing inspiration from the robust spatial representations in biological systems, namely, landmarks23,24, route knowledge25, and survey knowledge26, BSC-Nav implements a modular architecture that explicitly constructs and exploits analogous memory structures to achieve spatial cognition capabilities (Fig. 1b, detailed in Methods). BSC-Nav comprises two branches that are continuously updated to accumulate and organize spatial knowledge during environment exploration. First, a landmark memory module encodes salient environmental cues as associative triplets comprising spatial coordinates, semantic categories, and contextual descriptions, consolidating landmarks of the surroundings. This design yields abstract and sparse representations that prioritize salient instances, enabling efficient retrieval and forming a flexible, interpretable scaffold of the external space (detailed in Methods). Meanwhile, a cognitive map module accumulates egocentric observations along movement trajectories as route knowledge, which are then voxelized and organized into persistent allocentric representations as survey knowledge (detailed in Methods). Motivated by the broad idea in free-energy37,38 and predictive-processing39,40 accounts, that biological systems update internal representations in response to mismatches between observation and expectation, we implement a surprise-driven update strategy that selectively integrates novel or unexpected spatial observations. Here, this mechanism refers to a heuristic computational design, rather than a strict implementation of variational free-energy optimization41. To exploit structured spatial memory during task execution, BSC-Nav further incorporates a working memory module that adaptively retrieves landmarks, route knowledge, and survey knowledge for goal localization and trajectory planning (detailed in Methods). Analogous to visual-spatial working memory in biological systems28,29, this module dynamically composes spatial representations based on task demands and environmental familiarity. We propose a hierarchical retrieval strategy guided by the complexity of the goal. For simple targets, the agent leverages MLLMs22,42 in a text-only form to reason over semantic associations and contextual cues within the landmark memory (Fig. 1c). For more complex or ambiguous instructions, BSC-Nav activates association-enhanced retrieval over the cognitive map. In this process, MLLMs enrich the initial goal descriptions by inferring object-specific attributes and scene-level priors, transforming vague commands into semantically detailed representations. These enriched descriptions are then rendered into imagined visual prototypes using a text-to-image diffusion model43. The generated images are encoded and matched against dense visual features stored in the cognitive map to localize candidate target regions. This hierarchical retrieval strategy allows BSC-Nav to access complementary forms of spatial memory in response to target complexities (category-level and instance-level) and modalities (text and image), supporting precise localization for universal goal-directed navigation (Fig. 2). Precise localization via hierarchical retrieval in working memory. a For simple category-level goals, working memory prioritizes retrieval from the landmark memory module for rapid matching and candidate coordinate generation. b For complex instance-level goals, working memory employs association-enhanced retrieval, converting text instructions into visual features to query the cognitive map module. For image goals, the cognitive map is queried directly using extracted visual features. A more detailed illustration of candidate generation, ranking, and verification is provided in Supplementary Fig. 2. Working memory retrieval often produces multiple candidate coordinates, each associated with a confidence score and relative distance. In scenarios with multiple valid targets (e.g., category-level navigation), confidence scores alone may not reliably indicate correctness. Rather than greedily selecting the candidate with the highest confidence, BSC-Nav employs a composite ranking strategy that integrates distance- and confidence-based scores to determine an efficient, adaptive exploration sequence. Candidate targets are then sequentially visited and verified using online visual-semantic observations parsed by the MLLM until the goal is confirmed. A more detailed illustration of candidate generation, ranking, and verification is provided in Supplementary Fig. 1. Low-level movement policies are generated using heuristic planning along this sequence. During execution, MLLMs continuously parse spatial-semantic context from observations, verify landmark proximity, and confirm goal arrival, enabling affordance-aware action generation. Upon reaching the appropriate target locations, agents can invoke additional skills, such as grasping objects or answering questions, to accomplish more complex, task-driven objectives (Fig. 1c). Universal navigation across modalities and granularities Through brain-inspired spatial cognition that incorporates the three types of spatial knowledge, BSC-Nav demonstrates strong generalization across diverse navigation tasks and consistently outperforms recent strong baselines in both success rate and navigation efficiency. We conduct systematic evaluations across 8195 episodes in 62 indoor scenes from physically reconstructed environments (MP3D44 and HM3D45 datasets) in the Habitat simulator46, covering four representative navigation tasks (Fig. 3): Object-Goal Navigation (OGN)18,47,48, Open-Vocabulary Object Navigation (OVON)49, Text-Instance Navigation (TIN)50, and Image-Instance Navigation (IIN)51. In each episode, agents initialized at random positions execute discrete actions (forward 25 cm, turn left 30∘, turn right 30∘, and stop)52. Following standard protocols52, navigation succeeds only when agents execute a stop within 1.0 m of goals. Evaluation metrics include the Success Rate (SR)53, defined as the fraction of completed episodes for efficacy; and the Success weighted by Path Length (SPL)53, which scales the binary success by the ratio of the shortest path distance to the agent’s actual trajectory length for efficiency (detailed in Sec. 4.2). We compare BSC-Nav with end-to-end methods (PixNav11, DAgRL49, PSL50) and modular methods that instantiate only analogous landmark memory (VLFM13, MOD-IIN54, UniGoal33, GOAT36). Figure 3a summarizes the quantitative benchmark results for category-level and instance-level navigation, while Fig. 3b presents a representative example of navigation trajectories and target verification. Goal-oriented multi-modal navigation. a Performance comparison against baseline methods across different navigation tasks. Evaluations encompass category-level tasks, including object goal navigation (OGN) and open-vocabulary object navigation (OVON), as well as instance-level tasks, including text instance navigation (TIN) and image instance navigation (IIN). Quantitative results are averaged across different trial numbers: 2195 and 2000 episodes for OGN on HM3D and MP3D, respectively; 1000 episodes each for OVON, TIN, and IIN. For each task, efficacy and efficiency are quantified by Success Rate (SR, %) and Success weighted by Path Length (SPL, %), respectively, as indicated by the paired bar charts. Source data are provided as a Source Data file. b Visualization of navigation trajectories executed by BSC-Nav, illustrating the agent’s egocentric observations and the target verification process facilitated by MLLM interactions. As shown in Fig. 3a, BSC-Nav demonstrates superior spatial generalization in category-level navigation (OGN and OVON). For common targets in OGN tasks, BSC-Nav achieves 78.5% SR on HM3D (6 categories) and 56.5% on MP3D (20 categories with larger areas), surpassing the state-of-the-art method UniGoal by 24.0% and 15.5%, respectively. Unlike UniGoal, which organizes only landmark memory with abstract goals and scene graphs, BSC-Nav leverages structured spatial memory and thus obtains significant performance gains. In the more challenging OVON tasks involving 79 everyday objects (e.g., “kitchen lower cabinet”), BSC-Nav maintains 40.2 and 38.9% SR on MP3D’s unseen and seen validation sets, outperforming the supervised method DAgRL in zero-shot settings. The results of instance-level navigation (TIN and IIN) further validate BSC-Nav’s robust multi-modal generalization. In TIN tasks, it nearly doubles the SR compared to UniGoal and VLFM. In IIN tasks, direct matching of target images with cognitive map features yields 71.4% SR, outperforming UniGoal by 11.4%. Notably, BSC-Nav achieves significantly higher navigation efficiency across all tasks compared to baseline methods, as measured by SPL scores. This improvement stems from our distance- and confidence-based scoring strategy within the working memory module, which enables the agent to plan efficient exploration sequences by examining only a small number of candidate locations (Supplementary Fig. 3). Interestingly, OGN and IIN generally reach higher SR and SPL with fewer explored candidates than OVON and TIN (Supplementary Fig. 3). We attribute this difference to target ambiguity: OGN and IIN typically provide stronger semantic or visual grounding, whereas OVON and especially TIN require broader open-vocabulary reasoning or finer-grained instance disambiguation, making candidate verification more challenging. Besides, performance differences across target categories likely reflect both visual saliency and contextual regularity. Large and visually distinctive objects such as beds are often easier to localize and verify because they occupy more stable and semantically predictable regions, whereas smaller or less distinctive targets can introduce greater ambiguity during retrieval and stopping decisions. A representative visualization example is presented in Fig. 3b, including top-down trajectories, first-person observations, and MLLM-assisted target verification. We provide additional examples across various navigation tasks in Supplementary Movie 1 and Supplementary Fig. 4. Furthermore, detailed numerical results for all four navigation tasks are provided in Supplementary Tables 1–3. Higher-level spatially-aware skills By integrating structured spatial memory with MLLMs’ robust visual grounding and high-level planning capabilities, BSC-Nav demonstrates strong performance on higher-order spatial tasks, such as long-horizon navigation and spatial reasoning based on complex linguistic instructions. Figure 4 summarizes the overall performance, including quantitative results in Fig. 4a, b and representative qualitative examples in Fig. 4c, d. Higher-level spatially-aware skills enabled by BSC-Nav. a Comparison of BSC-Nav against baseline methods on long-horizon instruction-following tasks using the VLN-CE R2R benchmark. Quantitative results are averaged over 1000 episodes. Bar charts report Success Rate (SR, %) to quantify efficacy and Success weighted by Path Length (SPL, %) to quantify efficiency. Source data are provided as a Source Data file. b Performance comparison on active embodied question answering tasks using the A-EQA benchmark, with results averaged over 184 test samples and evaluated by the LLM-match score. The right panel provides a fine-grained performance breakdown of BSC-Nav, baselines, and human references across seven EQA question categories: object recognition (OR), object localization (OL), attribute recognition (AR), spatial understanding (SU), object state recognition (OSR), functional reasoning (FR), and world knowledge (WK). Source data are provided as a Source Data file. c,d Representative visualizations of BSC-Nav’s navigation trajectories during human instruction navigation (c) and embodied question answering (d) tasks. We evaluate BSC-Nav on a representative task termed Long-horizon Instruction-based Navigation (LIN)55,56, which requires agents to understand and execute complex instructions containing multiple intermediate goals and spatial constraints. For example, an instruction such as “Go through the glass door, pass between the sofa and the coffee table, walk to the refrigerator, then turn right and stop at the staircase entrance” demands nuanced vision-language reasoning and spatial understanding. To address this, we equip BSC-Nav with GPT-o357, a powerful MLLM with advanced reasoning capabilities. GPT-o3 decomposes complex instructions into spatially grounded subgoals (termed waypoints, such as “glass door”, “refrigerator”, and “staircase entrance”) by language instruction and initial visual observation of the agent. This hierarchical planning strategy transforms instruction-following into sequential goal-directed navigation steps, enabling BSC-Nav to reliably reach each waypoint and ultimately the final destination. We evaluate BSC-Nav on the VLN-CE R2R benchmark56, which includes 1,000 human-annotated long-horizon instructions from MP3D. As shown in Fig. 4a, BSC-Nav achieves 38.5% SR in zero-shot settings, only 8.5% below Uni-Navid12, the state-of-the-art vision-language-action model trained with extensive task-specific supervision. Notably, BSC-Nav attains 53.1% SPL, significantly outperforming all baseline methods in navigation efficiency. We attribute this result to the complementary strengths of structured spatial memory and MLLM-based reasoning: GPT-o3 decomposes complex instructions into semantically grounded subgoals, while BSC-Nav reuses the landmark memory and cognitive map to localize and reach these waypoints efficiently across long trajectories. Compared with purely reactive or heavily supervised baselines, this memory-centric design allows the agent to better preserve spatial consistency and reduce unnecessary exploration, which is especially important in zero-shot long-horizon instruction following. Meanwhile, the remaining gap in SR relative to the best supervised method suggests that fully resolving complex linguistic ambiguity and multi-step spatial dependencies remains challenging without instruction-level task-specific training. Representative navigation trajectories are visualized in Fig. 4c, including simplified waypoint descriptions and visual prototypes generated by working memory retrieval. Additional examples are provided in Supplementary Movie 2. The full quantitative comparison is provided in Supplementary Table 4. The value of spatial cognition extends beyond navigation to spatial reasoning and scene understanding. Here we consider a representative task termed Active Embodied Question Answering (A-EQA)58–60, which requires agents to answer spatially grounded questions through active exploration of the environment. Unlike static visual question answering, A-EQA demands spatial understanding, exploratory planning, and dynamic observation synthesis, all well aligned with BSC-Nav’s strengths. To address each question, BSC-Nav first parses target waypoints (instances or regions relevant to the query) and then conducts goal-directed exploration. Upon reaching the appropriate locations, GPT-4o42 integrates local observations with the original question to generate spatially grounded answers. We evaluate BSC-Nav on the A-EQA subset of the OpenEQA benchmark60, which includes 184 questions that span seven categories. We compare BSC-Nav with three representative baselines: a blind LLM that answers without visual observations; a question-agnostic frontier exploration strategy60; and Explore-EQA61, an active exploration method without structured spatial memory. The overall performance is measured by LLM-Match60, which computes semantic similarity between the generated and reference answers using LLMs (detailed in Methods). As shown in Fig. 4b, BSC-Nav achieves the highest LLM-Match score of 54.6, significantly outperforming all baselines. We attribute this advantage to the fact that A-EQA requires not only local visual understanding but also active exploration guided by reusable spatial memory. In contrast to blind LLMs or question-agnostic exploration, BSC-Nav can parse target-relevant waypoints, conduct goal-directed exploration, and accumulate spatial evidence before answering. The category-wise breakdown in Fig. 4b further shows particularly strong improvements in tasks that require fine-grained spatial localization and object-state awareness, where structured spatial memory is especially beneficial for efficient and accurate information gathering. The remaining gap to human performance underscores that embodied spatial reasoning still requires stronger integration of commonsense knowledge, causal inference, and abstraction from sparse observations. A representative example in Fig. 4d exhibits how BSC-Nav combines goal-directed exploration with structured spatial memory to address diverse spatial reasoning problems. The detailed LLM-Match scores across all question categories are reported in Supplementary Table 5. Real-world navigation and mobile manipulation To demonstrate BSC-Nav’s generalization beyond simulation and its broad applicability in downstream tasks, we deploy it on a custom-built mobile robotic platform within physical indoor environments. This evaluation demonstrates how structured spatial memory enables reliable long-range navigation and integrated manipulation in the real physical world. The mobile platform (Fig. 5a) features a compact mechanical design equipped with high-precision motion control, as well as modular sensory and actuation capabilities. We conduct a total of 75 navigation episodes in a two-story indoor space (around 200 m2, Fig. 5b), covering three types of goal-directed tasks: Object-Goal Navigation (OGN), Text-Instance Navigation (TIN), and Image-Instance Navigation (IIN), with five distinct targets per task (Fig. 5c). Each target is tested over five trials from randomly sampled starting positions, with an average path length of 23.4 m. Universal navigation capabilities in real-world scenarios. a The custom-built robotic platform integrates perception, navigation, and manipulation capabilities. b The indoor experimental environment spans 200 m2 and includes diverse functional zones (e.g., office, lounge, reception, and kitchen). c–g Real-world navigation performance across Object-Goal Navigation (OGN), Text-Instance Navigation (TIN), and Image-Instance Navigation (IIN) tasks with 15 distinct targets (c). For each target, we conducted n = 5 independent experimental trials with randomized starting positions. We report the success rate (SR) (d), final distance to goal (e), and mean navigation velocity (f). In (e) and (f), data are presented as box plots with individual data points overlaid. For all box plots, the center line indicates the median, the lower and upper bounds of the box represent the 25th and 75th percentiles, respectively, and the whiskers extend to the minimum and maximum values. Source data are provided as a Source Data file. g Representative examples of real-world navigation. Each shows a top-down trajectory with timestamps (left) and corresponding egocentric-allocentric views (right). BSC-Nav demonstrates robust spatial generalization in all real-world tasks. As shown in Fig. 5d, it achieves at least 3 out of 5 successful trials per target (SR is defined as reaching within 1.0 m), with IIN performing best, reaching 100% SR on 4 out of 5 targets. Both OGN and TIN reach 100% SR on two targets and exceed 66.7% SR across all cases. Even in failure cases, BSC-Nav reliably localizes to semantically plausible regions, demonstrating strong spatial awareness. The distance-to-goal distribution in Fig. 5e further confirms the precise stop location, with the final distance below 2.5 m in all cases and tight clustering between targets, particularly for IIN. Navigation efficiency is reflected in a mean velocity of 0.76 m/s with low variance (Fig. 5f), indicating stable and efficient movement. Representative navigation trajectories are visualized in Fig. 5g, with complete demonstrations available in Supplementary Movies 3–7. The trial-level raw data for all 75 real-world navigation episodes are provided in Supplementary Tables 6–8. To further demonstrate the advantages of the BSC-Nav architecture in real-world deployments, we conduct an ablation study on the Text Instance Navigation (TIN) task, comparing the full BSC-Nav against a landmark-memory-only variant (i.e., excluding the cognitive map). The configuration of this variant serves a dual critical purpose: (i) it acts as a proxy baseline for existing modular methods that model solely landmark-level spatial knowledge without incorporating dense visual-spatial representations; and (ii) it provides a controlled setting to rigorously analyze the specific contributions of different memory modules in physical environments. As shown in Supplementary Fig. 5c, the landmark-only baseline achieved an overall navigation success rate of merely 52%, substantially lower than that of the full BSC-Nav (78%). This pronounced performance gap arises because the baseline, stripped of the dense visual-spatial features provided by the cognitive map, struggles immensely with text-instance targets that necessitate fine-grained visual disambiguation. Constrained exclusively to category-level semantic matching, the baseline method is forced to resort to inefficient sequential exploration across a series of candidate waypoints. Supplementary Fig. 5d visualizes this candidate region exploration process for the landmark-only variant, and the detailed quantitative raw data are provided in Supplementary Table 9. Beyond navigation, BSC-Nav’s structured spatial memory supports robust mobile manipulation guided by natural language. Although prior embodied manipulation systems62,63 have shown promise in static, small-scale environments, they often lack the spatial generalization necessary for large-scale deployment. BSC-Nav overcomes this limitation by seamlessly integrating long-horizon navigation with goal-conditioned manipulation. In our demonstrations, natural language instructions are interpreted by GPT-464 to generate waypoint-action sequences. Upon reaching each waypoint, the agent executes a predefined manipulation primitive, such as grasp, place, and pour. Representative examples in Figure 6 show both single-step and multi-step tasks. Notably, the “make breakfast” task (Fig. 6c) involves identifying and interacting with three spatially distributed, open-vocabulary objects, emphasizing the critical role of structured spatial memory in long-horizon reasoning and reliable object grounding. Full demonstrations are provided in Supplementary Movies 8–10. Real-world mobile manipulation. The coordination between spatial memory-driven navigation and manipulation primitives enables the execution of long-horizon tasks specified through human instructions. a Single-waypoint task, cleaning stains on a marble table next to a shredder. b Object transfer task, relocating a cookie box from table to the kitchen island, requiring navigation between manipulation actions. c Complex multi-step task, preparing breakfast by sequentially navigating to and manipulating three open-vocabulary targets (oatmeal jar, coco balls, and milk bottle) to assemble ingredients on a plate. Full demonstrations are provided in Supplementary Movies 8--10. Discussion This work demonstrates that organizing embodied navigation around high-level principles inspired by biological spatial cognition can be effective in practice. By constructing a structured spatial memory comprising computational counterparts of landmarks, route knowledge, and survey knowledge, BSC-Nav improves navigation performance and extends the spatial capabilities that complement the perceptual and reasoning strengths of modern MLLMs. Conventional embodied agents often rely on reinforcement learning or imitation learning, acquiring task-specific policies through extensive trial-and-error or teleoperated demonstrations. Here, “embodied” refers to agents that perceive and act in closed-loop interaction with physical or simulated environments, rather than explicitly modeling morphology-dependent cognition in the stronger sense emphasized in embodiment theory65. While effective in controlled settings, prior methods remain fundamentally reactive, responding to immediate stimuli without persistent or reusable spatial knowledge. BSC-Nav addresses this limitation by integrating structured spatial memory with foundation models, enabling a transition from observation-driven pattern matching to multi-level spatial reasoning. BSC-Nav exhibits strong performance across navigation tasks of varying modalities and granularities, especially in open-vocabulary and instance-level settings. These results demonstrate how memory-centric spatial representations enable such agents to decouple planning from immediate perception, reuse prior experience across tasks, and translate high-level goals into concrete actions, which we view as important components of cognitive spatial intelligence. Importantly, BSC-Nav is not purely retrieval-based but also supports active exploration by ranking candidate targets, selectively visiting promising locations, and verifying goals through ongoing interaction with the environment. BSC-Nav provides a computational framework that operationalizes long-standing theories of spatial knowledge in biological systems16,17. While neuroscience has proposed that spatial cognition builds upon interconnected representations of landmarks, route knowledge, and survey knowledge, most experimental evidence has been behavioral or correlational. BSC-Nav demonstrates how these elements can be synergistically implemented to support long-horizon spatial understanding, planning, and reasoning. The cognitive map module further incorporates a surprise-driven update strategy inspired by the free-energy principle37,38 of representation updating, encouraging stronger incorporation of observations that are less consistent with existing memory. Furthermore, recent findings suggest that sequential route learning drives the formation of allocentric cognitive maps in the hippocampus-entorhinal circuits66, a process with similar ideas to BSC-Nav’s trajectory voxelization. These parallels reveal a convergent computational basis shared between cognitive science and AI. BSC-Nav is particularly suited to goal-directed embodied operation in partially observable environments, where agents must link semantic goals to persistent spatial structure, revisit previously seen regions, disambiguate candidate targets, and coordinate navigation with downstream interaction. This makes the framework directly relevant not only to service and assistive robotics, but also to embodied question answering and other long-horizon tasks that require accumulated, queryable, and updateable spatial experience. Extending BSC-Nav across environmental scales, however, poses distinct challenges rather than a single scaling problem. In compact environments, the key issue is representational economy, namely avoiding redundant memory growth while maintaining efficient retrieval. In larger or more open-ended environments, the main bottlenecks shift to memory capacity, map consistency, and retrieval efficiency. These observations highlight scalability as an important next consideration for structured spatial memory beyond the benchmark settings studied here. As embodied AI systems advance in perceptual and motor capabilities, it becomes increasingly important to evaluate not only whether agents can act successfully in physical environments but also how language and spatial cognition interact to support such behavior. In cognitive science, growing evidence suggests that language is not merely a detached symbolic interface but can itself serve as a source of grounding and a mode of action, while spatial knowledge and map-like representations are in turn deeply entangled with language experience67–69. Our results are broadly consistent with this perspective. In BSC-Nav, linguistic goals become most effective when they can be anchored to structured spatial memory, whereas spatial memory becomes more flexible when it can be queried, abstracted, and verified through language-guided reasoning. This interplay suggests that human-like spatial intelligence may depend not only on building internal models of space, but also on coupling them tightly with linguistic representations that support abstraction, disambiguation, and goal specification. From this viewpoint, a future embodied Turing test for spatial cognition70 may need to assess not only navigation accuracy, but also whether agents can construct reusable spatial representations, reason from sparse observations, and translate natural-language goals into grounded spatial plans in ways that remain robust across tasks and environments. Despite its effectiveness, the present work still has several limitations. First, although BSC-Nav is organized around high-level principles inspired by biological spatial cognition, it is not intended as a cognitively faithful model of human cognition. In particular, the language and visual reasoning components are implemented with modern MLLMs, which are used primarily as engineering modules for semantic grounding, visual interpretation, and high-level reasoning, rather than as biologically or developmentally plausible models of human language processing. Recent psycholinguistic work has likewise emphasized that strong linguistic generalizations learned by neural language models do not by themselves establish cognitively plausible mechanisms, especially for large-scale models trained far beyond human-like data conditions71. From this perspective, BSC-Nav shows that structured spatial memory can provide an effective computational interface between perception, language, and action, rather than offering a mechanistic account of how these functions are jointly realized in the brain. Second, while the framework has clear conceptual implications for memory-centric embodied agents, its theoretical contribution remains primarily architectural and computational. More formal analyses of when and why structured spatial memory improves generalization, efficiency, and language-guided behavior remain important directions for future work. Several future directions emerge from this work. Extending BSC-Nav to dynamic and unstructured environments will require spatial memory that can be continually revised as scenes evolve over time. Scaling to larger environments will in turn demand more efficient memory organization, including hierarchical abstraction, compact storage, and retrieval with bounded latency. Richer embodied settings may further benefit from integrating additional sensory streams, such as depth, audio, or tactile signals, together with uncertainty-aware updating. Beyond single-agent navigation, future systems should also support collaborative and multi-agent spatial reasoning, where memory can be shared, communicated, or jointly refined. More broadly, an important longer-term direction is to explore architectures in which memory, perception, and language are developed under stronger constraints of embodiment, experience, and computational efficiency. Progress along these directions may help narrow the gap between high-performing engineering systems and more cognitively grounded models of spatial intelligence. Methods The BSC-Nav framework This section describes the implementation details of BSC-Nav, including the three synergistic modules (Fig. 1b) and the pipeline of constructing and leveraging structured spatial memory (Supplementary Fig. 1). Observation space and pre-processing The observation space of BSC-Nav is defined as Ot={It,Dt,Pt}, comprising the RGB image It∈RH×W×3, the depth image Dt∈Rw×h and the pose of the agent Pt=(Xt,Yt,ϕta)∈R3 at time t, where Xt and Yt denote the agent’s 2D position in the world coordinate system (defaulting to the Cartesian coordinate centered at the SLAM72 initialization point), ϕt is the angle of yaw, and H and W are image width and height, respectively. For simplicity, we omit the time subscript t in subsequent descriptions. RGB images serve as input for both salient object detection and visual feature extraction, while depth images and pose information enable projection from pixel coordinates to the world coordinate system. In a simulation environment, the agent receives RGB images at a resolution of 680 × 680 pixels with the depth image aligned to the same frame, and its pose is obtained directly from the simulator’s ground-truth odometry. In real-world deployment, BSC-Nav acquires 848 × 480 RGB-D observations at 30 FPS from an Intel RealSense D435i depth camera mounted at 1.5 m above ground level. The agent’s pose is estimated in real time via 32-beam LiDAR-based SLAM. To mitigate sensing artifacts such as depth holes and edge discontinuities, the raw depth stream in the real-world is processed through a cascaded filtering pipeline, comprising threshold filtering (valid range [0.3, 8.0] m), spatio-temporal smoothing, and hole-filling filters. Landmark memory During active exploration, an observation O is processed by two branches in parallel to construct structured spatial memory. For landmark memory, we instantiate it as a list of 4-tuples as follows: where θk=(Xkw,Ykw,Zkw)∈R3 represents the 3D coordinates of the center of the k-th instance in the world coordinate system, ck∈C denotes the open-vocabulary category of the predefined category set C, ρk ∈ [0, 1] indicates detection confidence, and Tk is the description of the instance generated by GPT-4o42 based on observations, which encompasses texture, shape, and spatial contextual semantics. To obtain the world coordinates θk, we perform a series of coordinate transformations. Given a detected object with the bounding box center at pixel coordinates (uk, vk) and the corresponding depth value dk, we first compute the 3D point in the camera coordinate system, which is defined with its origin at the camera’s optical center, the xy-plane parallel to the image plane, and the z-axis aligned with the camera’s optical axis. The point in the camera coordinate system can be calculated through inverse perspective projection: where K∈R3×3 is the intrinsic matrix of the camera with focal lengths (fx, fy) and principal point (cx, cy). Subsequently, we transform this point into the world coordinate system. Given the robot pose Pt = (Xt, Yt, ϕt) from the observation space at time t, we construct the base-to-world transformation matrix: where Zbase represents the robot base height in the world frame. The point is then transformed through cascaded homogeneous transformation: where Tbasecam∈SE(3) represents the fixed rigid transformation from camera to robot base frame. The final world coordinates are extracted as θk=pkworld=(Xkw,Ykw,Zkw)⊤. In practice, we employ the open-vocabulary object detector YOLO-World73 for the perception of salient instances. We predefine several common object categories as the landmark category set C={“sofa”,“sink”,“bed”,…}, with a detection confidence threshold to exclude semantically ambiguous and non-salient instances. To prevent duplicate memorization of the same instance, we perform overlap detection for each newly detected landmark LN+1. Define the spatial overlap set as: U={Lj∈Mlandmark:∥θN+1−θj∥2<δoverlap∧cN+1=cj}. If U≠∅, we perform the memory fusion of existing landmarks: After memory fusion, all elements in U are removed from Mlandmark and Lfused is added. This update ensures that each landmark 4-tuple represents a unique spatial instance, avoiding information redundancy. Cognitive map In parallel with the landmark memory module, the cognitive map module continuously projects RGB observations from egocentric movement sequences (corresponding to the route knowledge) into voxelized trajectories, which refer to the intermediate discretized visual-spatial representations in a voxel grid, and progressively integrates and updates such voxelized trajectories into allocentric, map-like representations (corresponding to the survey knowledge). We define the cognitive map as a discrete voxelized representation: where V⊆Z3 denotes the discrete voxel index space, each voxel v = (vx, vy, vz) maintains a feature buffer Fv containing up to B feature vectors and ℏ is the dimension of the visual feature. To achieve fine-grained visual-spatial encoding, we employ DINO-v234 (a powerful self-supervised visual encoder) to extract patch-level features from continuous 2D RGB observations. Given an RGB image I∈RH×W×3, DINO-v2 produces patch tokens organized as a spatial grid Fpatch∈RH′×W′×D, where H′=H/s and W′=W/s with patch stride s. These patch-level features are projected from 2D image coordinates to corresponding voxel coordinates through a multi-stage transformation process. Let (i, j) denote the indices in the patch grid where i∈{0,1,…,H′−1} and j∈{0,1,…,W′−1}. For each patch (i, j), we first determine its center pixel coordinates in the original image: where s is the patch stride. Using the depth value dij sampled at this location, we compute the corresponding 3D point in the camera coordinate system: This point is then transformed into world coordinates using the robot pose P = (X, Y, ϕ) from the observation space: where Tworldbase is constructed from the robot pose as defined in Eq. (7). With pijworld=(Xijw,Yijw,Zijw)⊤, we discretize the continuous world coordinates into voxel indices: where Δ is the voxel size (spatial resolution) and G is the grid dimension. The offset G/2 centers the voxel grid at the world origin. The visual feature Fpatch[i,j]∈RD from patch (i, j) is then associated with voxel vij. In our implementation, we set Δ = 0.1 m and G = 1000 for simulation environments, yielding an effective spatial coverage of 100 × 100 m2 with a vertical range of [ − 10, 10] m. For real-world deployment, we adopt a coarser resolution Δ = 0.2 m with G = 2000, providing a 400 × 400 m2 coverage and a height range of [ − 15, 15] m to accommodate multi-floor structures and sensor calibration variance. To extract patch-level visual features for the cognitive map, we employ DINOv2 (ViT-L/14 with registers)34,35, which produces ℏ = 1024-dimensional feature vectors at a patch stride of s = 14. RGB observations are resized to 224 × 224 (simulation) or 252 × 448 (real-world, preserving the native camera aspect ratio), resulting in 256 or 576 patch tokens per frame, respectively. During depth-to-3D projection, we subsample depth maps at a rate of 1 per 1,000 points in simulation (1 per 200 in the real world) to balance computational cost with spatial coverage. Each projected point is assigned a radial Gaussian weight α=exp(−r2/(2σ2)), where r is the radial distance from the camera optical center and σ2 = 0.6, suppressing noisy features from peripheral and distant observations. In Mcog, we avoid redundantly storing all visual features from new observations, which would lead to information overload and inefficient retrieval. We also forgo conventional fusion methods, such as grid averaging or distance-based weighting31,74, which often introduce bias in visual feature representation. Instead, inspired by biological learning and memory, we maintain dynamic buffers for each voxel and introduce a surprise-based update strategy. Free-energy37,38 and predictive-processing39,40 accounts suggest that biological systems update internal models in response to mismatches between observation and expectation. Motivated by this broad perspective, BSC-Nav updates its cognitive map according to the degree of deviation between new observations and existing memory representations within a given spatial region, quantified as a heuristic surprise score. This score is used as an engineering criterion for selective memory updating and is not intended as a formal variational free-energy objective41. For each new visual feature fnew projected to voxel v∈Z3, we compute its surprise score as where Nn(v)={v′∈Z3:∥v−v′∥∞≤n} denotes the n-hop Chebyshev neighborhood around voxel v, FNn(v)=⋃v′∈Nn(v)Fv′ represents the union of all feature buffers in the neighborhood, and D(⋅,⋅) is the distance metric (e.g., cosine distance). We set n = 1 as each voxel’s surprise is computed against features stored in its 33 − 1 = 26 adjacent voxels (1-hop Chebyshev neighborhood). When S(fnew,v)>τ (the surprise threshold), fnew is added to buffer Fv. Since D is instantiated as the cosine distance between L2-normalized DINOv2 features, the computed surprise score S is strictly bounded within [0, 1]. We select τ = 0.5 as the natural midpoint of this spectrum, which ensures that a new feature is incorporated only when its average divergence from existing neighborhood representations outweighs its similarity. Adopting a lower threshold would admit near-redundant features and needlessly inflate memory without enhancing representational diversity. Conversely, a higher threshold risks discarding observations from moderately novel viewpoints, ultimately compromising the multi-perspective coverage of the cognitive map. If ∣Fv∣=B, the feature with the highest cosine similarity to fnew (i.e., the most redundant entry) within the buffer is replaced to maintain memory efficiency. In addition, a periodic deduplication step is applied: features within each voxel whose pairwise cosine similarity exceeds 0.95 are grouped via union-find clustering and merged by averaging, thereby compacting redundant encodings accumulated from overlapping viewpoints over time. This surprise-based update strategy achieves two key advantages: (i) it enhances the robustness of spatial knowledge by selectively caching diverse features across viewpoints and timepoints in adapting to dynamic environments; and (ii) it maintains memory storage and retrieval efficiency by avoiding redundant encoding of stable environmental elements. Working memory The working memory module is responsible for the hierarchical retrieval and strategic reorganization of spatial knowledge from both the landmark memory Mlandmark and the cognitive map Mcog, enabling BSC-Nav to address navigation tasks of varying modalities and granularities (Fig. 2b). Unlike the parallel construction and passive updating of the two memory branches, the working memory is activated only upon receiving navigation instructions. It employs a hierarchical retrieval strategy guided by instruction complexity. For simple and concrete goals, it prioritizes the fast retrieval of landmark memory. For fine-grained or image-based instructions, it further engages the cognitive map for precise visual-spatial localization (Fig. 2c). The structured knowledge base of landmark memory, including landmark categories, confidence scores, and contextual descriptions, provides a strong foundation for MLLMs to reason about target locations. Unlike direct rule matching, this approach enables context-aware reasoning and can infer the location of unrecorded targets based on spatial associations. For example, even if a “toaster” is not explicitly recorded, the system can infer its location by leveraging co-located landmarks such as “stove” and “kitchen island”. Specifically, we design retrieval prompts (see prompt templates in Supplementary Sec. A) that guide a text-only GPT-4 to integrate confidence scores and descriptive semantics to generate a set of candidate coordinates {θcandi}i=1K from Mlandmark. To bridge the modality gap between textual instructions and visual representations, we perform association-enhanced retrieval over the cognitive map. It first employs text-only GPT-4o to refine goal descriptions, enriching them with detailed texture and spatial context information. Specifically, in LIN tasks, initial visual observations are additionally provided as environmental priors to enhance the accuracy and specificity of refined descriptions. These enriched descriptions are then used to “imagine” potential visual appearances of targets through a text-to-image generation model, defaulting to Stable Diffusion 3.575. This “imagine-then-localize” process resembles human pre-navigation thinking. The generated imagined image is encoded by DINO-v2 to extract patch-level features {fi}i=1Q, followed by center-distance weighted pooling to obtain the instance visual representation: where (xi, yi) denotes the spatial coordinates of the i-th patch, (xc, yc) is the image center, and α is the temperature parameter. This weighting strategy suppresses background interference and enhances central features of the target instance. As an intermediate query, the pooled visual features are further matched against features stored in the cognitive map, returning the top-K voxel coordinate set with maximum cosine similarity. We perform similarity-weighted DBSCAN clustering76 on the voxel coordinate set to obtain the grid coordinates of cluster centers as the final spatial coordinate candidates. These grid coordinates need to be further back-projected to the world coordinate system, yielding {θcandi}i=1Q for subsequent low-level planning. We design a composite scoring function that integrates target existence probability and spatial distance to prioritize the set of candidate spatial coordinates. Specifically, for candidates retrieved from the landmark memory, we use detection confidence as the existence probability. For candidates retrieved from the cognitive map, we employ the cosine similarity of visual features. The priority scoring function is defined as: where pi represents the existence probability (confidence or similarity) of the i-th candidate, di is the Euclidean distance between the candidate and the starting point, dmax=maxjdj normalizes distance across candidates. The weighting hyperparameter λ balances existence probability and exploration efficiency, defaulting to λ = 0.5, assigning comparable weights to the target existence probability and spatial proximity. This configuration aligns with our empirical finding that localization accuracy and spatial distance are equally pivotal to navigation efficacy and efficiency. When multiple candidates exhibit similar existence probabilities, spatial proximity serves as the tie-breaker to encourage prioritized exploration of the nearest points. Low-level navigation policy generation Low-level navigation policies are typically derived from deterministic path planning conditioned on environmental constraints and high-level spatial goals52. BSC-Nav employs a hierarchical navigation strategy: the exploration sequence of each candidate coordinate serves as a high-level policy, while heuristic algorithms are used to generate actionable low-level policies toward each candidate coordinate. In simulation environments, we employ the greedy shortest path algorithm provided by the Habitat simulator46, which operates directly on 3D mesh representations of the scene to compute optimal action sequences within a discrete action space. For real-world deployment, we implement a two-tier planning architecture. Global planning is carried out using the A* algorithm77 on occupancy grid maps constructed via LiDAR-based SLAM, yielding globally optimal paths. Local planning leverages the Timed Elastic Band (TEB) algorithm78,79, which dynamically adjusts trajectories to avoid obstacles while maintaining efficiency. The TEB planner outputs continuous velocity commands to control the robot chassis, ensuring smooth and precise motion execution. Goal verification and affordance Upon arriving at a candidate coordinate, BSC-Nav verifies whether the navigation target is present at the location. The robot first performs a 360∘ rotational scan to capture a sequence of RGB images. Cosine similarities are then computed between the CLIP visual embeddings of these images and the goal’s text or visual embeddings to identify the viewing angle with the highest semantic alignment. The selected image is subsequently fed into GPT-4o for precise target verification. Beyond confirming target presence, GPT-4o is also prompted to generate a sequence of affordance-based actions to guide the robot in fine-tuning its pose, adjusting its relative position, and orientation for optimal proximity and visibility. This ensures favorable initial conditions for downstream manipulation tasks. Experimental details Simulator and datasets The simulation experiments are conducted under the Habitat 3.0 platform80, a commonly used simulation framework for embodied AI and human-robot interaction in domestic environments. We select this platform for its strong support for large-scale embodied navigation, including (i) habitat-sim, a high-performance simulator offering photorealistic rendering and physics-based interactions for mainstream indoor datasets, and (ii) habitat-lab, a modular benchmarking suite supporting a variety of navigation tasks with standardized pipelines and metrics. Here we evaluate BSC-Nav and baseline methods on four foundational navigation tasks: Object-goal navigation (OGN). This task81 comprises two benchmarks, including (i) 2195 episodes across 34 HM3D scenes involving 6 object categories, and (ii) 2000 episodes across 10 MP3D scenes with 20 object categories. Object-goal navigation (OGN). This task81 comprises two benchmarks, including (i) 2195 episodes across 34 HM3D scenes involving 6 object categories, and (ii) 2000 episodes across 10 MP3D scenes with 20 object categories. Open-vocabulary object navigation (OVON). This task49 extends to 79 open-vocabulary categories across 10 MP3D scenes to address the limited object categories in object-goal navigation. We sampled 1000 episodes each from the validation-seen and validation-unseen splits. The seen split includes categories present in training (though not exact instances), while the unseen split comprises entirely novel and semantically dissimilar categories. Open-vocabulary object navigation (OVON). This task49 extends to 79 open-vocabulary categories across 10 MP3D scenes to address the limited object categories in object-goal navigation. We sampled 1000 episodes each from the validation-seen and validation-unseen splits. The seen split includes categories present in training (though not exact instances), while the unseen split comprises entirely novel and semantically dissimilar categories. Text-instance navigation (TIN). This task50 provides natural language descriptions for 795 instances across 36 HM3D scenes. Descriptions encompass both intrinsic attributes (inherent object properties including shape, color, and material) and extrinsic attributes (surrounding environmental contexts), annotated with a strong MLLM termed CogVLM82. We evaluate all 1000 test episodes. Text-instance navigation (TIN). This task50 provides natural language descriptions for 795 instances across 36 HM3D scenes. Descriptions encompass both intrinsic attributes (inherent object properties including shape, color, and material) and extrinsic attributes (surrounding environmental contexts), annotated with a strong MLLM termed CogVLM82. We evaluate all 1000 test episodes. Image-instance navigation (IIN). This task51 employs single-view rendered images as navigation targets for instances across 34 HM3D scenes. We sampled 1000 episodes from the validation split. Image-instance navigation (IIN). This task51 employs single-view rendered images as navigation targets for instances across 34 HM3D scenes. We sampled 1000 episodes from the validation split. Beyond these foundational navigation tasks, we further evaluate BSC-Nav and baseline methods on higher-level spatially-aware skills: Long-horizon instruction-based navigation (LIN). We employ the VLN-CE R2R benchmark56, which provides 1000 navigation episodes in Habitat-lab based on human-annotated long-horizon instructions across 11 MP3D scenes. Long-horizon instruction-based navigation (LIN). We employ the VLN-CE R2R benchmark56, which provides 1000 navigation episodes in Habitat-lab based on human-annotated long-horizon instructions across 11 MP3D scenes. Active embodied question answering (A-EQA). We develop a custom evaluation pipeline within Habitat-lab using OpenEQA60. Agents are initialized at the first frame of each recorded exploration trajectory and are allowed to actively explore the environment to answer a given spatially grounded question. We evaluate the 184 test queries that span seven task categories. Active embodied question answering (A-EQA). We develop a custom evaluation pipeline within Habitat-lab using OpenEQA60. Agents are initialized at the first frame of each recorded exploration trajectory and are allowed to actively explore the environment to answer a given spatially grounded question. We evaluate the 184 test queries that span seven task categories. Evaluation metrics We adopt two-dimensional metrics53 to quantify navigation performance. Success Rate (SR) measures the proportion of successful navigation episodes relative to the total test episodes, assessing the agent’s capability to accurately navigate target object instances. Notably, in OGN, any instance of the target category constitutes a valid goal regardless of distance from the starting position, whereas instance-level navigation requires reaching a unique specified instance in the environment. Specifically, an episode is deemed successful if the Euclidean distance between the agent and the target object is within 1.0 m upon executing the “stop” action. While Success weighted by Path Length (SPL) jointly considers navigation success and path efficiency: where N denotes the total number of episodes, Si is the binary success indicator for episode i (1 for success, 0 for failure), Li* represents the geodesic shortest path length, and Li is the actual trajectory length executed by the agent. SPL values range from 0 to 1, with higher values indicating more efficient navigation. While SR quantifies task completion capability, SPL further evaluates the optimality of path planning, collectively providing a comprehensive assessment of navigation performance. For A-EQA tasks, we adopt the LLM-Match scoring metric proposed in Open-EQA60 to assess the correctness of agent responses. This metric accounts for the open-vocabulary nature of the agent responses, serving as a substitute for manual evaluation. Specifically, given a question Qi, a human annotated response Ai*, and an agent response Ai, the LLM is prompted to assign a score σi ∈ {1, . . . , 5}, where σi = 1 denotes an incorrect answer, σi = 5 denotes a correct answer, and intermediate values reflect varying degrees of partial correctness. The overall LLM-based correctness is then computed as follows: where NQ denotes the total number of questions. Following the Open-EQA protocol, we employ text-only GPT-4 under official prompts to ensure evaluation fairness. Hardware stack We implement BSC-Nav on a physical robotic platform for real-world deployment, comprising five core components: locomotion chassis, industrial computer, SLAM suite, robotic arm, and vision sensors. The locomotion system employs the Agilex Ranger-mini 3.0 platform, chosen for its balance between cost and payload capacity. It features Ackermann steering with zero-radius turning capability, enabling agile maneuvering in complex environments. The SLAM suite integrates a 32-beam LiDAR, IMU sensor, and depth-compensation camera to support collision detection and real-world localization. For manipulation capabilities, we equipped the platform with a Franka Emika Research 3 robotic arm. Two Intel RealSense D435i cameras serve as primary vision sensors (one mounted at 1.5 m above ground level and another at the manipulator’s end effector), providing 848 × 480 RGB-D imagery with an 87∘ field of view. To mitigate depth-sensing artifacts including holes and edge discontinuities, we apply spatio-temporal and hole-filling filters, constraining the effective sensing range to 0.3-8.0 m. An industrial-grade computer equipped with an NVIDIA RTX 4090 GPU handles all real-time processing, including BSC-Nav, SLAM computation, and manipulator control, ensuring end-to-end system responsiveness. Simulation environments Before task execution, BSC-Nav requires environmental perception to construct preliminary landmark memory and cognitive map representations. For simulation environments, we implement a frontier-based52 autonomous exploration strategy for spatial memory construction: at each time step, the system generates a height map through depth projection to identify boundaries between explored and unexplored regions. On traversable boundaries, the system selects the nearest frontier point to the current position as the next exploration target, utilizing Habitat’s scene-mesh-based greedy navigator for low-level action planning. Upon reaching each frontier point, the agent performs a 360-degree rotation to comprehensively perceive the surrounding environment. This exploration process continues until a predetermined iteration limit is reached, which is dynamically adjusted based on scene scale. The iteration count is defined as half of the traversable area. During task execution, both landmark memory and cognitive map maintain continuous updates. Parameter configurations are as follows: for landmark memory, the detector confidence threshold is set to 0.55, a standard operating point for YOLO-World on indoor datasets that balances precision against recall for salient object retention, with a spatial overlap distance of 1.0 m, chosen to match the navigation success radius so that landmarks within one success zone are merged as a single instance. For the cognitive map, we set a voxel resolution of δ = 0.1 m3, a grid dimension of G = 1000, and a buffer capacity of B = 10 per voxel. The grid configuration yields a 100 × 100 m2 spatial coverage that exceeds the largest indoor scenes in HM3D and MP3D (typically < 80 m span), while Δ = 0.1 m affords object-scale spatial resolution. The per-voxel buffer capacity B = 10 empirically balances feature diversity against memory footprint: with 1024-dimensional features, the worst-case working set remains within the 24 GB VRAM budget of our deployment GPU. For each navigation episode, the system first retrieves candidate target locations through the working memory module, with a maximum of K = 3 candidates for landmark memory retrieval and Q = 3 for cognitive map retrieval. Our empirical analysis shows that the majority of successful episodes locate the target at the first explored candidate, and beyond K, Q = 3 the cumulative success rate plateaus while path efficiency (SPL) steadily declines (Supplementary Fig. 3b,c), indicating diminishing returns in efficacy at the cost of exploration efficiency. We employ Stable Diffusion 3.5-Medium for cognitive map retrieval, generating three images per batch to mitigate stochasticity in visual imagination. The system subsequently navigates to each candidate location following the planned exploration sequence, performing target verification upon arrival at each candidate. Task success is determined when at least one candidate passes verification. If all candidates fail verification, the task is deemed unsuccessful. In simulation environments, the agent operates in a discrete action space comprising forward motion of 0.25 m and left/right turns of 30∘, with a maximum of 5,000 steps per episode and a success distance threshold of 1.0 m. The effective depth sensing range is [0.1, 10.0] m. These parameter settings follow the choices made by prior works during benchmark evaluations. Real-world deployment For real-world deployment, the initial phase of spatial memory construction relies on manual teleoperation, guided by two primary considerations. Firstly, strict safety constraints necessitate human oversight; unlike simulated environments, autonomous exploration (e.g., frontier-based methods) often struggles to navigate complex physical hazards safely in unconstrained settings. Given our physical environment features stairs, numerous glass doors and walls, and frequent human activity, manual guidance mitigates significant risks to both the robot and its payload. Secondly, employing manual exploration isolates our core contribution, i.e., the structured organization and retrieval of spatial memory, from the inherent failure rates of underlying exploration algorithms. During this pre-collection phase, an operator guides the robot through the environment at a walking pace, while the perception pipeline records RGB-D observations and builds the cognitive map and landmark memory in real time. During the subsequent task execution phase, BSC-Nav operates autonomously. Since a simulator mesh does not bound the physical environment, several adaptations are required relative to the simulation configuration as follows. The robot relies on LiDAR-based SLAM for real-time localization within a pre-built 2D occupancy grid map. At the start of each experiment, we must manually specify the robot’s initial pose by providing approximate pixel coordinates (x, y, θ) on the occupancy grid map. These pixel coordinates are converted to metric map coordinates via the map’s resolution and origin parameters and published to the SLAM system as the initial pose estimate. Several parameter adjustments are adopted to accommodate the physical sensing conditions. The detection confidence threshold is lowered to 0.4 to compensate for increased visual noise, motion blur, and lighting variability in uncontrolled environments, and the predefined landmark category set is reduced from 21 to 16. The voxel grid is configured with Δ = 0.2 m and G = 2000. The robot executes continuous velocity commands at a nominal linear speed of 0.2 m/s and an angular speed of 0.3 rad/s. A goal is considered reached when the agent’s position remains within a 0.5 m threshold for a stability duration of 5.0 s, with a per-goal navigation timeout of 300 s. The comprehensive hyperparameter configurations are provided in Supplementary Tables 12 and 13. We identify several failure modes observed during physical deployment: (i) Impassable regions. The local planner inflates obstacles on the occupancy grid by a safety margin to prevent collisions. In narrow passages or cluttered areas, the inflated costmap from the SLAM system may mark the entire passageway as non-traversable, causing the robot to become stuck or to abort the navigation goal due to a timeout. In such cases, the system proceeds to the next candidate location. If no candidates remain reachable, the episode is recorded as a failure. (ii) Calibration drift. Inaccurate initial pose calibration or accumulated SLAM drift over long trajectories can cause a systematic offset between the robot’s estimated position and the true coordinates stored in spatial memory. This misalignment degrades both candidate retrieval accuracy and goal verification, as the robot may arrive at a location displaced from the intended memory coordinate. (iii) Dynamic environmental disturbances. Large-scale environmental changes between memory construction and task execution, such as rearranged furniture, newly placed obstacles, or dense pedestrian traffic, can invalidate stored spatial memory and obstruct planned paths. When a navigation goal is persistently unreachable due to dynamic obstacles (e.g., sustained pedestrian flow blocking a corridor), the system times out, and the episode requires manual intervention to re-initialize the robot’s position and re-attempt the task. Computational footprint We present the computational requirements and real-time performance of BSC-Nav in both simulated and physical deployments. The minimum hardware requirement is a GPU with at least 24 GB of VRAM, primarily allocated for the real-time inference of DINOv2, YOLO-World, and Stable Diffusion 3.5 Medium, alongside 16 GB of system RAM dedicated to 3D coordinate projection and voxelized memory operations. Such resource configurations are readily satisfied by current-generation embodied computing platforms. In simulated environments (primarily Habitat-sim), the system executes approximately 10 spatial memory update cycles per second during exploration. Scene-scale exploration typically requires 5 to 10 minutes, depending on the navigable area. BSC-Nav operates at approximately 10 Hz during task execution in the Habitat simulator. LLMs and MLLMs (GPT-4, GPT-4o, and o3-mini) are invoked exclusively for high-level planning tasks, such as candidate waypoint selection, target association augmentation, and instruction decomposition. These API-based calls are executed asynchronously at a low frequency, ensuring they neither exhaust local computational resources nor impede real-time low-level control due to network latency. In our physical deployment, BSC-Nav employs a decoupled high- and low-frequency system architecture to ensure real-time responsiveness. High-frequency low-level control ( > 30 Hz), encompassing chassis motion control, obstacle avoidance, and real-time localization via LiDAR-based SLAM, operates directly on the robot’s edge computing unit. Low-to-medium-frequency perception and spatial memory updates ( ~ 10 Hz), including visual feature extraction (DINOv2 and YOLO-World), coordinate projection, and cognitive map updates, are offloaded to an industrial host PC (Fig. 5a). During task execution, the robot navigates at an average speed of 0.76 m/s, governed by heuristic path planning and real-time motion control (Fig. 5f). Real-time (1 × speed) navigation demonstrations on the physical robot are provided in Supplementary Movies 3–7. Foundation model specifications BSC-Nav seamlessly integrates multiple foundation models across its perception, reasoning, and generation stages. For visual feature extraction within the cognitive map module, we employ DINOv2 with registers (ViT-L/14)34,35, a self-supervised vision transformer that yields 1,024-dimensional patch-level features. For open-vocabulary object detection in the landmark memory module, we utilize YOLO-World-X (YOLOv8 Extra-Large, WorldV2)73 with a confidence threshold of 0.55 across 20 predefined object categories. For text-image semantic matching during target verification, we adopt CLIP (ViT-H/14, pre-trained with MetaCLIP FullCC)83. For visual imagination in association-augmented retrieval, we leverage Stable Diffusion 3.5 Medium43, generating images at a 512 × 512 resolution using 28 inference steps, a guidance scale of 7.0, and a maximum sequence length of 512 tokens. To drastically reduce GPU memory consumption, the diffusion model is quantized to 4-bit NormalFloat (NF4) and evaluated with bfloat16 precision, with CPU offloading. For high-level semantic reasoning and planning, BSC-Nav employs three OpenAI models accessed via API, all utilizing default inference parameters (temperature = 1.0, top-p = 1.0): GPT-4 (gpt-4-0613) for landmark memory retrieval and spatial reasoning; GPT-4o (gpt-4o-2024-11-20)42 for scene understanding, target verification, and description augmentation; and o3-mini57 for complex instruction decomposition in long-horizon navigation tasks. As these are closed-source proprietary models, their precise architectural details, parameter counts, and training data compositions remain undisclosed by OpenAI. To strictly facilitate reproducibility, we report the exact model version identifiers above, and all prompt templates are provided in Supplementary Sec. A and in our code repository. Baseline methods and comparison End-to-end navigation methods leverage deep neural networks to directly map egocentric observations to action sequences, implicitly encoding spatial-geometric priors and semantic knowledge within network parameters. While they streamline the decision-making process, they must simultaneously learn spatial memory tracking and action planning, necessitating extensive trajectory data or expert demonstrations for effective training. For instance, PixNav 11 performs greedy navigation by predicting optimal actions toward salient pixels in the current view. DAgRL 49 and PSL 50 integrate pre-trained vision-language encoders with historical action embeddings or contrastive learning to align visual observations with textual goals. Despite their strong performance in simulated environments, these methods are highly susceptible to data distribution shifts between training and testing environments, severely impairing their generalization to real-world physical deployments. Recent vision-language-action models 12,84,85 attempt to address these limitations by scaling model capacity and diversifying training trajectories. Uni-NavId exemplifies this trend, achieving robust performance in parsing complex multi-modal instructions. However, these large-scale models incur significant computational overhead, restrict action generation frequency, and fundamentally lack mechanisms for persistent, structured spatial memory integration. Unlike these frameworks, BSC-Nav leverages the explicit retrieval mechanism of its working memory to achieve a complete decoupling of high-level semantic goal planning from low-level physical motion control, thereby paving the way toward genuine zero-shot, general-purpose embodied navigation. Modular and map-based navigation methods represent a growing class of approaches that construct explicit spatial representations to support long-horizon planning and mitigate sim-to-real gaps. The necessity of internal spatial representation has been recognized since early computational models, such as the RNN-based dynamic forward models of Tani 86, which achieved sequential planning and symbol grounding without explicit global coordinates. However, limited by the perceptual technologies of their era, such models could not scale to semantically rich, large-scale scenarios. Driven by advances in foundation models, recent methods have increasingly adopted explicit memory structures. For example, GOAT36 and MOD-IIN54 project closed-set semantics or visual features onto top-down maps, while VLFM13 and UniGoal33 construct probabilistic frontier maps or abstract graph-based representations. More recently, GridMM30 builds a top-down dense grid memory map that aggregates fine-grained visual features via cross-modal attention (essentially modeling localized route knowledge), whereas ETPNav32 dynamically evolves sparse topological nodes in continuous environments (leaning towards a reasoning-capable landmark representation). While these methods advance beyond purely reactive processing, they predominantly suffer from incomplete spatial memory by relying on a single-granularity representation, either purely dense grids or purely sparse topologies. Furthermore, they heavily depend on task-specifically trained cross-modal Transformers or policies for memory attention and planning, which inherently constrain their generalization to unseen environments. BSC-Nav fundamentally distinguishes itself in both representational granularity and retrieval paradigms. It operationalizes a brain-inspired tripartite decomposition, preventing the semantic blurring typical of monolithic integration. Moreover, by introducing a transparent working memory interface equipped with visual imagination and MLLM-based logical reasoning, BSC-Nav replaces black-box policy attention with explicit, zero-shot inference. This allows the agent to flexibly align complex semantic intents with accumulated spatial structures without relying on heavy instruction-level fine-tuning, achieving robust spatial cognition across tasks and environments. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Supplementary information Supplementary Informations Description of Additional Supplementary Files Supplementary Movie 1 Supplementary Movie 2 Supplementary Movie 3 Supplementary Movie 4 Supplementary Movie 5 Supplementary Movie 6 Supplementary Movie 7 Supplementary Movie 8 Supplementary Movie 9 Supplementary Movie 10 Reporting Summary Source data Source Data Author contributions S.R., L.W. and H.S. conceived the study. S.R., L.W. and S.L. designed the computational model. S.R. and C.K. performed the main experiments, assisted by Q.Z. S.R. and Q.Z. completed the design of the embodied robot and algorithm deployment. L.W. and S.R. drafted the paper. H.S. and X.W. supervised the project. H.S. offered funding and critically revised the paper. All authors read and approved the final version. Peer review Peer review information Nature Communications thanks Giulia D’Angelo, Daniele Gatti and Takashi Ikegami for their contribution to the peer review of this work. A peer review file is available. Funding This work was supported by the NSFC Projects (Nos. 92370124, U25B6003, 62350080, 62576020, 62406160), the Beijing Natural Science Foundation (No. L247011), the Shandong Provincial Natural Science Foundation (No. ZR2022ZD01), the Beijing Major Science and Technology Project (No. Z251100008425003), the High Performance Computing Center, Tsinghua University, and the Fundamental Research Funds for the Central Universities. Data availability All simulation scene data and evaluation benchmarks used in this study are publicly available through open-source platforms or datasets. The MP3D and HM3D scene data can be accessed through the Habitat-sim repository (https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md). Navigation benchmarks including object-goal navigation, image-instance navigation, and VLN-CE R2R datasets are available through Habitat-lab (https://github.com/facebookresearch/habitat-lab/blob/main/DATASETS.md). The open-vocabulary object navigation benchmark and configuration files are from Yokoyama et al49. (https://github.com/naokiyokoyama/ovon). Text-instance navigation data and configurations are from Sun et al50. (https://github.com/XinyuSun/PSL-InstanceNav). Active embodied question-answering data are from the OpenEQA dataset by Majumdar et al60. (https://github.com/facebookresearch/open-eqa). Source data are provided with this paper. Code availability The BSC-Nav implementation for both simulation and real-world experiments, along with data analysis and visualization scripts, is publicly available on GitHub (https://github.com/Heathcliff-saku/BSC-Nav) and on Zenodo87. The repository is organized into two branches: the sim-branch contains simulation implementations, benchmark configurations, and evaluation scripts; the phy-branch includes low-level control interfaces adapted for our robotic platform and BSC-Nav deployment scripts. Researchers can refer to these implementations to deploy BSC-Nav on custom-built embodied agents. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors contributed equally: Shouwei Ruan, Liyuan Wang. Contributor Information Xingxing Wei, Email: xxwei@buaa.edu.cn. Hang Su, Email: suhangss@mail.tsinghua.edu.cn. Supplementary information The online version contains supplementary material available at 10.1038/s41467-026-74358-5. References Associated Data Supplementary Materials Supplementary Informations Description of Additional Supplementary Files Supplementary Movie 1 Supplementary Movie 2 Supplementary Movie 3 Supplementary Movie 4 Supplementary Movie 5 Supplementary Movie 6 Supplementary Movie 7 Supplementary Movie 8 Supplementary Movie 9 Supplementary Movie 10 Reporting Summary Source Data Data Availability Statement All simulation scene data and evaluation benchmarks used in this study are publicly available through open-source platforms or datasets. The MP3D and HM3D scene data can be accessed through the Habitat-sim repository (https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md). Navigation benchmarks including object-goal navigation, image-instance navigation, and VLN-CE R2R datasets are available through Habitat-lab (https://github.com/facebookresearch/habitat-lab/blob/main/DATASETS.md). The open-vocabulary object navigation benchmark and configuration files are from Yokoyama et al49. (https://github.com/naokiyokoyama/ovon). Text-instance navigation data and configurations are from Sun et al50. (https://github.com/XinyuSun/PSL-InstanceNav). Active embodied question-answering data are from the OpenEQA dataset by Majumdar et al60. (https://github.com/facebookresearch/open-eqa). Source data are provided with this paper. The BSC-Nav implementation for both simulation and real-world experiments, along with data analysis and visualization scripts, is publicly available on GitHub (https://github.com/Heathcliff-saku/BSC-Nav) and on Zenodo87. The repository is organized into two branches: the sim-branch contains simulation implementations, benchmark configurations, and evaluation scripts; the phy-branch includes low-level control interfaces adapted for our robotic platform and BSC-Nav deployment scripts. Researchers can refer to these implementations to deploy BSC-Nav on custom-built embodied agents.