DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation

Tianjun Gu, Linfeng Li, Xuhong Wang, Chenghua Gong, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan · 2025

Read DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation by Tianjun Gu, Linfeng Li, Xuhong Wang, Chenghua Gong, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21969.txt.

Introduction

Tianjun Gu and co-authors build a robot guide that finds its way in new homes with no prior map in a 2025 paper. Home robots must plan fine paths and grasp what a room is for at the same time. Fresh vision-language tools help, but their step-by-step views break the flow of space and their memory is loose.

Their system splits the work into two streams, much like the human brain. One stream fuses scene sense with a space map, and the other pairs search with a policy model to pick each move. A safety layer keeps runs both safe and quick. On three home test sets, the system set new best scores for found goals.

Catalog record

CREATORTianjun Gu, Linfeng Li, Xuhong Wang, Chenghua Gong, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21969

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation Tianjun Gu, Linfeng Li, Xuhong Wang, Chenghua Gong, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan Abstract Adaptive navigation in unfamiliar environments is crucial for household service robots but remains challenging due to the need for both low-level path planning and high-level scene understanding. While recent vision-language model (VLM) based zero-shot approaches reduce dependence on prior maps and scene-specific training data, they face significant limitations: spatiotemporal discontinuity from discrete observations, unstructured memory representations, and insufficient task understanding leading to navigation failures. We propose DORAEMON (Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation), a novel cognitive-inspired framework consisting of Ventral and Dorsal Streams that mimics human navigation capabilities. The Dorsal Stream implements the Hierarchical Semantic-Spatial Fusion and Topology Map to handle spatiotemporal discontinuities, while the Ventral Stream combines RAG-VLM and Policy-VLM to improve decision-making. Our approach also develops Nav-Ensurance to ensure navigation safety and efficiency. We evaluate DORAEMON on the HM3D, MP3D, and GOAT datasets, where it achieves state-of-the-art performance on both success rate (SR) and success weighted by path length (SPL) metrics, significantly outperforming existing methods. We also introduce a new evaluation metric (AORI) to assess navigation intelligence better. Comprehensive experiments demonstrate DORAEMON’s effectiveness in zero-shot autonomous navigation without requiring prior map building or pre-training. 1 East China Normal University, 2 Shanghai AI Lab, 3 Shanghai Innovation Institute † † footnotetext: Homepage: https://grady10086.github.io/DORAEMON/ 1 Introduction Adaptive navigation in complex and unseen environments [ 2 ] is a key capability for household service robots. This task requires robots to move from a random starting point to the location of a target object without any prior knowledge of the environment. For humans, navigation appears almost trivial thanks to spatial cognitive abilities honed through evolution. For robots, however, navigation remains a highly challenging problem: it demands not only low-level path planning to avoid obstacles and reach the destination, but also high-level scene understanding to interpret and make sense of the surrounding environment. Most existing navigation methods rely on the construction of prior maps [ 4 ] or require extensive scene-specific data for task-oriented pre-training [ 36 ] . However, in entirely unfamiliar environments, building maps in advance can be time-consuming and labor-intensive, and the scarcity of scene data further limits the practicality of these approaches. Recently, some works [ 14 , 34 ] have begun to explore zero-training and zero-shot navigation strategies based on vision-language models (VLMs). Relying on textual descriptions of the current task, image inputs, and previously observed historical information, these approaches achieve navigation without dependence on environment- or task-specific data, gradually shedding the reliance on scene priors. Although VLM-based zero-shot navigation methods offer a novel perspective for adapting to unfamiliar environments, they still face numerous challenges in practical applications. On the one hand, VLMs typically take task descriptions and observation histories as input [ 34 ] [ 14 ] . However, due to the discrete nature of input image descriptions at each time step, this spatiotemporal discontinuity often makes it difficult for VLMs to understand the relationships between targets and obstacles in complex environments. On the other hand, while many existing navigation systems incorporate some form of memory functionality, most VLM methods [ 30 , 25 , 27 ] adopt a single-step decision paradigm, treating historical information merely as a reference log. Even though End-to-End methods like VLMnav [ 14 ] utilize historical information, they typically store this information in a flat, unstructured manner, which fundamentally limits their ability to perform long-range navigation. Additionally, VLMs sometimes insufficient understanding of task semantics often leads to poor decision-making (e.g., going to look for a TV but finding a computer instead), and the lack of reliable check mechanisms for navigation states frequently results in unreliable behaviors such as spinning in place during navigation tasks. Fig 1 conceptually illustrates limitations of traditional VLN methods and contrasts them with the cognitive-inspired approach of our DORAEMON. Figure 1: (a) Illustrates limitation of typical VLM navigation (red arrow) in an unfamiliar environment. (b) DORAEMON’s cognitive inspiration: a Decentralized Ontology-aware approach with a Dorsal Stream for “What” and a Ventral Stream for “Where”. (c) DORAEMON constructs the Topology Map and uses Hierarchical Semantic-Spatial Fusion(green arrow). Inspired by cognitive science ‘Decentralized Ontology’ principles [ 3 ] suggesting human knowledge is often represented and accessed in a distributed and context-dependent manner rather than via a single monolithic structure, we propose the Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation (DORAEMON), which consists mainly of Ventral Stream and Dorsal Stream. The Ventral Stream processes object identity (“what”) information, while the Dorsal Stream handles spatial (“where”) processing in the human brain. Our decentralized architecture distributes cognitive functions across complementary streams: The Dorsal Stream addresses spatio-temporal discontinuities through a Topology Map and a Hierarchical Semantic-Spatial Fusion, allowing our agent to reason accurately about target-environment relationships. Additionally, the Ventral Stream improves task understanding by utilizing a Retrieval-augmented Generation model (RAG-VLM) and Policy-VLM for navigation. Additionally, DORAEMON features a Nav-Ensurance system that enables agents to autonomously detect and respond to abnormal conditions, such as becoming stuck or blocked during navigation. To evaluate navigation performance more comprehensively, we propose a new metric called the Adaptive Online Route Index (AORI). In summary, the main contributions of this work are: • We propose DORAEMON, a novel adaptive navigation framework inspired by cognitive principles of decentralized knowledge, consisting of ventral and Dorsal Streams, enabling End-to-End and zero-shot navigation in completely unfamiliar environments without pre-training. • We propose the Dorsal Stream, which involves designing a Topology Map and a Hierarchical Semantic-Spatial Fusion Network to effectively manage spatio-temporal discontinuities. Additionally, we introduce the Ventral Stream, incorporating a synergistic reasoning component that combines RAG-VLM for understanding ontological tasks and Policy-VLM for enhanced task comprehension and policy planning. • We develop Nav-Ensurance, which includes multi-dimensional stuck detection and context-aware escape mechanisms. We propose a new evaluation metric called AORI to quantify the efficiency of the agent’s exploration. Our method demonstrates state-of-the-art performance across various navigation tasks. 2 Related Work 2.1 Zero-shot Object Goal Navigation Object navigation methods are broadly supervised or zero-shot. Supervised approaches train visual encoders with reinforcement/imitation learning [ 23 , 17 , 31 , 11 ] or build semantic maps from training data [ 45 , 24 , 49 ] , struggling with novel scenarios due to data dependency. Zero-shot methods address this using open-vocabulary understanding, increasingly leveraging foundation models like Vision-Language Models (VLMs) and Large Language Models (LLMs). LLMs provide commonsense reasoning via object-room correlation [ 41 , 51 , 39 ] , semantic mapping [ 43 ] , and chain-of-thought planning [ 6 , 41 , 33 , 5 ] , while VLMs align visual observations with textual goals. These foundation model-guided techniques include image-based methods mapping targets to visual embeddings [ 38 , 22 , 13 , 1 ] and map-based approaches using frontier [ 50 , 45 , 51 , 10 , 19 , 43 , 33 ] or waypoint-based maps [ 39 ] with LLM/VLM reasoning. VLM-based strategies either use VLMs for recognition with traditional planning and extra perception models [ 27 , 46 , 21 , 44 , 37 ] , or, like PIVOT [ 25 ] and VLMnav [

Read the complete text