HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environment
Zecheng Yin, Hao Zhao, Zhen Li · 2025
Read HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environment by Zecheng Yin, Hao Zhao, Zhen Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22917.txt.
Introduction
Zecheng Yin, with Hao Zhao and Zhen Li, looks at robots that must find a named object in a place they have never seen. Cameras give close views, while top-down maps give a wider sense of the layout. Most past systems used only one of those views.
The authors build HyPerNav, which joins both views with help from vision-language models. Those models link what a robot sees to words for objects and rooms. Tests show the combined view helps the robot search with more sense and fewer wasted moves. Readers get a clear plan for robots that must explore homes or offices on their own.
Catalog record
| CREATOR | Zecheng Yin, Hao Zhao, Zhen Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22917
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environment Zecheng Yin, Hao Zhao, Zhen Li Abstract Objective-oriented navigation(ObjNav) enables robot to navigate to target object directly and autonomously in an unknown environment. Effective perception in navigation in unknown environment is critical for autonomous robots. While egocentric observations from RGB-D sensors provide abundant local information, real-time top-down maps offer valuable global context for ObjNav. Nevertheless, the majority of existing studies focus on a single source, seldom integrating these two complementary perceptual modalities, despite the fact that humans naturally attend to both. With the rapid advancement of Vision-Language Models(VLMs), we propose Hybrid Perception Navigation (HyPerNav), leveraging VLMs’ strong reasoning and vision-language understanding capabilities to jointly perceive both local and global information to enhance the effectiveness and intelligence of navigation in unknown environments. In both massive simulation evaluation and real-world validation, our methods achieved state-of-the-art performance against popular baselines. Benefiting from hybrid perception approach, our method captures richer cues and finds the objects more effectively, by simultaneously leveraging information understanding from egocentric observations and the top-down map. Our ablation study further proved that either of the hybrid perception contributes to the navigation performance. The code and datasets are publicly available. I Introduction Navigating to target objective from human language is a key ability for fully autonomous robots. Robots can accomplish amazing human language following with assisting spatial information in instructions or map such as Vison Language Navigation in Continuous Environment (VLN-CE) [ 10 , 18 , 13 , 34 ] . However, the environment prior from instructions or pre-constructed map limit the autonomy and generalization of navigation. We focus on Object-oriented navigation (ObjNav). Unlike the above, ObjNav requires the robot to directly moves to target in unknown environments given only the object goal text without any prior to the environment, thus demanding more on spatial perception, clue discovery and common sense reasoning in the unexplored environment. The evaluation of such task is the navigation success rate (SR) and Success weighted by Path Length(SPL) [ 1 ] . The success rate implies the possibility the robot successfully navigates to target objects and the SPL implies how close the robot trajectory to the shortest path to the objective. To address the Object Navigation (ObjNav) problem, various approaches have been proposed, including classical exploration methods, learning-based methods, and training-free methods. Classical approaches, such as frontier-based exploration [ 25 ] , provide a basic baseline: the robot systematically traverses unknown areas while using a target detector to identify the goal. Once the target is detected during exploration, the robot navigates to it—examples include [ 3 ] . However, this strategy often requires extensive exploration time and is prone to failure in large or complex environments. Learning-based methods employ reinforcement learning (RL) to improve the effectiveness of navigation. These approaches train an end-to-end navigation policy via dataset before deploying it in evaluation scenarios, as exemplified by [ 4 ] . Recent learning-based models [ 31 , 23 ] also utilize diffusion to predict possible trajectories in egocentric observations to objectives. and its follow-up works. However, such methods face significant challenges in data acquisition and generalization, as they require large amounts of diverse training data—a requirement that currently limits their scalability and real-world applicability. Subsequent work [ 3 , 9 ] , extends the goal modality from text to images and incorporates lifelong memory mechanisms, but does not fundamentally address the core challenges in navigation policy learning. Training-free methods have gained increasing popularity due to the rapid advancements in VLMs and similarity matching models [ 12 , 19 ] . Some approaches leverage the commonsense reasoning capabilities of VLM to rank the frontiers for navigation [ 11 ] . Some [ 5 , 26 ] combine the egocentric observation and objective text prompt as a Vision Question Answering(VQA) problem for VLM to guide the navigation. [ 8 ] follows [ 5 ] in navigation guidance and adds Retrieval-Augmented Generation(RAG) for object memory management. Another line of work [ 27 ] employs the BLIP model [ 12 ] to perform text-image matching in egocentric observation, enabling the robot to select the most promising frontier from local observations for intelligent exploration. Similarly, [ 30 ] applies the same similarity-based strategy but combined all observations onto a global map, allowing for holistic goal localization. Additionally, some efforts focus on training end-to-end large VLMs for embodied navigation [ 6 , 14 , 35 ] ; however, these are currently limited by the scarcity of large-scale datasets and the computational constraints of resource-limited robotic platforms. The mentioned works accomplish remarkable progress, however, they primarily rely on either local, egocentric perception [ 31 , 27 , 26 , 5 , 8 ] or global information derived from top-down navigation maps [ 30 , 4 , 29 ] . Crucially, these approaches fail to organically integrate these two complementary sources of perceptual information. Moreover, several of them suffer from limited generalization capabilities [ 4 , 23 , 14 ] . To address these limitations, we propose Hybrid Perception Navigation (HyPerNav), with the following contributions: • We propose a simple yet effective training-free VLM-based method that leverages hybrid perception, combining local and global perception for effective object-oriented navigation. • We conducted simulation experiments with quantitative results and real-world validation demonstrating state-of-the-art performance. • Navigation intelligence scales with advancing VLMs still far from their potential. Our publicly released code provides a solid platform for future research. II Related Works RGB-D egocentric perception serves as the fundamental sensory input for embodied agents, providing detailed observations of nearby objects, providing cues for goal-directed navigation, enabling collision avoidance, and facilitating the gradual construction of a global navigation map. [ 3 ] detects goal objects using [ 36 ] during classical frontier exploration [ 25 ] and memorizes them along the trajectory. However, its ObjNav strategy relies solely on exhaustive exploration and object detector performance, resulting in inefficient and often ineffective navigation. This egocentric, detection-based exploration paradigm has been advanced by [ 27 , 11 , 28 ] , which leverage text-image similarity models [ 12 , 32 ] or VLMs to rank frontier views by their relevance to the goal, guiding more intelligent exploration. For example, VLFM uses BLIP2 [ 12 ] to compute similarity between a frontier image and a prompt like ”there seems to be a bed ahead”, selecting the most promising direction. This idea also extends to global perception: instead of comparing frontiers, [ 30 ] projects similarity scores onto a top-down map obstacles to guide exploration at the global level. Learning-based methods have also been proposed. [ 23 , 31 ] use diffusion models to generate real-time, egocentric future trajectories in local observation space. In contrast, [ 7 , 29 ] predict target locations or trajectories directly in the top-down map. However, these approaches typically focus on either local or global observations during training and often suffer from slow inference due to the nature of diffusion sampling. End-to-end embodied foundation models [ 14 , 6 ] have emerged, but their development is constrained by the scarcity of diverse, high-quality training data required for robust cross-environment navigation. VLM-guidance-based navigation methods are emerging as a promising direction toward more intelligent object navigation. VLMs take image and language inputs and