STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigation
Haokun Zhu, Zongtai Li, Zhixuan Liu, Wenshan Wang, Ji Zhang, Jonathan Francis, Jean Oh · 2025
Read STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigation by Haokun Zhu, Zongtai Li, Zhixuan Liu, Wenshan Wang, Ji Zhang, Jonathan Francis, Jean Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06729.txt.
Introduction
Zhu and six co-authors present a study from twenty twenty-five on guiding robots to find objects indoors. Image-and-text models bring rich prior knowledge to this search task. Yet they struggle to grasp a full room layout, and asking them at each step wastes time. The team's system builds a layered map as the robot moves. The map holds viewpoints, object nodes, and room nodes in a clear structure. Search then runs in two stages, high-level planning and close exploration.
Tests on three fake indoor benchmarks gave top scores for success and path thrift. Gains held on both counts over prior methods. The team also ran the system on a real robot in ten rooms. It handled fifteen search tasks with strong steadiness. The authors back layered maps for robot search.
Catalog record
| CREATOR | Haokun Zhu, Zongtai Li, Zhixuan Liu, Wenshan Wang, Ji Zhang, Jonathan Francis, Jean Oh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigation Haokun Zhu, Zongtai Li, Zhixuan Liu, Wenshan Wang, Ji Zhang, Jonathan Francis, Jean Oh Abstract Vision-Language Models (VLMs) have been increasingly integrated into object navigation tasks for their rich prior knowledge and strong reasoning abilities. However, applying VLMs to navigation presents two key challenges: effectively parsing and structuring complex environment information and determining when and how to query VLMs. Insufficient environment understanding and over-reliance on VLMs (e.g. querying at every step) can easily lead to unnecessary backtracking and reduced navigation efficiency, especially in large continuous environments. To address these challenges, we propose a novel framework that incrementally constructs a multi-layer environment representation consisting of viewpoints, object nodes, and room nodes during navigation. Viewpoints and object nodes facilitate intra-room exploration and accurate target localization, while room nodes support efficient inter-room planning. Building on this structured representation, we propose a novel two-stage navigation policy, integrating high-level planning guided by VLM reasoning with low-level VLM-assisted exploration to efficiently and reliably locate a goal object. We evaluated our approach on three simulated benchmarks (HM3D v1&v2 , RoboTHOR, and MP3D), and achieved state-of-the-art performance on both the success rate (SR% ↑ 13.1 % \mathord{\uparrow}\,13.1\% ) and navigation efficiency (SPL% ↑ 6.2 % \mathord{\uparrow}\,6.2\% ). We further validate our method on a real robot platform, demonstrating strong robustness across 120 episodes in 10 different indoor environments. Project page is available at https://zwandering.github.io/STRIVE.github.io/ . Fig. 1: STRIVE can conduct zero-shot object navigation in diverse and complex real-world environments by leveraging our novel multi-layer representation and efficient two-stage navigation policy. † † footnotetext: ∗ Equal contribution. I Introduction Object navigation is a fundamental task in robotics, where an agent must locate an instance of a given object category in unknown environments. This task is particularly challenging, as it requires the agent to understand complex visual information, reason about spatial relationships, and make decisions based on both current and past observations. Advances in Vision-Language Models (VLMs) [ 1 , 2 , 3 ] have demonstrated strong capabilities in contextual visual understanding and common-sense reasoning. Building on this, recent works [ 4 , 5 , 6 , 7 , 8 ] have integrated VLMs into object navigation tasks, utilizing their rich prior knowledge, visual understanding, and commonsense reasoning abilities to guide navigation. However, existing approaches often face two significant challenges: First, the input to VLMs typically lacks a structured representation of the environment and is often restricted to local observations. Without a coherent global view that integrates both current and previous observations, VLMs struggle to reason effectively about the environment and fail to make reasonable navigation decisions. Second, existing methods [ 6 , 8 ] typically rely on VLMs to select among all frontier viewpoints at each step, without utilizing navigation progress or environment layouts to effectively guide VLMs’ reasoning process. Besides, due to VLMs’ limited understanding of 3D spatial information [ 9 , 10 , 11 ] , they cannot jointly reason about the spatial relationships and the navigation history when evaluating each viewpoint. As a result, their evaluation of viewpoints is largely based on viewpoints’ local semantic information, which often leads to redundant navigation behaviors such as backtracking or repeated exploration. To address these challenges, we propose STRIVE ( ST ructured R epresentation I ntegrating V LM Reasoning for E fficient Object Navigation), a novel framework that incrementally learns a structured representation of the environment and utilizes VLMs’ reasoning abilities to guide the navigation. This representation consists of 3 layers: object nodes, viewpoint nodes, and room nodes. Object nodes represent all observed objects, provide rich semantic information about the environment and assist in target localization; Viewpoint nodes discretize the environment into a set of key locations, facilitating structured and efficient intra-room exploration ; Room nodes further segment the environment into distinct rooms and facilitate room-level reasoning by the VLM. This multi-layer representation enables a more comprehensive understanding of the environment, allowing VLM to better utilize their reasoning abilities for more effective decision-making. Furthermore, we design an efficient two-stage navigation policy based on this representation, combining high-level planning guided by the VLM’s reasoning and VLM-assisted low-level exploration. Specifically, for the high-level planning, instead of making step-by-step decisions among all viewpoint nodes, the VLM selects the next room to explore based on the spatial layout and semantic information of each room. For low-level exploration within rooms, we employ a traditional frontier-based algorithm for efficient exploration, while leveraging the VLM to decide whether continued exploration of the current room is necessary . Making high-level planning on rooms effectively mitigates the issue of VLMs’ insufficient 3D spatial understanding and prevents redundant actions, thereby enhancing navigation efficiency. We evaluate our method on four widely-used simulated benchmarks: HM3D-v1 [ 12 ] , HM3D-v2 [ 13 ] , RoboTHOR [ 14 ] , and MP3D [ 15 ] . STRIVE achieves state-of-the-art (SOTA) results, significantly outperforming 12 existing methods in both Success Rate (SR) and navigation efficiency, measured by Success weighted by Path Length (SPL). This highlights the effectiveness of our proposed multi-layer representation and the VLM-guided reasoning policy in improving object navigation. Specifically, STRIVE achieves 62.9% SR and 34.2% SPL on HM3D-v1, 79.6% SR and 38.7% SPL on HM3d-v2 , 68.1% SR and 36.3% SPL on RoboTHOR, and 52.3% SR and 23.1% SPL on MP3D. Besides, we also conduct 15 real-world experiments across 10 different indoor environments on a Mecanum wheel platform [ 16 ] , demonstrating the effectiveness and robustness of our method in real-world scenarios. II Related Works Fig. 2: Overview of STRIVE. We construct a multi-layer representation \mathcal{R} ( Sec. III-A ) on-the-fly, consisting of object, viewpoint, and room nodes, which serves as a structured input for VLM. Based on \mathcal{R} , we introduce a two-stage navigation policy, where the VLM reasons and plans at room-level ( Sec. III-B2 ), while the agent explores in room at the viewpoint-level using a VLM-assisted frontier-based navigation strategy ( Sec. III-B1 ) and VLM-based target verification ( Sec. III-B3 ). Object Navigation. Existing object navigation methods are typically categorized into end-to-end learning approaches and modular approaches. End-to-end methods [ 17 , 18 , 19 , 20 , 21 , 22 , 23 ] use reinforcement learning to directly map observations to actions, but often suffer from low sample efficiency and poor generalization. In contrast, modular methods [ 24 , 25 , 26 , 27 , 28 , 24 , 4 , 29 ] decompose navigation into steps such as mapping, planning, and action execution, and often build semantic maps in bird’s-eye view or 3D space to facilitate more interpretable and scalable navigation behavior. With the emergence of foundation models [ 1 ] , object navigation has advanced towards zero-shot, open-vocabulary setting [ 27 , 28 , 30 , 4 , 29 ] . We also leverage VLM’s reasoning abilities to improve zero-shot object navigation. but we employ a novel representation and a two-stage policy, enabling more efficient and effective VLM guidance. VLM-guided Navigation. With internet-scale training data, Vision-Language Models (VLMs) [ 31 , 1 , 2 , 3 ] have shown strong commen-sense reasoning abilities and have been widely applied in Object Navigation tasks to guide the decision-making process. For example, InstructNav