SemNav: A Model-Based Planner for Zero-Shot Object Goal Navigation Using Vision-Foundation Models

Arnab Debnath, Gregory J. Stein, Jana Košecká · 2025

Read SemNav: A Model-Based Planner for Zero-Shot Object Goal Navigation Using Vision-Foundation Models by Arnab Debnath, Gregory J. Stein, Jana Košecká (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03516.txt.

Introduction

Arnab Debnath, Gregory Stein, and Jana Košecká sent a robot hunting for objects in unseen homes in a 2025 paper, with no task training. Object-goal navigation usually needs masses of labeled scenes or long reinforcement learning in simulation, and learned habits often fail in new layouts. Their SemNav system instead reasons on the spot, pairing vision foundation models, which read scenes and guess where a target likely hides, with a model-based planner.

The planner explores frontiers, the edges of the known map, with long-horizon decisions rather than reactive wandering. On a standard three-dimensional home dataset in simulation, the method reached state-of-the-art success weighted by path length among zero-shot systems. Foundation perception plus explicit planning scales to new buildings gracefully.

Catalog record

CREATORArnab Debnath, Gregory J. Stein, Jana Košecká
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03516

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SemNav: A Model-Based Planner for Zero-Shot Object Goal Navigation Using Vision-Foundation Models Arnab Debnath, Gregory J. Stein, Jana Košecká Abstract Object goal navigation is a fundamental task in embodied AI, where an agent is instructed to locate a target object in an unexplored environment. Traditional learning-based methods rely heavily on large-scale annotated data or require extensive interaction with the environment in a reinforcement learning setting, often failing to generalize to novel environments and limiting scalability. To overcome these challenges, we explore a zero-shot setting where the agent operates without task-specific training, enabling more scalable and adaptable solution. Recent advances in Vision Foundation Models (VFMs) offer powerful capabilities for visual understanding and reasoning, making them ideal for agents to comprehend scenes, identify relevant regions, and infer the likely locations of objects. In this work, we present a zero-shot object goal navigation framework that integrates the perceptual strength of VFMs with a model-based planner that is capable of long-horizon decision making through frontier exploration. We evaluate our approach on the HM3D dataset using the Habitat simulator and demonstrate that our method achieves state-of-the-art performance in terms of success weighted by path length for zero-shot object goal navigation. 1 Introduction Object-goal navigation is a core challenge in embodied AI, where an agent is tasked with locating and navigating to an instance of a specified object within an unfamiliar environment. Unlike map-based or point-goal navigation, this task requires both semantic understanding and active perception, as the agent must interpret visual cues to identify the target object without prior knowledge of its location. This capability is essential for a wide range of real-world applications, including service robots in domestic environments, assistive robots for the elderly, and search-and-rescue operations in disaster-stricken areas. Over the years, various learning-based methods have been proposed for object-goal navigation. Supervised approaches Ramakrishnan et al. 2022 rely on large-scale annotated datasets, while reinforcement learning (RL) techniques Chaplot et al. 2020 ; Pal et al. 2021 ; Yadav et al. 2023 learn policies through trial-and-error guided by reward shaping. However, these methods are limited to closed sets of object categories and are tightly coupled to their training environments. Extending them to new objects or environments often requires costly retraining, hindering scalability and real-world deployment where generalization to unseen categories and dynamic layouts is essential. Recent advances in VFMs have greatly enhanced the generalization capabilities of perception systems. Open-vocabulary detectors Li et al. 2022a ; Liu et al. 2024 and segmentation models Dong et al. 2023 ; Kirillov et al. 2023 can recognize diverse object categories without a fixed training set, while vision-language models (VLM) and large language models (LLM) provide strong commonsense reasoning about objects and their spatial contexts. VLFM Yokoyama et al. 2023 leverages BLIP-2 Li et al. 2023a to compute the cosine similarity between the current observation and a target image, guiding the agent toward visually similar regions. ESC Zhou et al. 2023 utilizes GLIP Li et al. 2022a for detecting candidate objects and rooms, and employs an LLM to reason about the likelihood of the target object being present in a specific room or near certain objects. While these approaches show promise, their reliance on image-text matching or reasoning explicitly with discovered objects can hinder exploration when the target object is far away, occluded, or when the detection is poor. In this work, we explore the advancements in VFMs that enables us to use more general prompts which can encourage the agent to move through a hallway or enter a nearby room when the object is not currently visible. Figure 1 : Overview of SemNav Our approach adopts a frontier-based exploration strategy, treating frontiers as long-term subgoals. We leverage VLM to assign the likelihood of finding the target object to these frontiers, guiding the agent’s search. Many existing frontier-based methods are inherently myopic: they score each frontier using a heuristic and greedily select the one with the highest score, often leading to suboptimal and inefficient behavior in complex environments. To address this, we build upon the model-based frontier planning strategy, Learning Over Subgoals Planner (LSP) Stein et al. 2018 , which evaluates the expected navigation cost of selecting a frontier given the current belief about the environment. This allows the agent to reason about the long-term consequences of its actions and supports more efficient long-horizon planning. Our contributions are summarized below: • We propose a semantic frontier-based exploration framework, SemNav, that leverages VLM to assign cost to frontiers for zero-shot object-goal navigation in unseen environments. • We integrate a non-greedy, model-based frontier planning strategy (LSP) to reason about long-term exploration costs, enabling more efficient and informed decision-making compared to traditional greedy frontier selection methods. 2 Related Work Zero-shot object-goal navigation leverages VLM and LLMs to enable open-vocabulary object understanding and semantic reasoning without task-specific training. Recent works explore various strategies, including transfer learning Al-Halah et al. 2022 ; Majumdar et al. 2022 , frontier scoring with VLM and LLMs Yokoyama et al. 2023 ; Yu et al. 2023 , structured scene reasoning Zhou et al. 2023 , and topological exploration guided by LLM Wu et al. 2024 . 3 Approach An agent is spawned at a random location within an unknown environment and receives a target object category (e.g., couch). At each time step, the agent receives visual observations as egocentric RGB-D images and utilizes a GPS+Compass sensor. The agent can take several actions at each step: MOVE_FORWARD, TURN_LEFT, TURN_RIGHT, or STOP . The MOVE_FORWARD action moves the agent forward by {0.25m} , and the TURN actions rotate the agent by 30\degree in the specified direction. An episode is considered successful if the agent takes a STOP action within 1m of any instance of the target object category within 500 timesteps. Overview At each timestep t , the robot receives an RGB-D observation. The depth image is used to update a partial map of the environment, identifying both navigable areas and obstacles. Frontiers are extracted as potential exploration targets. To guide exploration, a VFM model (GPT-4o) is queried using the current RGB image and the target object. The model estimates the likelihood of finding the target object in the direction of the current observation. This score is associated with the frontier. A long-horizon global planner LSP Stein et al. 2018 , selects the most promising frontier that minimizes the expected cost of navigation. A local navigation module then computes a low-level action to move the agent toward the selected frontier. Once the target object is detected, the local navigation module guides the robot to its location and issues a STOP action ( Fig. 1 ). 3.1 Mapping module We maintain an egocentric, partial map of the environment that is updated incrementally as the agent explores. At each timestep, the depth image is converted into a 3D point cloud, from which free space and obstacles are identified using a height threshold. Frontiers are extracted as boundaries between free and unknown space. For each frontier, the center point along its boundary is selected as the navigation target. In addition to the geometric map, we also construct a two-channel value map, inspired by VLFM Yokoyama et al. 2023 , consisting of a semantic value, V and a confidence score map, C . The

Read the complete text