DialNav: Multi-turn Dialog Navigation with a Remote Guide

Leekyeung Han, Hyunji Min, Gyeom Hwangbo, Jonghyun Choi, Paul Hongsuck Seo · 2025

Read DialNav: Multi-turn Dialog Navigation with a Remote Guide by Leekyeung Han, Hyunji Min, Gyeom Hwangbo, Jonghyun Choi, Paul Hongsuck Seo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12894.txt.

Introduction

Leekyeung Han, Hyunji Min and three co-authors launch DialNav in 2025, a joint task where a mover and a far guide talk in turns to reach a goal. Unlike past tasks, the guide must guess where the mover is, so talk is key, not extra.

To back the task they share RAIN, a set of human talks paired with paths through photo-true scenes. A full test suite grades both moves and talk. Trials with varied mover and guide models show where the task still strains systems. Data, code, and tests are public so others can build on the task.

Catalog record

CREATORLeekyeung Han, Hyunji Min, Gyeom Hwangbo, Jonghyun Choi, Paul Hongsuck Seo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12894

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DialNav: Multi-turn Dialog Navigation with a Remote Guide Leekyeung Han, Hyunji Min, Gyeom Hwangbo, Jonghyun Choi, Paul Hongsuck Seo Abstract We introduce DialNav, a novel collaborative embodied dialog task, where a navigation agent (Navigator) and a remote guide (Guide) engage in multi-turn dialog to reach a goal location. Unlike prior work, DialNav aims for holistic evaluation and requires the Guide to infer the Navigator’s location, making communication essential for task success. To support this task, we collect and release Remote Assistance in Navigation (RAIN) dataset, human-human dialog paired with navigation trajectories in photorealistic environments. We design a comprehensive benchmark to evaluate both navigation and dialog, and conduct extensive experiments analyzing the impact of different Navigator and Guide models. We highlight key challenges and publicly release the dataset, code, and evaluation framework to foster future research in embodied dialog. Our code and dataset are available at: https://happilee12.github.io/DialNav . Figure 1 : Overview of the proposed DialNav task. Navigator is tasked with reaching the goal region \mathcal{R} (yellow circle in the map) from the initial node b based on an ambiguous initial instruction I , which provides only a hint about \mathcal{R} ( e.g. , ‘ The goal room contains a carpet ’). During navigation, Navigator can ask questions to obtain additional guidance (orange text boxes). Guide has knowledge of the environment but lacks information of Navigator’s location. For successful navigation, Guide must estimate Navigator’s location through dialog \mathcal{D} and provide directions from the estimated position to the goal region (blue text boxes). Note that each QA pair in \mathcal{D} is mapped to a node (red and purple nodes mapped to dialog turns by dotted arrows) in the navigation trajectory \mathcal{T} . 1 Introduction An embodied AI agent with a physical body perceives and interacts with the world, responding to human instructions or environmental stimuli. Since an embodied agent physically engages with its surroundings, misinterpreting human commands or hallucinating unintended actions can cause inconvenience or even physical harm. To mitigate these risks, the agent must seek clarification when faced with ambiguous tasks. A natural approach is dialog-based interaction, allowing the agent to refine its understanding before acting. This enhances both safety and effectiveness in task execution. Despite the importance of communicative ability in embodied AI, its inherent challenges hinders progress. First, collecting dialog-based data is costly, as it requires two individuals to engage in real-time interaction while performing the task. Secondly, a dialog-based task necessitates not only equipping the agent with the ability to ask questions but also constructing a counterpart model capable of providing responses. Additionally, a framework that enables task execution through question-and-answer interactions is required. Lastly, the dynamic and interdependent nature of the task makes it difficult to evaluate performance. Due to these challenges, prior research has predominantly focused on task execution rather than leveraging dialog for task completion. Some studies have already acknowledged the importance of dialog in embodied AI and has explored tasks involving dynamic additional instructions from a guide [ 23 , 24 , 31 , 13 , 25 ] . However, these studies often assume that the guide has full knowledge of current situation—an unrealistic assumption in practical settings. This omniscient guide model reduces the incentive for the guide to carefully consider the performer’s questions and, in turn, discourages the performer from formulating high-quality inquiries. Even a vague request like “help” can elicit a perfect response from an all-knowing guide, making the dialog less meaningful. To address this limitation, we propose DialNav. DialNav is a cooperative navigation task between Navigator and Remote Guide, where Guide is not aware of Navigator’s location but only familiar with the environment. This setting closely mirrors real-world scenarios, such as when a lost person calls a friend for directions. Here, Navigator must formulate high-quality, detailed questions, as the effectiveness of Guide’s guidance depends on the clarity of the queries. Likewise, Guide is incentivized to pay close attention to Navigator’s questions, as they lack direct awareness of Navigator’s location. Some prior works [ 2 , 9 ] have explored non-omniscient guides, but under restricted conditions— [ 2 ] evaluates agents using static dialog histories in a physical maze, while [ 9 ] limits the guide’s view to a 2D grid map, reducing spatial fidelity. In contrast, DialNav supports long-horizon dialog and navigation in realistic indoor environments, enabling richer and more purposeful communication. To support this task, we introduce the Remote Assistance in Navigation dataset (RAIN), a visual navigation dataset with multi-turn human-human dialog. To this end, we developed a data collection tool, specifically designed to capture complex interactions in DialNav. As RAIN dataset can be used to train and evaluate the core capabilities of Navigator and Guide under standardized conditions, we conduct experiments with Navigator and Guide models across various setups and module configurations, providing valuable insights into this new task. We will publicly release the complete codebase, data collection tool, dataset, and training and evaluation frameworks, ensuring reproducibility and providing foundational resources for future research. Our main contributions are summarized as follows: • We introduce a cooperative embodied dialog navigation task involving Navigator and remote Guide. • We collect and release RAIN, which captures human-human interactions for DialNav. • Navigator and Guide are trained and tested under various setups within a holistic dialog-and-navigation pipeline. • We discuss key challenges and their implications for fostering future research. 2 Related Work Embodied Dialog Recent advancements in embodied dialog have led to the development of new benchmarks and datasets across question answering [ 7 , 21 ] , navigation [ 31 , 10 ] , and manipulation [ 13 , 25 , 30 , 22 ] . Although these benchmarks incorporate dialog, they either assume an omniscient guide [ 28 , 23 , 13 , 25 , 22 ] or restrict evaluation to task execution only [ 31 , 2 , 15 ] , thereby underestimating the role of dialog ( e.g. , asking for help without a question [ 23 ] ). Our work models two dialog-enabled agents, Navigator and remote Guide, and emphasizes the importance of dialog generation and understanding within the full communication-action loop. The most closely related work is Talk the Walk [ 9 ] but it operates in a highly constrained setting with limited agent actions and simplified 2D grid map for the guide. In contrast, our task supports long-horizon, multi-turn dialog in a realistic navigation environments. Vision and Language Navigation Vision-and-Language Navigation (VLN) is a prominent multimodal embodied task where an agent navigates a visual environment based on language instructions. Anderson et al. 2018 pioneered the field by introducing the first photo-realistic VLN task, dataset, and a simulator. Since then, a wide range of VLN benchmarks have emerged, but they often face limitations, either providing overly detailed instructions [ 17 , 18 ] unnatural for human interaction or offering ambiguous guidance [ 27 , 40 ] insufficient for successful navigation. To overcome these limitations, this study explores a realistic dialog-based navigation task. Subtasks for DialNav DialNav targets the holistic evaluation of the communication-and-navigation process, integrating multiple subtasks that have previously been addressed only in isolation. Instruction generation [ 12 , 41 , 38 , 33 , 36 , 35 , 4 , 11 ] , explored primarily

Read the complete text