WalkGPT: Grounded Vision–Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation
Rafi Ibn Sultan, Hui Zhu, Xiangyu Zhou, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu · 2026
Read WalkGPT: Grounded Vision–Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation by Rafi Ibn Sultan, Hui Zhu, Xiangyu Zhou, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10703.txt.
Introduction
Rafi Ibn Sultan and co-authors introduce WalkGPT, a vision-language model for grounded pedestrian navigation help, aimed at accessibility. Existing models describe scenes but hallucinate objects and judge depth poorly, which is risky for guidance. WalkGPT answers a navigation query in conversation while producing masks for accessible and harmful features, plus relative depth, without user-supplied cues. A multi-scale query projector and a calibrated text projector supply the grounding. A new benchmark, PAVE, pairs tens of thousands of street-level images with accessibility questions.
Catalog record
| CREATOR | Rafi Ibn Sultan, Hui Zhu, Xiangyu Zhou, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55730 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10703
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WalkGPT: Grounded Vision–Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation Rafi Ibn Sultan, Hui Zhu, Xiangyu Zhou, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu Abstract Ensuring accessible pedestrian navigation requires reasoning about both semantic and spatial aspects of complex urban scenes, a challenge that existing Large Vision–Language Models (LVLMs) struggle to meet. Although these models can describe visual content, their lack of explicit grounding leads to object hallucinations and unreliable depth reasoning, limiting their usefulness for accessibility guidance. We introduce WalkGPT , a pixel-grounded LVLM for the new task of Grounded Navigation Guide , unifying language reasoning and segmentation within a single architecture for depth-aware accessibility guidance. Given a pedestrian-view image and a navigation query, WalkGPT generates a conversational response with segmentation masks that delineate accessible and harmful features, along with relative depth estimation. The model incorporates a Multi-Scale Query Projector (MSQP) that shapes the final image tokens by aggregating them along text tokens across spatial hierarchies, and a Calibrated Text Projector (CTP), guided by a proposed Region Alignment Loss, that maps language embeddings into segmentation-aware representations. These components enable fine-grained grounding and depth inference without user-provided cues or anchor points, allowing the model to generate complete and realistic navigation guidance. We also introduce PAVE , a large-scale benchmark of 41k pedestrian-view images paired with accessibility-aware questions and depth-grounded answers. Experiments show that WalkGPT achieves strong grounded reasoning and segmentation performance. The source code and dataset are available on the project website . 1 Introduction Safe and accessible pedestrian navigation is essential for independent mobility in diverse environments. However, pedestrian routes often contain static and dynamic barriers such as stairs, uneven terrain, parked vehicles, and temporary obstructions 43 ; 15 ; 28 , posing significant risks for individuals with mobility challenges 51 . While most automated navigation systems are designed for vehicles 20 ; 61 , pedestrian-level navigation remains largely underexplored. Addressing this gap requires systems that understand the environment from a first-person view and reason about geometry, depth, and accessibility to navigate complex urban scenes. Figure 1 : Overview of WalkGPT for accessibility-aware grounded navigation guide. The model grounds language on segmentation masks enriched with depth information, providing holistic spatial understanding that captures both object shapes and depth cues for interpretable accessibility analysis . Recent advances in Large Vision–Language Models (LVLMs) 31 ; 30 ; 12 ; 27 demonstrate strong visual understanding and language reasoning capabilities, suggesting potential for conversational guidance in pedestrian navigation. However, existing LVLMs lack explicit spatial reasoning 6 ; 3 ; 9 needed to infer geometric structure and depth relationships in real-world scenes. Recent spatial-aware approaches 5 ; 6 ; 11 attempt to address this but often rely on user-provided visual cues to estimate object depth, which is impractical for pedestrian navigation because users cannot manually supply such inputs. LVLMs also tend to hallucinate, describing objects not present in the scene 58 , which can lead to hazardous guidance. Grounded LVLMs 38 ; 25 ; 56 ; 57 ; 39 extend conventional LVLMs by aligning textual references with corresponding image regions, improving spatial consistency and reducing hallucinations 58 . Their segmentation masks provide explicit shape cues, making them promising for pedestrian navigation. However, these masks are 2D and lack the depth information required for spatial reasoning. Consequently, grounded LVLMs remain limited in understanding relative distances and spatial hierarchy, which are critical for safe, accessibility-aware navigation. Thus, existing grounded LVLMs have not been adapted for pedestrian navigation, a domain challenged by complex urban scenes and the absence of large-scale pedestrian-view datasets with joint question–answering and grounding annotations. These limitations motivate the research question: Can we develop a grounded LVLM capable of providing depth-aware navigation guidance for pedestrians? To address these limitations, we propose WalkGPT , a grounded LVLM with spatial reasoning for the task of Grounded Navigation Guide . As illustrated in Figure 1 , given a pedestrian-view image and a user query about the path ahead, WalkGPT generates a grounded response integrating free-form reasoning with depth-aware segmentation masks for fine-grained accessibility guidance. Unlike prior spatial reasoning approaches 11 ; 5 , it operates without requiring user-provided visual cues or anchor points. By grounding accessible and harmful elements along with their segmentation masks and relative distances, WalkGPT provides comprehensive navigation feedback that supports safe movement, including for individuals with disabilities. WalkGPT, a unified architecture (illustrated in Figure 2 ), enables grounded conversations by reasoning over spatial cues from text and segmentation for navigation understanding. It introduces two novel modules: the Multi-Scale Query Projector (MSQP), which aggregates multi-scale visual context, and the Calibrated Text Projector (CTP), guided by a proposed Region Alignment Loss to refine language-to-vision grounding. Using structured tokens such as <p> , <assessment> , <SEG> , and <distance> , WalkGPT structures its outputs to explicitly link language with segmentation and distance cues, enabling automated depth-aware navigation guidance without user-provided inputs. As no existing dataset addressed this problem, we introduce PAVE , a large-scale VQA dataset of 41k pedestrian-view images with accessibility-aware questions and answers containing embedded depth information. Our main contributions are summarized as follows: • We introduce WalkGPT , the first-of-its-kind LVLM for pedestrian accessibility via grounded spatial reasoning. • We propose a streamlined architecture aligning visual and language representations to enhance pixel-level grounding using the novel MSQP and CTP with structured token supervision. • We curate PAVE , a large-scale VQA dataset with depth annotations for accessibility and spatial understanding. • WalkGPT achieves state-of-the-art performance on grounded navigation guidance and sets a benchmark for accessibility-aware AI in pedestrian environments. 2 Related Works Grounded Large Vision–Language Models (LVLMs). LVLMs 2 ; 27 ; 31 ; 12 ; 4 have advanced multimodal understanding by integrating large language models (LLMs) with visual encoders through large-scale pretraining and instruction tuning. In particular, pixel-grounded LVLMs enable region-level understanding where textual descriptions are explicitly linked to image pixels. Early works performed coarse grounding via bounding boxes 52 ; 8 ; 37 ; 49 ; 7 ; 24 ; 33 ; 55 . More fine-grained grounding using integrated segmentation encoder–decoder architectures has followed two directions: reasoning segmentation 25 ; 39 ; 57 ; 45 ; 53 and visually grounded conversation 56 ; 38 ; 48 ; 47 , both aiming to unify semantic reasoning with pixel-level grounding. Beyond these general-domain efforts, domain-specific studies have adapted visually grounded models to medical 10 and remote-sensing imagery 42 . However, grounded LVLMs have not yet been explored for accessibility-aware pedestrian navigation, which requires spatial reasoning in complex real-world environments. Spatial Reasoning LVLMs. LVLMs 11 ; 6 ; 5 ; 16 designed for spatial reasoning typically use depth maps or visual anchors to capture spatial relations between objects, whereas others 34 handle spatial reasoning without performing depth estimation. Although effective for general spatial understanding, these models often depend on user-provided visual cues or anchor points and, in some cases, omit depth prediction, which limits their suitability for navigation-oriented scenarios. Figure 2 : Overview of WalkGPT for grounded navigation guidance. (a) Overall framework. (b) The Multi-Scale Query Projector (MSQP) , which aggregates multi-level visual features into spatially aligned image tokens for language reasoning. (c) The Calibrated Text Projector (CTP) , guided by the proposed Region Alignment Loss , maps <SEG> tokens into the visual