Navig-AI-tion: Navigation by Contextual AI and Spatial Audio Conference: Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772363.3799295 ISBN: 979-8-4007-2281-3/2026/04 CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation CCS: Human-centered computing Auditory feedback
Mathias N. Lystbæk, Haley Adams, Ranjith Kagathi Ananda, Eric J Gonzalez, Luca Ballan, Qiuxuan Wu, Andrea Colacço, Peter Tan, Mar Gonzalez-Franco · 2026
Read Navig-AI-tion: Navigation by Contextual AI and Spatial Audio Conference: Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772363.3799295 ISBN: 979-8-4007-2281-3/2026/04 CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation CCS: Human-centered computing Auditory feedback by Mathias N. Lystbæk, Haley Adams, Ranjith Kagathi Ananda, Eric J Gonzalez, Luca Ballan, Qiuxuan Wu, Andrea Colacço, Peter Tan, Mar Gonzalez-Franco (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13200.txt.
Introduction
Mathias N. Lystbæk and co-authors present Navig-AI-tion, an audio-only walking navigation system with added context. Audio directions alone can leave users lost, since cardinal directions are vague and context is thin. Their system pairs a Vision Language Model with a spatial audio cue. Landmarks drawn from the scene anchor each instruction in familiar terms. A directional cue sounds when the user faces the wrong way and shows the turn to make. In a small user study, the system cut route errors versus both rivals tested.
Catalog record
| CREATOR | Mathias N. Lystbæk, Haley Adams, Ranjith Kagathi Ananda, Eric J Gonzalez, Luca Ballan, Qiuxuan Wu, Andrea Colacço, Peter Tan, Mar Gonzalez-Franco |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31635 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Navig-AI-tion: Navigation by Contextual AI and Spatial Audio Conference: Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772363.3799295 ISBN: 979-8-4007-2281-3/2026/04 CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation CCS: Human-centered computing Auditory feedback Mathias N. Lystbæk, Haley Adams, Ranjith Kagathi Ananda, Eric J Gonzalez, Luca Ballan, Qiuxuan Wu, Andrea Colacço, Peter Tan, Mar Gonzalez-Franco Abstract . Audio-only walking navigation can leave users disoriented, relying on vague cardinal directions and lacking real-time environmental context, leading to frequent errors. To address this, we present a novel system that integrates a Vision Language Model (VLM) with a spatial audio cue. Our system extracts environmental landmarks to anchor navigation instructions and, crucially, provides a directional spatial audio signal when the user faces the wrong direction, indicating the precise turn direction. In a user study (n=12), the spatial audio cue with VLM reduced route deviations compared to both VLM-only and Google Maps (audio-only) baseline systems. Users reported that the spatial audio cue effectively supported orientation and that landmark-anchored instructions provided a better navigation experience over audio-only Google Maps. This work serves as an initial look at the utility of future audio-only navigation systems for incorporating directional cues, especially real-time corrective spatial audio. Figure 1. Augmented navigation using AI and spatial audio for display-less smart glasses (a). A simplified overview of the system architecture (b). Two panels. (a) An illustration of a walking user receiving a Spatial Audio Beacon and a spoken instruction, 'Turn left at the fountain', produced by a Vision Language Model using GPS and world context. (b) A system architecture flow diagram: Online Maps API connects to Directions, which connects to World Tracking; AR Core Extensions and AR Foundation feed World Tracking and Prompting; World Tracking connects to Prompting and Audio; Prompting connects to Audio and Vision Language Model; Vision Language Model connects to a Multimodal LLM in the cloud; Audio connects to a Head-Related Transfer Functions (HRTFs) Audio pipeline and Text-to-Speech (TTS). † † cc-license: by 1. Introduction Traditional GPS navigation has enabled a mobility revolution, yet there are legitimate questions about how much engagement with the environment current systems provide ( 12 ) . Modern turn-by-turn navigation excels at getting us from A to B, but growing evidence suggests that this could come at the cost of understanding and remembering the spaces we navigate ( 11 ; 7 ; 6 ) . Without coupling with the environment, these systems risk reducing navigation to blindly following an endless list of instructions ( 8 ) . Head-worn devices with cameras, like AR glasses, open new possibilities for enhancing navigation, particularly in complex walking scenarios where minimal screen interactions are desirable. Prior work has shown that spatial audio cues can provide more seamless orientation guidance, subtly assisting users in re-orienting themselves ( 5 ; 2 ; 13 ; 10 ) . Meanwhile, head-attached cameras combined with multimodal Large Language Models (LLMs), or Vision Language Models (VLMs), can generate contextual, landmark-based navigation instructions, e.g., replacing “turn left in 50 feet” with “turn left at the fountain” (cf. Figure 1 a). The combination of VLM-powered visual context and spatial audio has the potential to create a less disruptive navigation experience grounded in both sensory input and the environment. This is precisely the approach we prototype and evaluate in this paper. While our goal is to support orientation in audio - only contexts, our preliminary study did not find improvements in spatial awareness (pointing accuracy). However, we observe reductions in route deviations and distance walked. 2. Related Work The pervasive use of turn-by-turn navigation apps (e.g., Google Maps, Apple Maps, Waze) is a testament to their effectiveness for reaching destinations ( 16 ) . However, this convenience can come at a cognitive cost: the more people rely on GPS navigation, the worse their perceived navigation skills and environmental knowledge tends to become ( 22 ; 15 ) . 2.1. Audio cues for navigation Audio cues have been shown to effectively support walking navigation as an alternative to screen-based instructions ( 5 ; 20 ; 10 ; 14 ; 23 ; 21 ) , with spatial audio in particular helping users find the correct direction ( 5 ; 19 ; 20 ) . Notably, 5 introduced Soundscape ( 5 ) , which guided users via continuous spatial audio and improved spatial recollection of points of interest (POIs). Our approach is similar, however, we only present the spatial audio cue when the user is not facing the suggested direction, lowering auditory load. 2.2. VLMs for navigation Recent advances in LLMs enable richer, more context-aware descriptions of surroundings and more natural interactions ( 1 ) , especially multimodal variants (VLMs) that integrate visual and textual data. To the best of our knowledge, few systems integrate VLMs for wayfinding. One example is NaviGPT ( 25 ) , which combines Apple Maps (route guidance), LiDAR (obstacle detection), and GPT-4 (information processing and instructions) for real-time navigation on iPhone. In contrast to NaviGPT, our system is designed exclusively for audio-based delivery and supports display-less devices, anticipating future hardware form factors (e.g., Meta Ray-Bans). Additionally, we present a user evaluation yielding key design insights into VLM integration for wayfinding that can guide future development in this emerging field. 3. System The system augments navigation instructions with context about the user and their environment, including orientation and a point-of-view photo. Furthermore, the system provides spatial auditory feedback when the user is not facing the next turn, giving the user an indication of the direction that they should walk. The spatial audio cue is otherwise turned off, allowing the user to listen to their own audio while walking (e.g., music, videos, or podcasts). The system consists of five components working in sequence to (1) collect navigation instructions, (2) track the user’s position and orientation, (3) present audio feedback, (4) capture the user’s perspective and prompt the VLM, and (5) extract landmarks and augment the original instruction. See Figure 1 b for a diagram overview. 3.1. Directions Walking directions are queried based on a list of POI GPS coordinates (in this work, we used the Google Maps Platform API), returning JSON-formatted steps with walking instructions for the desired origin and destination. 3.2. World Tracking The World Tracking component tracks the user and keeps track of where turn waypoints and POIs are relative to the user, also checking if the user has reached a turn. The system only captures an image from the user’s point-of-view when it is about to prompt the VLM. AR Foundation provides the core functionality for phone tracking, camera capture, and spatial audio positioning based on user orientation. AR Core Extensions adds accurate GPS coordinate capture and virtual anchor placement. User orientation is determined via Google’s Visual Positioning System (VPS). (a) Routes. (b) Apparatus. (c) Results on Distance and Deviations. Figure 2. (a) Three color-coded routes, marked with route start and end for each route RX, where “X” is 1 to 3, along with color-matched stars and circles indicating landmarks along the routes. “RX D-E” indicates the dead end