NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision People Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3790589 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Accessibility technologies CCS: Human-centered computing Accessibility systems and tools CCS: Human-centered computing Accessibility design and evaluation methods CCS: Human-centered computing Mixed / augmented reality
Ruijia Chen, Yuheng Wu, Charlie Houseago, Filipe Gaspar, Filippo Aleotti, Dorian Gálvez-López, Oliver Johnston, Diego Mazala, Guillermo Garcia-Hernando, Maryam Bandukda, Gabriel Brostow, Jessica Van Brummelen · 2026
Read NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision People Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3790589 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Accessibility technologies CCS: Human-centered computing Accessibility systems and tools CCS: Human-centered computing Accessibility design and evaluation methods CCS: Human-centered computing Mixed / augmented reality by Ruijia Chen, Yuheng Wu, Charlie Houseago, Filipe Gaspar, Filippo Aleotti, Dorian Gálvez-López, Oliver Johnston, Diego Mazala, Guillermo Garcia-Hernando, Maryam Bandukda, Gabriel Brostow, Jessica Van Brummelen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08837.txt.
Introduction
Ruijia Chen and colleagues help blind and low vision people mark places by voice on the move. GPS-based tools drift by meters and were never tested for making notes. NaviNote pairs vision-based precise positioning with an agentic design, and tests with 18 participants show better navigation and a richer sense of place.
Catalog record
| CREATOR | Ruijia Chen, Yuheng Wu, Charlie Houseago, Filipe Gaspar, Filippo Aleotti, Dorian Gálvez-López, Oliver Johnston, Diego Mazala, Guillermo Garcia-Hernando, Maryam Bandukda, Gabriel Brostow, Jessica Van Brummelen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103409 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision People Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3790589 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Accessibility technologies CCS: Human-centered computing Accessibility systems and tools CCS: Human-centered computing Accessibility design and evaluation methods CCS: Human-centered computing Mixed / augmented reality Ruijia Chen, Yuheng Wu, Charlie Houseago, Filipe Gaspar, Filippo Aleotti, Dorian Gálvez-López, Oliver Johnston, Diego Mazala, Guillermo Garcia-Hernando, Maryam Bandukda, Gabriel Brostow, Jessica Van Brummelen Abstract . GPS and smartphones enable users to place location-based annotations, capturing rich environmental context. Previous research demonstrates that blind and low vision (BLV) people can use annotations to explore unfamiliar areas. However, current commercial systems allowing BLV users to create annotations have never been evaluated, and current GPS-based systems can deviate several meters. Motivated by high-accuracy visual positioning technology, we first conducted a formative study with 24 BLV participants to envision a more accurate and inclusive annotation system. Surprisingly, many participants viewed the high-accuracy technology not just as an annotation system but also as a tool for precise last-few-meters navigation. Guided by participant feedback, we developed NaviNote, which combines vision-based high-precision localization with an agentic architecture to enable voice-based annotation authoring and navigation. Evaluating NaviNote with 18 BLV participants showed that it significantly improved navigation performance and supported users in understanding and annotating their surroundings. Based on these findings, we discuss design considerations for future accessible annotation authoring systems. Figure 1 . NaviNote enables blind and low vision (BLV) users to explore their surroundings via a five-stage pipeline after localizing their precise positions using Visual Positioning System (VPS) in a pre-scanned area on a smartphone: (1) User asks about the area using natural language; (2) User navigates to a specific location within the area by following the turn-by-turn instructions NaviNote provides; (3) User listens to nearby spatial annotations created by other BLV users; (4) User asks follow-up questions on spatial annotations and from other information sources; (5) User creates their own spatial annotations. Arrows in the figure indicate the internal flow of the pipeline. During the interaction, the user wears a vest that holds the smartphone with camera facing forward. A five-stage pipeline is shown with 5 panels. Each panel shows a user with a vest and speech bubbles communicating between \system{} and the user. In (1), the user asks, ``Where am I?'' The system responds: ``You're at the park entrance. There's a statue nearby…'' The user follows up: ``Please, guide me to the statue!''; In (2) \system{} provides navigation instructions, ``Walk straight for 15 meters…'', ``… podium ahead at 10 o'clock''; In (3) the user listens to a nearby spatial annotation, ``Be careful, there are 16 steps ahead''; In (4) the user asks: ``What did others say about the statue?'' The system responds: ``One annotation notes the statue is of Cicero, a famed orator, lawyer and politician.''; In (5) the user creates their own spatial annotation, ``There's a nice café with accessible restrooms here.''. During the interaction, the user wears a vest that holds the smartphone with camera facing forward, and localizes their precise positions using Visual Positioning System (VPS) in a pre-scanned area on the smartphone. † † cc-license: by 1. Introduction Exploring and understanding one’s surroundings is essential for human well-being ( Russell et al. 2013 ; Finlay et al. 2019 ) yet remains a persistent challenge for blind and low vision (BLV) people. Large language models (LLM) and computer vision (CV) capabilities have supported BLV users with in-situ exploration by detecting nearby objects ( Islam et al. 2024 ; Meshram et al. 2019 ; Morrison et al. 2023 ; Lin et al. 2025 ) , recognizing text ( Huang et al. 2019 ; Prabha et al. 2022 ) , and providing scene descriptions ( Penuela et al. 2024 ; Hao et al. 2024b ; Magay et al. 2024 ; Hao et al. 2024a ; Microsoft 2024 ; Rao et al. 2021 ; Hao et al. 2024b ; Kuribayashi et al. 2025 ) , yet these camera-based AI tools impose limitations for BLV users around aiming the camera correctly ( Coughlan et al. 2020a ; Chang et al. 2024 ) , verifying information, and addressing LLM hallucinations (i.e., models generate plausible-sounding but unfaithful or nonsensical information ( Ji et al. 2023 ) ) ( Alharbi et al. 2024 ; Herskovitz et al. 2023 ; Yu et al. 2024 ) . They also cannot provide information about objects outside the camera’s frame. Location-based annotations complement existing tools by providing verified contextual information outside the camera view (e.g., an entrance behind the user). Prior work, Footnotes ( Gleason et al. 2018 ) , has shown their value in providing multifaceted information to BLV users that LLM- and CV-based tools alone cannot provide. However, BLV users play a passive role in FootNotes as well as other assistive systems for exploration ( Kaniwa et al. 2024 ; Coughlan et al. 2020b ; Froehlich et al. 2025 ) , consuming information along predetermined routes rather than contributing their own situated knowledge. Existing systems are also limited by the precision of localization technologies, often relying on the 10-meter accuracy of GPS ( Merry and Bettinger 2019 ) or similar, which limits the utility of annotations as they cannot be precisely placed or accessed. New emerging technologies could bridge this gap, allowing the creation of accessible, empowering interfaces for BLV users to act as active in-situ explorers and annotation contributors. First, multi-modal large language models (MLLM) can enable voice-first interfaces for users to interact accessibly with their surroundings ( Kaniwa et al. 2024 ) . Second, the latest Visual Positioning Systems (VPS) are far more precise than GPS at tracking where the user’s phone is in 3D space ( Platform 2025 ; Horvath et al. 2025 ) , without needing to aim cameras at specific targets. We propose that together, these technologies offer opportunities for high-accuracy, conversational, in-situ spatial annotations. We therefore seek to answer the following research questions in the context of such a system: • RQ1: What types of annotations are most valuable to BLV users, and in what scenarios are they most applicable? • RQ2: How can we design an accessible interface to enable BLV users to author in-situ spatial annotations? • RQ3: How can BLV users effectively access and utilize spatial annotations? To answer these questions, we first conducted a formative study involving 24 BLV participants, identifying six key annotation categories: Safety , Accessibility , Amenity , Layout , Attraction , and Experience . Beyond annotations, we learned our proposed technology could address a critical prerequisite for exploration: precise “last-few-meters” navigation, providing turn-by-turn instructions during the final meters of travel where traditional GPS often fails ( Saha et al. 2019a ) . Motivated by these findings, we designed NaviNote , a voice-based system combining VPS technology with an AI-powered conversational interface. Utilizing sub-meter-level localization (i.e., position error of less than one meter), NaviNote provides turn-by-turn instructions and an audio compass to facilitate safe navigation for BLV users, and supports