GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance
Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Jiwan Chung, Jisung Kim, Ji Hoon Joung, Youngjae Yu · 2025
Read GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance by Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Jiwan Chung, Jisung Kim, Ji Hoon Joung, Youngjae Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12844.txt.
Introduction
Junhyeok Kim and co-authors release GuideDog in this 2025 paper, a dataset of street scenes described for blind and low-vision users. Over 2.2 billion people live with vision loss, and safe independent travel stays hard. Language models could guide them, but training data written to accessibility standards barely exists. Expert annotation is slow and costly.
The dataset holds twenty-two thousand image and description pairs from forty-six countries, two thousand of them checked by humans. The pipeline flips annotation into verification: models draft, experts correct against established guidance standards. A companion benchmark of 818 questions tests object recognition and depth sense. Current models stumbled on depth and on following the guidance standards. The authors offer both resources to push assistive navigation forward.
Catalog record
| CREATOR | Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Jiwan Chung, Jisung Kim, Ji Hoon Joung, Youngjae Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance Junhyeok Kim, Jaewoo Park, Junhee Park, Sangeyl Lee, Jiwan Chung, Jisung Kim, Ji Hoon Joung, Youngjae Yu Abstract For people affected by blindness and low vision (BLV), safe and independent navigation remains a major challenge, impacting over 2.2 billion individuals worldwide. Although multimodal large language models (MLLMs) offer new opportunities for assistive navigation, progress has been limited by the scarcity of accessibility-aware datasets, because creating them requires labor-intensive expert annotation. To this end, we introduce GuideDog , a novel dataset containing 22K image-description pairs (2K human-verified) capturing real-world pedestrian scenes across 46 countries. Our human-AI pipeline shifts annotation from generation to verification, grounded in established BLV guidance standards from experts and research, improving scalability while maintaining quality. We also present GuideDogQA , an 818-sample benchmark evaluating object recognition and depth perception. Experiments reveal that depth perception and adherence to these standards remain challenging for current MLLMs. 1 Introduction Figure 1: An overview of the accessibility-aware guidance generation task, wherein MLLMs describe the overall situation ( S1 ), identify hazardous obstacles ( S2 ), and summarize recommended actions for BLV. ( S3 ) An estimated 2.2 billion individuals worldwide are affected by blindness and low vision (BLV), including approximately 36 million who are completely blind Bourne et al. (2017) . For this population, safe and independent mobility remains a significant daily challenge, as navigating unfamiliar or obstacle-filled environments poses substantial risks. Prior work has highlighted the severity of this issue, reporting that approximately 7% of visually impaired individuals experience falls at least once a month Manduchi and Kurniawan (2011) . These challenges motivate the development of assistive technologies that support safe navigation and environmental understanding. Dataset # Samples Modality Source Geo-Diverse Annotation BLV Involvement Task VizWiz ( Gurari et al., 2018 ) 31K Image User photos ✗ Human Captured by BLV VQA VIALM ( Zhao et al., 2024 ) 200 Image Web images ✗ Human Expert manual Guidance Merchant et al. (2024) 48 Image VizWiz ✗ Human ✗ Guidance WalkVLM ( Yuan et al., 2024 ) 12K Video Web + recorded 10 locations Human Survey Guidance EgoBlind ( Xiao et al., 2025 ) 1.3K Video Web ✗ Human + AI Annotated partial Video QA GuideDog (Ours) 22K Image Web 183 locations Human + AI Standards-based Guidance + QA Table 1: Comparison of existing BLV guidance datasets. GuideDog uniquely offers geographic diversity across 183 locations, standards-based BLV involvement, and comprehensive task coverage at scale. Early BLV assistive approaches, including electronic travel aids and computer vision–based systems, primarily focus on obstacle detection and avoidance Kandalan and Namuduri (2020) ; Lin et al. (2019) . While effective for basic safety, these methods struggle to capture richer contextual information required in complex real-world environments Liu et al. (2023b) ; Xie et al. (2024) . Recent advances in multimodal large language models (MLLMs) Liu et al. (2023a) ; Achiam et al. (2023) offer new opportunities for higher-level scene understanding, and have begun to be explored for BLV assistance Xie et al. (2024) ; Merchant et al. (2024) ; Zhao et al. (2024) ; Hwang et al. (2023) ; Kim et al. (2025) ; Han et al. (2025) . However, effective guidance data for BLV assistance remains difficult to obtain. Sighted annotators often struggle to anticipate BLV-specific needs, making them unsuitable for BLV-aware annotation Tigwell (2021) ; Islam et al. (2024) ; Morris (2020) , and collecting diverse egocentric data requires costly manual exploration Han et al. (2025) . Consequently, annotation relies on a small pool of domain experts Gurari et al. (2018) ; Zhao et al. (2024) , resulting in BLV benchmarks that remain extremely small and limited in diversity, as summarized in Table 1 and illustrated in Figure 2 Zhao et al. (2024) ; Merchant et al. (2024) . Figure 2: Side-by-side comparison of examples from (a) GuideDog and (b) VIALM Zhao et al. (2024) . GuideDog consists of real-world scenes from a pedestrian viewpoint, while VIALM comprises home and supermarket images. In this work, we introduce GuideDog , a large-scale accessibility-aware dataset for blind and low vision (BLV) guidance, grounded in structured standards distilled from established BLV guidelines (see Figure 1 ). The dataset is constructed using a verification-centric human–AI pipeline that replaces free-form annotation with structured generation of silver labels followed by human verification into high-fidelity gold labels Kim et al. (2024) ; Papadopoulos et al. (2016) , enabling scalable data collection while maintaining annotation quality. To capture realistic, in-the-wild scenarios reflective of BLV users’ daily mobility experiences, GuideDog samples frames from geographically and visually diverse ‘walking videos’. These web-sourced videos are captured from an egocentric viewpoint across diverse real-world landscapes, mirroring the perspective of individuals with BLV. GuideDog contains a 22K-image dataset (including 2K human-verified data) covering a broad range of real-world settings, each annotated under GuideDog standards. Additionally, we construct GuideDogQA , an evaluation subset featuring multiple-choice question-answer pairs. As GuideDog contains real-world scenes, GuideDogQA enables fine-grained evaluation of visual perception, such as verifying which objects are truly present (from correct and incorrect options) and determining relative depths among detected objects, an essential capability for both BLV assistance systems and broader egocentric applications like robotics. Using GuideDog , we evaluate several open-source and proprietary models. Our results show that proprietary models, such as GPT-4o, demonstrate superior zero-shot capabilities for BLV guidance. On GuideDogQA , open-source models lead in object recognition but lag significantly in depth perception compared to proprietary counterparts. Notably, fine-tuning with silver labels further boosts performance. These findings underscore that accurate spatial understanding is critical for effective BLV assistance. We hope these benchmarks foster further research into MLLMs for BLV assistive technologies and real-world visual perception. Our contributions are: 1. GuideDog , an accessibility-aware dataset with 22K real-world scene image–description pairs designed for blind and low vision (BLV) users, along with GuideDogQA , an 818-sample QA benchmark for evaluating fine-grained visual perception in real-world scenes. 2. A scalable dataset construction pipeline that shifts annotation from generation to human verification, using GuideDog standards distilled from established BLV guidance. 3. Experimental results demonstrating that spatial understanding and adherence to BLV-specific standards remain key challenges for current MLLMs. 2 GuideDog Dataset We present GuideDog , an egocentric multimodal dataset designed to evaluate MLLMs on BLV guidance in diverse real-world environments. To ensure scalability without compromising quality, we employ a two-stage pipeline that prioritizes verification over generation ( Section 2.2 ). First, an automated pipeline generates high-quality silver labels guided by established GuideDog standards; then, trained annotators verify and refine these labels to produce authoritative gold labels for evaluation following guidance from expert organizations and prior research. Figure 3: An overview of the GuideDog generation pipeline, ensuring all stages adhere to GuideDog standards ( S1 , S2 , S3 ). For the collected scene image displayed at the center top, (1) in accordance with S1 , the MLLM first extracts a comprehensive scene description; (2) following S2 , both off-the-shelf models and the MLLM identify obstacles; (3) next, the extracted information is incorporated into a BLV-specific instruction designed to adhere to S3 , generating a silver label that satisfies S1 , S2 , and S3 ; and (4) finally, human annotators