SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering

Chen Chen, Cuong Nguyen, Alexa Siu, Dingzeyu Li, Nadir Weibel · 2025

Read SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering by Chen Chen, Cuong Nguyen, Alexa Siu, Dingzeyu Li, Nadir Weibel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14567.txt.

Introduction

Chen Chen and four co-authors present SweeperBot, a system that helps screen reader users explore three dimensional models. Creators often supply only brief alternative text, which lacks detail. Blind and low vision users need richer ways to compare shapes and parts. SweeperBot answers visual questions about a model in words.

It first selects a good viewing angle, then combines generative and recognition foundation models. Ten blind and low vision experts tested the system for exploring and comparing models. A second study with thirty sighted participants checked description quality. Both studies support feasibility. The work points to more independent access to three dimensional content.

Catalog record

CREATORChen Chen, Cuong Nguyen, Alexa Siu, Dingzeyu Li, Nadir Weibel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14567

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering Chen Chen, Cuong Nguyen, Alexa Siu, Dingzeyu Li, Nadir Weibel Abstract . Accessing 3D models remains challenging for S creen R eader (SR) users. While some existing 3D viewers allow creators to provide alternative text, they often lack sufficient detail about the 3D models. Grounded on a formative study, this paper introduces SweeperBot , a system that enables SR users to leverage visual question answering to explore and compare 3D models. SweeperBot answers SR users’ visual questions by combining an optimal view selection technique with the strength of generative- and recognition-based foundation models. An expert review with 10 10 B lind and L ow- V ision (BLV) users with SR experience demonstrated the feasibility of using SweeperBot to assist BLV users in exploring and comparing 3D models. The quality of the descriptions generated by SweeperBot was validated by a second survey study with 30 30 sighted participants. Figure 1 . (a) SweeperBot generates descriptions based on B lind and L ow V ision (BLV) users’ visual questions (top left). The description created by SweeperBot ( pink box , right) more accurately answers the visual question, compared to the baselines ( blue boxes , right) using the canonical view chosen by the creator. (b - c) The generated table supports BLV users to navigate the generated descriptions with existing S creen R eaders (SRs). (d) SweeperBot’s interface with an SR-accessible editable table. 1. Introduction 3D models are crucial in many real-world applications, valued for their intuitiveness and interactivity. When browsing 3D content online, users often need to view and compare multiple 3D models. For example, while purchasing furniture from IKEA, customers can use the “view in 3D” feature to examine and compare 3D models of products directly in the browser ( 60 ) . However, this process heavily relies on visual perception, making it challenging for B lind and L ow- V ision (BLV) users. Technologies like S creen R eaders (SRs) are widely used by BLV users to access non-textual content by vocalizing the alt ernative ( alt ) texts ( 5 ) . Some 3D viewers ( e.g., ( 24 ; 8 ) ) support the inclusion of alt text, providing a description of the rendered 3D model. However, this is insufficient because the alt texts may not include the key information that BLV users need ( 61 ) . For example, sellers of the sneakers shown in Figure 1 a may only provide descriptions of the overall design style while omitting details like the design patterns and the bottom sole styles. Yet, this detailed information is essential for BLV customers to fully understand the aesthetics and functionality of the shoes, enabling them to make informed purchasing decisions. Few 3D viewers like Babylon.js ( 7 ) allow creators to write alt texts for selected key objects, which SRs can vocalize based on cursor interactions. But creating them is time-consuming and tedious. Consuming these alt texts also requires BLV users to navigate views using a mouse, which is frustrating and inefficient, as keyboard shortcuts are often considered as primary input and browsing strategies for navigating focused elements with mainstream SRs ( 12 ) . Automatic V isual Q uestion A nswering (VQA) ( Antol2015 ) systems like GenAssist ( 27 ) are promising to aid BLV users in accessing visual content. These interfaces that are often powered by V ision L anguage F oundation M odels (VLFMs) allow users to ask questions about an image. It is not trivial to extend current VQA systems to support 3D understanding. These systems typically require an image as input, meaning a rendered camera view of the 3D model must be selected beforehand. This “ canonical view ” approach is undesirable, as a 3D model can be viewed from countless perspectives. There might be important details at the back or at the bottom that are not covered by the canonical view. Without the context of additional views, VLFMs would lack sufficient information to generate useful responses to the user’s question about the 3D model, leading to hallucinations (Figure 1 a). It remains challenging to extend VQA systems to support the multi-view nature of 3D models. To understand the current practices and users’ expectations regarding accessing 3D models, a formative study was conducted by interviewing two blind users with 3D and SR experience. With three key considerations, SweeperBot was prototyped as a browser-based application with an editable table (Figure 1 d). BLV users can compose free-form visual questions or commands to query about the overall content and/or details of 3D models. SweeperBot then analyzes the visual questions and the 3D model, followed by updating the answers in the editable table. This is achieved through a novel VQA pipeline, including view sampling , view selections , and answer generations . Figure 1 a illustrates how SweeperBot assists BLV users in accessing 3D models retrieved from the “shoes” catalog. Unlike descriptions generated using the canonical view, SweeperBot can accurately understand the logo design of the shoes by leveraging the selected relevant views. BLV users can use existing SRs to hear, navigate, and access the table. With generated descriptions, BLV users can ask follow-up questions to gain a full mental understanding of the 3D models (Figure 1 b - c). An expert review with 10 BLV users with SR experience demonstrated the feasibility and effectiveness of using SweeperBot to assist BLV users in exploring and comparing 3D models. A second survey study with 30 sighted users validated the quality of the SweeperBot-generated descriptions compared to baselines relying on the canonical views selected by the creators of 3D assets. 2. Related Work 2.1. Accessing 3D Models and Scenes via Audio Using audible speech is promising to help BLV users access 3D models and scenes without requiring additional hardware. Even for low-vision users, text-to-speech and auditory systems remain widely used despite their residual vision ( 55 ; 18 ) . S creen R eaders (SRs) are widely used by people who are blind, visually impaired, or may otherwise struggle to read on-screen content ( 21 ; 50 ; 55 ) . Winkle ( 61 ) speculated on a set of usability expectations that SRs need to consider when trying to make 3D content accessible. However, requiring BLV users to rely on the mouse to navigate the camera as sighted users do is challenging and frustrating ( 12 ) . A few 3D viewers ( e.g., ( 24 ; 7 ) ) let developers to specify what , when , and how the alt texts should be announced, but they often lack crucial details that BLV users need. Recent scene description smartphone applications allow BLV users to ask visual questions related to surroundings - a real-world 3D content equivalent captured by rear-facing camera. VizWiz introduced a mobile application that enables BLV users to upload images and visual questions, which are then answered by the crowd ( 10 ) . Visual interpretation services like Aira ( 6 ) and CrowdViz ( Crowdviz2015 ) provide real-time video-audio connections between BLV users and sighted assistants. Similarly, commercial application Be My Eyes ( 9 ) has been used by 300 K+ BLV users

Read the complete text