TouchScribe: Augmenting Non-Visual Hand-Object Interactions with Automated Live Visual Descriptions Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Syste

Ruei-Che Chang, Rosiana Natalie, Wenqian Xu, Jovan Zheng Feng Yap, Tiange Luo, Venkatesh Potluri, Anhong Guo · 2026

Read TouchScribe: Augmenting Non-Visual Hand-Object Interactions with Automated Live Visual Descriptions Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Syste by Ruei-Che Chang, Rosiana Natalie, Wenqian Xu, Jovan Zheng Feng Yap, Tiange Luo, Venkatesh Potluri, Anhong Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07802.txt.

Description

Blind and low-vision people learn much about objects by handling them, yet touch alone cannot reveal color, text, or fine visual detail. Ruei-Che Chang and six co-authors present TouchScribe, a system that watches hand-object interaction from the user's viewpoint and narrates what the hands are exploring.

A custom model recognizes both common gestures, like holding two items side by side to compare them, and gestures particular to blind users. These include pointing to explore color or swiping to read text. Feedback adapts as the hands move, from interaction state to object labels to visual details. User studies and technical tests show the descriptions genuinely aid object grasp, pointing toward richer assistive technology.

Catalog record

CREATORRuei-Che Chang, Rosiana Natalie, Wenqian Xu, Jovan Zheng Feng Yap, Tiange Luo, Venkatesh Potluri, Anhong Guo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-07802

Browse the full library · About the Tower of Babel