Supporting Multimodal Data Interaction on Refreshable Tactile Displays: An Architecture to Combine Touch and Conversational AI

Samuel Reinders Monash University, Munazza Zaib Monash University, Matthew Butler Monash University, Bongshin Lee Yonsei University, Ingrid Zukerman Monash University, Lizhen Qu Monash University, Kim Marriott Monash University · 2026

Read Supporting Multimodal Data Interaction on Refreshable Tactile Displays: An Architecture to Combine Touch and Conversational AI by Samuel Reinders Monash University, Munazza Zaib Monash University, Matthew Butler Monash University, Bongshin Lee Yonsei University, Ingrid Zukerman Monash University, Lizhen Qu Monash University, Kim Marriott Monash University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15280.txt.

Description

Reinders and colleagues present a multi-mode data use design combining chat AI with refreshable tactile displays for people who are blind or have low vision. To support labs and builders building open charts with touch screens, the team offers the design and a sample build. Their system is the first to combine touch input with a chat agent on a touch screen. It enables questions fusing touch context with voice words, such as asking about the trend between two points.

The design tackles key hurdles: touch feeling on screens, visual-to-touch coding, joining touch context with chat AI, and syncing multi-mode results. The work gives two things: a design joining screen gear, outside touch feeling, and chat AI; and a sample build showing promise. This work provides a base for later study in multi-mode open charting.

Catalog record

CREATORSamuel Reinders Monash University, Munazza Zaib Monash University, Matthew Butler Monash University, Bongshin Lee Yonsei University, Ingrid Zukerman Monash University, Lizhen Qu Monash University, Kim Marriott Monash University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15280

Browse the full library · About the Tower of Babel