Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conferen
Paige S. DeVries, Michaela Okosi, Ming Li, Nora Dunphy, Gidey Gezae, Dante Conway, Abraham Glasser, Raja Kushalnagar, Christian Vogler · 2026
Read Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conferen by Paige S. DeVries, Michaela Okosi, Ming Li, Nora Dunphy, Gidey Gezae, Dante Conway, Abraham Glasser, Raja Kushalnagar, Christian Vogler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15209.txt.
Introduction
Paige S. DeVries, Michaela Okosi, and a team of co-authors test whether smart assistants work for deaf users who speak. Their 2026 arXiv paper deals with intelligent personal assistants. Many deaf and hard of hearing people use their voice in daily life.
Yet the assistants often fail to parse diverse accents, deaf speech among them. That leaves non-signing, speaking deaf users largely shut out. The authors compare voice-based options with a touch interface powered by a language model. The study weighs which route gives these users real, working, daily access in practice.
Catalog record
| CREATOR | Paige S. DeVries, Michaela Okosi, Ming Li, Nora Dunphy, Gidey Gezae, Dante Conway, Abraham Glasser, Raja Kushalnagar, Christian Vogler |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791869 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Accessibility technologies CCS: Human-centered computing Empirical studies in accessibility CCS: Human-centered computing Accessibility systems and tools Paige S. DeVries, Michaela Okosi, Ming Li, Nora Dunphy, Gidey Gezae, Dante Conway, Abraham Glasser, Raja Kushalnagar, Christian Vogler Abstract . We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people who can use their voice in everyday communication. The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals. Using an Echo Show, we compare the usability of natural language input via spoken English; with Alexa’s automatic speech recognition and a Wizard-of-Oz setting with a trained facilitator re-speaking commands against that of a large language model (LLM)-assisted touch interface in a mixed-methods study. The touch method was navigated through an LLM-powered ‘task prompter,’ which integrated the user’s history and smart environment to suggest contextually-appropriate commands. Quantitative results showed no significant differences across both spoken English conditions vs LLM-assisted touch. Qualitative results showed variability in opinions on the usability of each method. Ultimately, it will be necessary to have robust deaf-accented speech recognized natively by IPAs. † † cc-license: by 1. Introduction Intelligent personal assistants (IPAs), such as Amazon Alexa ( 18 ) and Google Assistant ( 14 ) , offer the convenience of voice-controlled functionality but present significant accessibility challenges for Deaf and Hard of Hearing (DHH) individuals ( 1 ; 9 ; 11 ; 13 ; 22 ) and even more limited options for DHH non-signers with dysarthric speech. Currently, Automatic Speech Recognition (ASR) still struggles to understand the variability of deaf speech ( 25 ; 4 ; 15 ; 24 ; 17 ) . The level of dysarthric speech often depends on when the individual lost their ability to hear and the level of hearing loss. The consensus on a functionally equivalent input method for DHH users still remains a question ( 16 ; 25 ) . Current IPAs restrict non-typical verbal input options due to their over-reliance on ASR technologies unable to adapt to non-standard speech patterns. Non-verbal approaches currently pose a significant latency that negatively impacts user experience and accessibility ( 10 ) . As the development and daily integration of these technologies grows exponentially, the need for interfaces designed for DHH usability becomes more urgent. There has been much investigation of sign language input as a possibility in lieu of speech interfaces, but very little research on those who are DHH and use their voice. With respect to DHH people who speak, this paper aims to provide much-needed research into the the usability of input methods for IPAs that are technically feasible today. This group of DHH people currently, in principle, has three options for interacting with current-generation IPAs: (1) their own speech, (2) using specially trained software for their deaf accent, which re-speaks the recognized commands into a form that IPAs can understand, and (3) touchscreen interfaces. With respect to (1), as mentioned above, many DHH people express dysarthric speech that makes it difficult for them to interact with IPAs directly. Nevertheless, it is important to study to what extent this is a problem in current practice. With respect to (2), re-speaking software for dysarthric speech is commercially available ( 31 ) , but induces a delay and requires the user to train the system in advance. With respect to (3), although touchscreen input may seem like a sufficient alternative, its current usability is less efficient than that of natural language input, as previously shown in studies on DHH participants ( 28 ) in part due to the time required to navigate the user interface and type commands. However, with touchscreens, historically, such approaches have not been context-aware enough to provide the most relevant options that a user might select at any given time. With the commercialization of generative artificial intelligence (AI) that utilizes large language models (LLMs), such as ChatGPT, there are new opportunities for efficient and adaptable user interfaces through the use of situational context and previous user commands. This opens up new opportunities for improvements to touch-based interaction that can benefit DHH people. In this paper, we aim to assess and compare the usability of what currently appear to be the most promising options for non-signing DHH people to interact with IPAs: (1) a novel LLM-assisted touch interface, shown in Figure 1 , which leverages the situational context and ongoing interaction with each user in an attempt to make touch-based options more effective, and (2) spoken English usability, divided into two approaches: (2a) natural deaf speech with the built-in ASR technology in IPAs, and (2b) re-speaking deaf-accented dysarthric speech to an IPA in more easily intelligible English. Figure 1. A user selecting LLM-populated options on a touchscreen in our LLM Touch interface. A user is selecting options generated by the Large Language Model via a tablet showing a selection of 8 buttons. 1.1. Study Approach The approach chosen for this paper was a mixed method study that allowed participants to explore three input methods (LLM-assisted touch, natural deaf speech, and re-speaking). For re-speaking, we originally considered using automatic speech recognition technologies that have been developed for those that have non-standard speech patterns (like VoiceITT and Project Relate ( 31 ; 23 ) ), but ultimately decided against it as it would have required participants to spend hours on training the system. From an experimental design standpoint, this would have posed a risk of the training experience unduly influencing the participant ratings compared to the other input methods that we tested. To work around this re-speaking barrier, for a valid comparison between speech and touch, we asked participants to test both their natural speech with Alexa’s built-in ASR, and Wizard-of-Oz-facilitated re-speaking, where a trained researcher listened to and watched the participant on video, then re-voiced their spoken commands. A human facilitator, in a sense, represents the best case for re-speaking, with no prior training for participants required, and a higher ceiling of understanding deaf-accented speech than the currently available commercial options. We evaluated usability both quantitatively through usability questionnaires and qualitatively through user interviews to gain further insight into the usability scores. 1.2. Research Questions The research questions for the study presented in this paper are: • RQ1: How do DHH IPA users who use their voice perceive the usability and usefulness of LLM-generated task suggestions compared to deaf speech input? • RQ2: How do the satisfaction ratings of DHH users, who use their voice, compare between natural deaf speech and Wizard-of-Oz Facilitated English speech methods? In the remainder of this paper, we discuss related work, then describe the mixed-method experimental design, provide quantitative and qualitative results, discuss their implications, and note the limitations of the study design. 2. Related Work The limitations of current IPAs demonstrate a need for improved accessibility that AI and machine learning