It’s Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces Conference: Speech AI for All: The What, How, and Who of Measurement Workshop at the CHI Conference on Human Factors in Computing Systems; April 16, 2026; Barcelona, Spain CCS: Human-centered computing Accessibility design and evaluation methods
Blade Frisch, Will Wade, Dylan Gaines, Michelle Kinsella, Betts Peters, Tamara Broderick, Keith Vertanen · 2026
Read It’s Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces Conference: Speech AI for All: The What, How, and Who of Measurement Workshop at the CHI Conference on Human Factors in Computing Systems; April 16, 2026; Barcelona, Spain CCS: Human-centered computing Accessibility design and evaluation methods by Blade Frisch, Will Wade, Dylan Gaines, Michelle Kinsella, Betts Peters, Tamara Broderick, Keith Vertanen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24854.txt.
Introduction
Blade Frisch and six colleagues examine how to design and judge AI powered communication aids. Augmentative and alternative communication systems help people who cannot rely on speech to express themselves. AI features promise to widen what users can do with these systems. Standard evaluation metrics, though, capture efficiency better than desire. People are intersectional, with layered identities and nuanced wants for their tools. A metric that misses those wants can steer design the wrong way.
Six problem spaces within AAC are explored one by one. For each, the authors consider how AI might help and how success should be measured instead. They propose more robust evaluation methods that build users' intersectional nuances into the score. Wider issues recur across the spaces, and the proposed methods are offered as a way to address them. The paper addresses a CHI 2026 workshop on speech AI measurement.
Catalog record
| CREATOR | Blade Frisch, Will Wade, Dylan Gaines, Michelle Kinsella, Betts Peters, Tamara Broderick, Keith Vertanen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24854
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
It’s Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces Conference: Speech AI for All: The What, How, and Who of Measurement Workshop at the CHI Conference on Human Factors in Computing Systems; April 16, 2026; Barcelona, Spain CCS: Human-centered computing Accessibility design and evaluation methods Blade Frisch, Will Wade, Dylan Gaines, Michelle Kinsella, Betts Peters, Tamara Broderick, Keith Vertanen Abstract . Artificial intelligence (AI) can enhance what people who use augmentative and alternative communication (AAC) are able to do with their systems. However, evaluating AI-powered AAC interfaces can be difficult. People are intersectional beings and current evaluation metrics can struggle to capture the multifaceted and nuanced desires people may have for their AAC. We explore the complicated nature of six AAC problem spaces, explore how AI might be used in these spaces, and suggest more robust methods of evaluation that take the intersectional nuances of people into account. We also discuss broader issues that arise across these problem spaces and how they could be addressed using our proposed evaluation methods. 1. Introduction Metrics are essential for evaluating software performance and user experience. In order to determine how well a piece of software is performing, we must find a way to measure that performance. In machine learning and natural language processing, quantitative benchmarks are the standard for validating model efficacy. However, when these and other artificial intelligence (AI)-based approaches are integrated into augmentative and alternative communication (AAC) systems, traditional efficiency metrics often fail to capture the complexity of a user’s requirements. Evaluating AI-powered AAC requires a combination of technical performance metrics and human-centric data (e.g., self-reported usability measurements ( 2 ) , interviews ( 47 ) , usability testing ( 8 ) ). We argue that the evaluation of AI-powered AAC interfaces must move beyond the assumption that a user’s needs and abilities are static and singular. We also argue that these evaluations should not be governed solely by technical requirements and must include the needs and desires of AAC users. While technical performance metrics are necessary for system development, any finite set of technical evaluations is inherently incomplete. We propose that a richer, pluralistic set of evaluation methods can better capture misfits between an AAC user and their communication context. Our primary contributions are: • Six AAC design considerations — We analyze six design considerations to take into account when evaluating AAC interfaces: speed and accuracy, mental and physical effort, agency in identity presentation, adapting to communication contexts, turn-taking, and fluctuating physical ability. • Possible AI-powered features for AAC — We propose various ways AI might be leveraged to improve the efficacy and user satisfaction with AAC systems. • Multidimensional evaluation methods — We propose evaluation methods that combine technical system metrics with human-centered design research methods to capture a broader understanding of AAC users’ needs. 2. Positioning Existing research on AI in assistive technology is often bifurcated. Research often focuses on optimizing predictive models for speed ( 38 ) . While technical performance metrics are useful for evaluating a model’s performance, they may not be measuring the whole story of what matters to AAC users. If they are the sole method used to evaluate a model, they can impose artificial limitations on AAC systems solely to satisfy technical requirements rather than to ensure the system is addressing what truly matters to AAC users. Such a focus is similar to the medical model of disability, which risks pathologizing a communication disability over supporting the unique identities and desires of disabled people ( 33 ; 67 ) . We must move beyond imposing solutions on AAC users and instead work with them to create new approaches. It is critical to engage with AAC users in the design and intervention process ( 40 ) . This should include learning about what is important to AAC users and how to measure an AAC system’s ability to support those priorities, rather than selecting measurements because they are convenient from a technical perspective. AAC users and advocates have cautioned against viewing AI and other technological advances as “cures” for disability ( 48 ) . While recent work has explored AI bias in domains like facial recognition ( 15 ) and negative outcomes that can come from introducing AI-powered systems ( 58 ) , more work is needed to explore these issues in the AAC domain. This paper bridges these perspectives. We examine the validity of standard technical performance metrics through the lens of intersectionality and argue that AAC lacks a unified evaluation strategy that accounts for the dynamic nature of communication or the diverse needs and preferences of users. We propose evaluation methods that combine AAC users’ wants and needs with technical performance metrics. 3. Related Work AAC users are not one easily-defined group. There are many different identities people have (e.g., race, gender, societal role, views, disability status), and a person exists at the intersection of these identities ( 52 ) . This intersectionality can influence how they interact with technology. Problems arise when disability is reduced to only one identity; instead, disability can, and should, be viewed as intersectional ( 33 ; 67 ) . Rosemarie Garland-Thomson uses this lens to describe disability as a “misfit”: a relational encounter between an individual’s body-mind and an unaccommodating environment ( 25 ) . AI models often attempt to simplify this intersectionality by optimizing for a single target identity. As demonstrated by 15 ( 15 ) , such simplification can cause models to perform poorly for users at the intersection of marginalized identities ( 15 ) . In the context of AAC, this leads to technoableism ( 49 ) , where systems are optimized for a hypothetical standard user and ignore that disability is intersectional and that individuals have multiple identities. Technical performance metrics can provide useful data about individual components of an interface or model, but they must be part of a larger suite of tools, methods, and measurements to better capture the intersectional nature of the people who use AAC and AI. AAC users desire agency in the communication process, retaining control over who they interact with and how they communicate with people ( 21 ) . People change their behavior based on a myriad of social factors ( 27 ) , and being disabled can lead to being stigmatized in social situations ( 28 ) . Designing AAC interfaces without supporting the agency of AAC users to control their communication and their presentation of self can impact how people use AAC and how others perceive them. When designing AAC interfaces, it may be important to consider how users value their ability to respond in a timely manner during conversations to follow social norms of communication ( 5 ) . By definition, communication involves more than just one person. AAC users will be communicating with other people, and the characteristics of these conversations may affect how AAC users and their AAC systems are perceived by others ( 31 ) . Communication partners will bring pre-existing beliefs and biases to conversations with an AAC user. The partners’ opinions may be further shaped by the AAC strategies and access methods used by the AAC user ( 36 ) . These perceptions can affect the