Who Gets Left Behind? Auditing Disability Inclusivity in Large Language Models

Deepika Dash, Yeshil Bangera, Mithil Bangera, Gouthami Vadithya, Srikant Panda · 2025

Read Who Gets Left Behind? Auditing Disability Inclusivity in Large Language Models by Deepika Dash, Yeshil Bangera, Mithil Bangera, Gouthami Vadithya, Srikant Panda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00963.txt.

Introduction

Deepika Dash leads a team of five authors on disability inclusivity in large language models. Such models are used more and more for accessibility guidance. Yet many disability groups remain underserved by their advice. The team presents a taxonomy-aligned benchmark of accessibility questions. The questions are human-validated and general-purpose in scope.

Models are evaluated along three dimensions. They are breadth within answers, balance across nine disability categories, and depth of support. Seventeen proprietary and open-weight models are tested. Vision, hearing, and mobility are addressed often. Speech, genetic and developmental, sensory-cognitive, and mental health lag behind. Depth is also concentrated in a few categories. The findings highlight levers like taxonomy-aware prompting and training.

Catalog record

CREATORDeepika Dash, Yeshil Bangera, Mithil Bangera, Gouthami Vadithya, Srikant Panda
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Who Gets Left Behind? Auditing Disability Inclusivity in Large Language Models Deepika Dash, Yeshil Bangera, Mithil Bangera, Gouthami Vadithya, Srikant Panda Abstract Large Language Models (LLMs) are increasingly used for accessibility guidance, yet many disability groups remain underserved by their advice. To address this gap, we present taxonomy aligned benchmark 1 1 1 Preprint. Under review. of human validated, general purpose accessibility questions, designed to systematically audit inclusivity across disabilities. Our benchmark evaluates models along three dimensions: Question-Level Coverage (breadth within answers), Disability-Level Coverage (balance across nine disability categories), and Depth (specificity of support). Applying this framework to 17 proprietary and open-weight models reveals persistent inclusivity gaps: Vision, Hearing, and Mobility are frequently addressed, while Speech, Genetic/Developmental, Sensory-Cognitive, and Mental Health remain under served. Depth is similarly concentrated in a few categories but sparse elsewhere. These findings reveal who gets left behind in current LLM accessibility guidance and highlight actionable levers: taxonomy-aware prompting/training and evaluations that jointly audit breadth, balance, and depth. 1 Introduction Large Language Models (LLMs) are rapidly transforming how people access information, seek guidance, and interact with technology OpenAI et al. 2023 ; Brown et al. 2020 . Recent advancements such as faster inference techniques Srikant Panda 2025 , FS-DAG and LayoutLM architectures Xu et al. 2020 ; Agarwal et al. 2025b , and new conversational frameworks Meghwani et al. 2025 ; Pattnayak et al. 2025 have significantly expanded the capabilities of LLM-based systems. These developments enhance performance in key areas like document understanding, table extraction Avinash Rajeshchandra Rai 2025 , and retrieval-augmented generation (RAG) applications. Their potential is particularly salient for the more than 1.3 billion people worldwide living with disabilities World Health Organization 2023 , for whom accessible design is not optional but essential in education, healthcare, workplaces, and public services. As LLMs are increasingly embedded in assistants, and content tools, they now play a central role in shaping accessibility guidance. Yet a critical question remains: do LLMs provide comprehensive and balanced support across the full spectrum of disability needs ? Existing NLP fairness research has primarily examined demographic dimensions such as gender, race, safety or political ideology Panda et al. 2025b ; Bolukbasi et al. 2016 ; Fulay et al. 2024 ; Sun et al. 2023 ; Kumar et al. 2024 , or focused on detecting toxic Patel et al. 2025 ; Agarwal et al. 2025a and ableist language Smith et al. 2024 ; Bender et al. 2021 . A smaller body of work has begun to investigate disability-specific concerns. For example, recent studies have shown how LLMs may disadvantage disabled people in hiring scenarios such as resume screening Glazko et al. 2024 , or examined the accuracy of responses when disability-related queries are posed Panda et al. 2025a ; Anonymous 2025 ; Gadiraju et al. 2023 . These efforts highlight bias and correctness issues, but they typically center on narrow contexts or single disability categories. In contrast, an inclusive system should span recommendations for vision, hearing, speech, mobility, neurological, genetic/developmental, learning, sensory-cognitive, and mental health needs. For instance, in response to “How can hospitals be made more accessible?” , failure to mention multiple categories risks incomplete and inequitable guidance. Figure 1: Disability Auditing Framework We address this gap by benchmarking that not only valuates if whether LLMs mention disabilities, but whether they do so with breadth, balance, and depth. Our framework (Figure 1 ) operationalizes these dimensions across nine disability categories, using human validated, general purpose questions about design principles, systemic barriers, and policy considerations. This setup isolates whether models can surface inclusive, cross category guidance. Our Contributions 1. Benchmark and framework. Benchmark for accessibility inclusivity, together with LLM-as-judge evaluation pipeline that grounds evidence extraction in disability categories and enables reproducible metric computation with cross-judge reliability checks. 2. Empirical analysis. We benchmark 17 proprietary & open-weight models, finding that they they cover only about half of relevant categories per question. Coverage is systematically skewed (Vision, Hearing, Mobility overrepresented; Speech, Genetic/Developmental, Sensory-Cognitive, & Mental Health underrepresented), and Depth remains sparse. 3. Mitigation levers. Our analysis highlights actionable interventions for improving inclusivity through taxonomy aware prompting. By framing accessibility inclusivity as a measurable axis of fairness, our work complements prior benchmarks that treat disabilities uniformly or focus narrowly on bias detection. We argue that ensuring balanced and comprehensive disability representation is essential for building AI systems that align with global accessibility standards and equitably serve all users. 2 Related Work 2.1 Accessibility Frameworks and Taxonomies Our taxonomy of nine disability categories aligns with established accessibility frameworks. The WHO International Classification of Functioning, Disability and Health (ICF) World Health Organization 2001 emphasizes functioning within environmental contexts, motivating breadth-oriented assessment across heterogeneous impairments. Complementarily, the W3C Web Content Accessibility Guidelines (WCAG 2.2) World Wide Web Consortium 2023 articulate actionable criteria for making digital content accessible to people with a wide range of disabilities. These standards foreground inclusive design beyond toxicity detection or style guidance, and inform our choice to evaluate coverage , balance , and depth of accessibility knowledge. 2.2 Disability-Centered Harms and Bias in NLP A growing body of work shows that LLMs can produce both allocational and representational harms affecting disabled people. Kamruzzaman and Kim 2025 finds that when disability is disclosed in job applications, LLM-driven hiring systems tend to rank those candidates lower, indicating systemic disadvantages and the need for fairness auditing and inclusive AI design. AccessEval Anonymous 2025 builds a paired-query benchmark across 16 open-source LLMs and shows substantially higher negative bias and misinformation on disability related queries especially in finance, healthcare, and speech/hearing highlighting domain specific risks. Venkit et al. 2023 demonstrates explicit disability bias in commercial sentiment and toxicity services using the BITS corpus, suggesting that moderation pipelines may systematically mishandle disability contexts. Beyond text-only interfaces, Korpan 2025 shows that LLM-shaped robot caregiving narratives simplify disability and assign lower sentiment to disability and LGBTQ+ identities, revealing downstream risks for HRI design. Language choices and representational frames shape how disability is encoded in NLP systems. Hutchinson et al. 2020 argues that models absorb disability stereotypes from data, yielding harmful or inaccurate outputs and motivating inclusive data curation with disabled community involvement. Herold et al. 2022 analyze on various disability biases can propagate into downstream accessibility tools. Complementing these, Venkit et al. 2022 uses perturbation sensitivity analysis to show pervasive implicit disability bias across both embeddings and transformers, suggesting architecture-agnostic representational harms. Intersectional and contextual dynamics complicate these harms and their detection. Hassan et al. 2021 examines ableist bias through an intersectional lens and finds compounded disadvantages when disability intersects with race and gender, underscoring the need for intersectional mitigation. Gadiraju et al. 2023 centers disability perspectives and reports that LLM outputs, while sometimes helpful, frequently reflect misunderstandings, stereotypes, and subtle biases reinforcing the value of participatory, disability-centered design. Phutane et al. 2025 compares human and model ratings of 100 ableist comments, showing toxicity classifiers and LLMs underrate toxicity relative to people with disabilities, though LLMs’ ableism ratings align somewhat better revealing gaps in harm perception and explanation. Rizvi et al. 2025 concludes that current LLMs are ill-equipped to detect anti-autistic ableism because they over-rely on keyword cues rather than context, which risks suppressing community voices. Rizvi et al. 2024 introduces AUTALIC, a

Read the complete text