Disaggregated Health Data in LLMs: Evaluating Data Equity in the Context of Asian American Representation
Uvini Balasuriya Mudiyanselage, Bharat Jayprakash, Kookjin Lee, K. Hazel Kwon · 2025
Read Disaggregated Health Data in LLMs: Evaluating Data Equity in the Context of Asian American Representation by Uvini Balasuriya Mudiyanselage, Bharat Jayprakash, Kookjin Lee, K. Hazel Kwon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01091.txt.
Introduction
Uvini Balasuriya Mudiyanselage and co-authors check whether large language models give fair, split health data for Asian American groups in a 2025 paper. Models such as ChatGPT are now used in place of search for health facts. Yet Asian American groups are often lumped as one bloc, which hides gaps between sub groups such as Korean and Chinese communities.
Health study has long split data by sub group to expose unfair gaps, so it makes a good test case. The team uses stats and learning tools to judge whether model replies give split, even handed facts. Their findings feed debate on fair design for model based systems that serve diverse users.
Catalog record
| CREATOR | Uvini Balasuriya Mudiyanselage, Bharat Jayprakash, Kookjin Lee, K. Hazel Kwon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01091
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Disaggregated Health Data in LLMs: Evaluating Data Equity in the Context of Asian American Representation Uvini Balasuriya Mudiyanselage, Bharat Jayprakash, Kookjin Lee, K. Hazel Kwon Abstract Large language models (LLMs), such as ChatGPT and Claude, have emerged as essential tools for information retrieval, often serving as alternatives to traditional search engines. However, ensuring that these models provide accurate and equitable information tailored to diverse demographic groups remains an important challenge. This study investigates the capability of LLMs to retrieve disaggregated health-related information for sub-ethnic groups within the Asian American population, such as Korean and Chinese communities. Data disaggregation has been a critical practice in health research to address inequities, making it an ideal domain for evaluating representation equity in LLM outputs. We apply a suite of statistical and machine learning tools to assess whether LLMs deliver appropriately disaggregated and equitable information. By focusing on Asian American sub-ethnic groups—a highly diverse population often aggregated in traditional analyses—we highlight how LLMs handle complex disparities in health data. Our findings contribute to ongoing discussions about responsible AI, particularly in ensuring data equity in the outputs of LLM-based systems. Introduction This study examines the ability of large language models (LLMs) to retrieve disaggregated health information. LLMs, such as GPT-3 ( Brown et al. 2020 ) , Claude ( Anthropic 2023 ) , Perplexity ( AI 2023b ) , and many others, have transformed the way people seek information. These generative AI services have become widespread, serving as significant supplements—or even alternatives—to traditional web search-based information retrieval systems. Their content generation capabilities, combined with human-machine conversational interactivity, assist the broader public in accessing scientific information and knowledge with ease. However, the proliferation of LLM-based systems has raised concerns about the risks associated with their use. As a result, the concept of ‘responsible AI’ has recently gained prominence in the development and deployment of AI services, emphasizing the need to incorporate ethical considerations into practice. Among the various rights-based principles underpinning responsible AI practices, this study focuses on ‘data equity.’ Specifically, it examines the extent of representation equity in the disaggregated health information retrieved by LLMs. Data disaggregation and data equity are inseparable. Without disaggregated data collection and analysis, data equity cannot be effectively integrated into policymaking. Conversely, without a commitment to data equity, the disaggregated assessment of underserved communities becomes impossible. By systematically reviewing ways in which LLMs generate disaggregated information, this paper evaluates the state of representation equity in publicly available health-related data concerning ethnic groups. Data Disaggregation and Data Equity in LLMs Policymakers and experts have recently reached a consensus that data equity is a necessary condition for addressing various contemporary equity issues in a multiethnic society like the U.S. Notably, the White House recently launched the Equitable Data Working Group to emphasize the importance of data equity in policymaking: “Equitable data…allow for rigorous assessment of the extent to which government programs and policies yield consistently fair, just, and impartial treatment of all individuals. Equitable data illuminate opportunities for targeted actions that will result in demonstrably improved outcomes for underserved communities” ( The White House 2022 ) . In consultation with 50 civil society and research organizations, the Working Group recommended data disaggregation as one of its top priorities. Data disaggregation involves collecting, analyzing, and reporting data by purposefully accounting for underserved populations, ensuring the most marginalized groups are represented in the data used for policy decision-making. For instance, in the U.S., “people of Middle Eastern and North African heritage and subgroups of Asian American, Native Hawaiian, and Pacific Islanders” are poorly represented within the current demographic categories. This exclusion leaves them unseen in government statistics and masks critical inequities ( The White House 2022 ) . Disaggregated data practices can reveal disparities in various aspects of well-being. For example, among Asian American populations, Indian Americans had a median household income of $123,700 in 2019, whereas Burmese Americans had a median income of $44,400 ( Pew Research Center 2021 ) . Approximately 25% of Burmese Americans and 19% of Hmong Americans live below the poverty line, compared to the national average of 10.5% ( Pew Research Center 2021 ) . Regarding educational attainment, while many Asian American subgroups have high levels of achievement, disparities persist. For example, 74% of Indian Americans aged 25 and older hold at least a bachelor’s degree, compared to only 15% of Cambodian Americans and 14% of Laotian Americans ( Pew Research Center 2019 ) . Additionally, only about 73% of Pacific Islander students graduated from high school on time, compared to the national average of 86% ( National Center for Education Statistics 2021 ) . While these examples underscore the importance of data disaggregation in identifying and addressing inequities among underrepresented minority groups, its role in generating informational outputs for LLMs remains largely unexplored. In the context of LLMs, data equity issues are even more nuanced. In traditional population research, where survey data is created and used, data equity discussions typically focus on problems associated with data collection, such as sampling errors (due to difficulty accessing participants from underrepresented ethnic communities), non-response errors (stemming from a community’s distrust of institutional data collection, rooted in historical and cultural memories), and coverage errors (caused by structural factors like biased census districting) ( Shimkhada, Scheitler, and Ponce 2021 ; Sasa and Yellow Horse 2022 ) . In contrast, the LLM-based decision system introduces additional layers of complexity. LLMs likely train itself by using not raw data (e.g., survey responses) but instead publicly available secondary informational content (e.g., policy reports, news articles, research papers) derived from raw data. Consequently, biases inherent in the raw data are compounded by biases introduced during the creation of secondary sources. Furthermore, certain “interesting” findings may appear across multiple content forms and be repeatedly fed into the system (e.g., Pew survey results ( Pew Research Center 2019 ; Pew Research Center 2021 ) cited in news and scholarly articles). Moreover, how algorithms prioritize and process these sources remains unclear. Finally, the generated outputs of LLMs can become looped back into their training phases, amplifying existing biases. As a result, data equity in LLMs permeates the entire data lifecycle—from input to algorithmic processing to output. Given this complexity, this study aims to contribute to discussions on data equity in LLMs. Focusing on “representation equity,” ( Stonier et al. 2023 ) , we analyze the quality of disaggregated information retrieved by LLMs. This study centers on Asian Americans—a diverse racial population comprising numerous sub-ethnic groups—who stand to benefit significantly from improved data disaggregation practices. In terms of issue domain, we focus on health. Health policy is one of the key sectors that have shown the advancement in practicing data disaggregation. For example, when cancer rates are reported in an aggregated manner, the average rate for Asian women appears lower than that for non-Hispanic White women. However, disaggregated records reveal that Laotian women have cancer rates more than nine times higher than those of White women (WHIAANHPI, ( the White House 2023 ) ). Similarly, disaggregated data show that Vietnamese American women have the highest rates of cervical cancer among all racial/ethnic groups ( National Cancer Institute 2020 ) . Native Hawaiian and Pacific Islanders (NHPI), along with Indian and