Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research

Sasha Ronaghi, Emma-Louise Aveling, Maria Levis, Rachel L. Ross, Emily Alsentzer, and Sara Singer, for the iPATH investigators · 2026

Read Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research by Sasha Ronaghi, Emma-Louise Aveling, Maria Levis, Rachel L. Ross, Emily Alsentzer, and Sara Singer, for the iPATH investigators (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14478.txt.

Introduction

Sasha Ronaghi and a team of health researchers study AI help for qualitative research in this 2026 paper. Large language models show promise for improving the efficiency of qualitative analysis in large, multi-site health-services research. Yet guidance on how to integrate them into that work is limited. Evidence of their impact on real research methods and outcomes is also scarce.

The team developed a model- and task-agnostic framework for human-LLM qualitative analysis. It is designed to support diverse analytic aims. Their title stresses scale and rigor together. Readers get a method for using language models on interview and field data without giving up the care qualitative work demands.

Catalog record

CREATORSasha Ronaghi, Emma-Louise Aveling, Maria Levis, Rachel L. Ross, Emily Alsentzer, and Sara Singer, for the iPATH investigators
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14478

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research Sasha Ronaghi, Emma-Louise Aveling, Maria Levis, Rachel L. Ross, Emily Alsentzer, and Sara Singer, for the iPATH investigators Abstract Large language models (LLMs) show promise for improving the efficiency of qualitative analysis in large, multi-site health-services research. Yet methodological guidance for LLM integration into qualitative analysis and evidence of their impact on real-world research methods and outcomes remain limited. We developed a model- and task-agnostic framework for designing human-LLM qualitative analysis methods to support diverse analytic aims. Within a multi-site study of diabetes care at Federally Qualified Health Centers (FQHCs), we leveraged the framework to implement human-LLM methods for (1) qualitative synthesis of researcher-generated summaries to produce comparative feedback reports and (2) deductive coding of 167 interview transcripts to refine a practice-transformation intervention. LLM assistance enabled timely feedback to practitioners and the incorporation of large-scale qualitative data to inform theory and practice changes. This work demonstrates how LLMs can be integrated into applied health-services research to enhance efficiency while preserving rigor, offering guidance for continued innovation with LLMs in qualitative research. keywords Qualitative Analysis, Large Language Models, Health Services Research, Human–AI Collaboration 1 Introduction Qualitative approaches are critical to health-services research because they identify the personal and interpersonal experiences, organizational dynamics, and contextual factors that shape how care is delivered and received 24 . Yet the demands of large-scale, applied health studies often exceed available time and resources, and findings must be timely to remain relevant for practitioners 34 . Large Language Models (LLMs) may enhance analytic efficiency 1 ; 13 , but their use also risks reducing analysis to superficial, context-blind classifications 9 ; 23 and compromising methodological transparency 1 ; 36 ; 3 . Methodological guidance for integrating LLMs into real-world qualitative analysis workflows remains limited. Prior work on human-AI collaboration emphasizes maintaining rigor by using AI for specific research tasks and ensuring researchers retain interpretive control 15 ; 8 . However, the flexibility of all-purpose chatbot interfaces like ChatGPT has encouraged more generalized usage across the entire research process 36 ; 20 ; 26 . Such ubiquity risks undermining methodological transparency, a cornerstone of rigor in qualitative analysis 40 . Guidance is therefore needed on how to leverage the generality of LLMs and emerging LLM-based analysis tools 16 ; 10 into formalized, task-specific human-LLM qualitative analysis methods, particularly in applied studies where analytic goals, data sources, and most helpful way to report findings are highly context-dependent. Furthermore, there is a need for greater understanding of how LLM use ultimately influences real-world research methods and outcomes. Novel LLM-based systems for qualitative analysis designed by computer scientists illustrate methodological advancements, but do not focus on integrating LLMs into research studies 16 ; 10 . Qualitative scholars have compared LLMs to human analysts for qualitative coding 38 ; 22 ; 39 ; 7 and thematic analysis 41 ; 30 ; 25 ; 17 . Although they highlight the promise and limitations of LLMs for qualitative analysis, these studies typically prompt LLMs through commercial chat interfaces, limiting their ability to demonstrate LLM usability on large-scale, confidential datasets. Integrating LLMs into ongoing, real-world research studies through extensive collaboration between qualitative and computer science researchers will enable understanding of how LLMs can add value for analyzing large, heterogeneous datasets typical of applied health research. In this paper, we propose a framework for designing task-specific human–LLM qualitative analysis methods. We demonstrate how this framework provides methodological guidance for the integration of efficient and rigorous LLM-assisted qualitative analysis within a large, applied health-services research study focused on understanding and improving diabetes care practices at Federally Qualified Health Centers (FQHCs) 37 . The study spans four research teams from California, Ohio, Massachusetts, and Puerto Rico, involving more than 35 research personnel. Between April and December 2024, researchers conducted a comparative case study across 12 FQHCs, including 167 interviews with clinicians, administrators, patient representatives, and other key stakeholders, to identify organizational conditions and processes that supported or impeded effective diabetes care. The team sought to generate findings for scholarly dissemination, provide actionable feedback to practitioners, and refine a practice transformation intervention designed to improve diabetes care at FQHCs. After refining the intervention, research teams intend to implement the intervention across eight additional FQHC sites and evaluate its impact on patient outcomes. Through a collaboration of qualitative and computer science researchers, we utilized LLMs in two distinct qualitative analysis tasks after data collection for the comparative case study across 12 FQHCs. Task 1 involved qualitative synthesis to generate comparative summary reports for feedback to FQHCs, analyzing high-level summaries produced by the research teams about key elements of diabetes care ( \sim 31,200 total words). Task 2 entailed deductive qualitative coding of 167 interview transcripts ( \sim 8,600 total minutes, \sim 1,327,000 total words) to refine the planned intervention. 2 Results We present a framework for designing task-specific human-LLM qualitative analysis methods and report results from its application to two analytic tasks: (1) qualitative synthesis to generate comparative summary reports for feedback to FQHCs, and (2) deductive qualitative coding to refine the planned study intervention. For each task, we followed the four steps described below and depicted in Fig. 1 . Figure 1: The framework we developed and applied for developing task-specific human-LLM qualitative analysis methods. Step 1: Define Task We manually completed the task on a small data sample to clarify goals, finalize output format, document workflow, and set quality expectations. We also identified which components must be done by qualitative researchers to ensure their familiarity with the data, necessary for future evaluation and interpretation of LLM outputs. Step 2: Design human-LLM method We divided the task into discrete parts to surface actions by qualitative researchers difficult to identify by viewing the task holistically, such as applying context-dependent judgment or drawing on prior knowledge. For each part, we specified its purpose, identified the researcher’s tacit contributions, and considered whether to use LLMs, allowing us to test different configurations of human and LLM involvement within each task. Step 3: Evaluate human-LLM method at small-scale Two researchers compared outputs generated with and without LLM assistance on a small dataset, and their assessments informed team discussion. This approach enabled rigorous evaluation through blending the diverse expertise of the team 18 . One challenge in evaluating LLMs for qualitative analysis is the absence of a clear “gold standard,” since replicability is a contested marker of quality in qualitative research 32 . Common quantitative measures such as inter-rater reliability or overlapping thematic coverage 22 ; 16 ; 41 fail to capture whether use of an LLM can achieve the same interpretative depth of insight as qualitative investigators and overlook that appropriate quality markers are contingent on task and study specifics. Instead, we evaluated findings against task-specific goals and established criteria of qualitative rigor such as: grounding in the data, integration of theory and data, alignment with the research question, significance and relevance to the field, and usefulness for practitioners 40 . These evaluations guided how LLM outputs were integrated and interpreted when applied to the larger dataset. In both tasks, we considered the human-LLM approach sufficient for application to the larger dataset when the variation in outputs between manual and

Read the complete text