The AI Co-Ethnographer: How Far Can Automation Take Qualitative Research?

Fabian Retkowski, Andreas Sudmann, Alexander Waibel · 2025

Read The AI Co-Ethnographer: How Far Can Automation Take Qualitative Research? by Fabian Retkowski, Andreas Sudmann, Alexander Waibel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00012.txt.

Introduction

Fabian Retkowski, Andreas Sudmann and Alexander Waibel introduce an AI co-ethnographer for qualitative research, in work from 2025. Coding interviews and field notes by hand is slow and hard to scale without losing depth. Their pipeline goes beyond assigning codes: it runs open coding, merges and consolidates codes, applies them across the material and hunts for patterns, covering the full analytic arc in one system. The paper asks how far such automation can carry interpretive work before human judgement must take over.

Catalog record

CREATORFabian Retkowski, Andreas Sudmann, Alexander Waibel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The AI Co-Ethnographer: How Far Can Automation Take Qualitative Research? Fabian Retkowski, Andreas Sudmann, Alexander Waibel Abstract Qualitative research often involves labor-intensive processes that are difficult to scale while preserving analytical depth. This paper introduces The AI Co-Ethnographer (AICoE), a novel end-to-end pipeline developed for qualitative research and designed to move beyond the limitations of simply automating code assignments, offering a more integrated approach. AICoE organizes the entire process, encompassing open coding, code consolidation, code application, and even pattern discovery, leading to a comprehensive analysis of qualitative data. 1 Introduction Qualitative data analysis is a crucial research approach in the humanities, cultural studies, and social sciences, focusing on the synchronic and diachronic analysis and interpretation of non-numerical data such as texts, images, or audio files to gain insights into complex social phenomena, cultural expressions, and individual experiences Creswell and Poth 2017 ; Denzin et al. 2023 . Coding is central to this process, structuring and interpreting research materials such as interviews, field notes, or group discussions by systematically assigning analytically relevant concepts to text segments or other data forms Holton 2007 ; Bernard 2011 ; Harding 2013 ; Bernard et al. 2016 . Although coding offers a formalized structure for data analysis, its application remains context-specific and flexible, adapting to the nuances of the research question and subject matter Elliott 2018 . In many contexts, specifically in ethnographic approaches, coding is inherently iterative and closely tied to an ongoing process of collecting and reflecting on data. Codes evolve dynamically through an iterative process where they are merged, adjusted, added, or replaced as researchers engage with the data, identify patterns, and refine their conceptual understanding. This process may involve open or axial coding, deductively or inductively, quantitatively or qualitatively, and can be centered on interpretation or description. Ritchie et al. 2014 ; Creswell 2015 ; Saldana 2015 . However, manual coding faces significant limitations. Scalability remains a critical challenge when researchers encounter larger datasets that require extensive time and resources to code effectively Miles et al. 2019 . It also increases the risk of intra- and intercoder unreliability, just to mention a few typical challenges. These constraints have spurred interdisciplinary efforts to automate the coding process over the past decade. Automated speech recognition (ASR) has emerged as a significant enabler in this landscape, allowing researchers to efficiently transcribe large volumes of interview data and prepare them for further analysis and processing Nguyen et al. 2021 . Related qualitative data processing tasks such as text summarization Hori et al. 2002 ; Retkowski and Waibel 2024b ; Zhang et al. 2024 , question answering Singhal et al. 2025 , and topic segmentation Zechner and Waibel 2000 ; Retkowski and Waibel 2024a have similarly benefited from computational advancements, providing researchers with tools to condense information and identify thematic boundaries. Recently, large language models (LLMs) have demonstrated new epistemic capabilities to annotate research data, yet with certain limitations, such as understanding the broader context of codes Tuschling et al. 2023 ; Fischer and Biemann 2024 ; Rasheed et al. 2024 ; Ziems et al. 2024 . In parallel, the concept of Agentic LLMs has emerged, designed to operate autonomously with goal-directed behaviors Xi et al. 2023 . For example, the AI Scientist Lu et al. 2024 showcased an end-to-end automated workflow for writing scientific papers, from hypothesis generation, experimental design and manuscript drafting. This work illustrates the potential for autonomous agents to manage complex, multi-stage research processes. Inspired by these advances, our approach seeks to explore similar automation in the domain of qualitative research, also as an alternative to AI-assisted data analysis with proprietary systems like MaxQDA. Figure 1: Conceptual Illustration of the AI Co-Ethnographer Pipeline With the AI Co-Ethnographer (AICoE), we introduce a novel end-to-end pipeline that extends beyond the conventional focus on code assignments. The AICoE is part of a broader infrastructure for AI-assisted knowledge production, integrating diverse qualitative analysis methods, from open coding to pattern discovery. Whereas prior research has largely concentrated on automating the mapping of codes to text segments, our approach encompasses a more comprehensive qualitative analysis process. The pipeline extends the capabilities beyond the deductive application of pre-defined codes. Crucially, it also enables inductive code development and application, a process where novel codes are developed directly from the data itself instead of being pre-defined. 2 Related Research LLM development has spurred transdisciplinary efforts to automate scholarly work, especially qualitative textual analysis Morgan 2023 ; Petersen-Frey et al. 2023 ; Fischer and Biemann 2024 ; Lu et al. 2024 ; Franken and Vepřek 2025 , including ethnographically focused research Dippel and Sudmann 2023 . This builds on a rich history of computational methods in qualitative research, from early tools like the General Inquirer Stone and Hunt 1963 and Salton‘s vector space model Salton et al. 1975 , to machine learning-based annotation Sebastiani 2002 , and open-source platforms like WordFreak Morton and LaCivita 2003 and WebAnno Yimam et al. 2014 . More recently, Spinoso-Di Piano et al. 2023 introduced the Qualitative Code Suggestion (QCS) task, which assists in coding by providing a ranked list of predefined codes for a given text passage. To evaluate QCS, the authors present CVDQuoding, an annotated dataset of interviews with women at risk of cardiovascular disease. Human evaluation shows that their system provides relevant suggestions, highlighting its potential as an assistive tool. However, limitations remain, including a focus on code assignment rather than full codebook development and a lack of evaluation in applied research settings. Similarly, Ziems et al. 2024 evaluated the potential of LLMs for automating social science tasks, focusing on their zero-shot capabilities. Their findings indicate that LLMs demonstrate proficiency in both classification and explanation, suggesting their ability to augment the social science research pipeline. However, the authors do not recommend LLMs as a replacement for traditional methods. 3 Methodology The AI Co-Ethnographer is composed of a comprehensive pipeline underpinned by LLMs to automate key qualitative research processes while aiming to preserve the interpretative depth central to ethnography. Building on recent advances in LLMs, the system mirrors several stages of qualitative analysis (see Figure 1 ): open coding, code consolidation, code application, and pattern finding. This approach enables scalable and consistent analysis of large volumes of qualitative data while mimicking ethnographic research practices. 3.1 Open Coding A first step can be called open coding , where individual interviews are processed separately by the LLM. By isolating analyses per interview, the chosen research design addresses both the context window limitations of LLMs and the ethnographic principle of maintaining close connection to primary data. The system may suggest up to N codes per interview, balancing descriptive and interpretive coding approaches and, in doing so, automating a time-consuming element of qualitative analysis. 3.2 Code Consolidation The code consolidation stage transitions to a global perspective and synthesizes findings across all interviews into a unified codebook. The synthesis process analyzes code overlap and merges similar concepts, culminating in a maximum of up to M consolidated codes. This stage represents a crucial bridge between individual narratives and broader theoretical development, akin to manual axial coding but computationally scaled. 3.3 Code Application The pipeline returns to a local perspective in the code application stage, where each consolidated code is systematically applied

Read the complete text