LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understanding
Ying Na, Shihui Feng · 2025
Read LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understanding by Ying Na, Shihui Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19734.txt.
Introduction
Ying Na and Shihui Feng use chat models to code class talk in a set frame in a 2025 paper. Talk data shows how students build facts in joint work, a key learn source. Hand coding is slow, and chat models strain on the rich links that talk carries.
Their frame calls each line's code from talk traits: what act a line does and what event it joins. Split prompts weigh each trait on its own terms for the call. The aim is sound auto codes that keep talk sense whole.
Catalog record
| CREATOR | Ying Na, Shihui Feng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19734
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understanding Ying Na, Shihui Feng Abstract Dialogue data has been a key source for understanding learning processes, offering critical insights into how students engage in collaborative discussions and how these interactions shape their knowledge construction. The advent of Large Language Models (LLMs) has introduced promising opportunities for advancing qualitative research, particularly in the automated coding of dialogue data. However, the inherent contextual complexity of dialogue presents unique challenges for these models, especially in understanding and interpreting complex contextual information. This study addresses these challenges by developing a novel LLM-assisted automated coding approach for dialogue data. The novelty of our proposed framework is threefold: 1) We predict the code for an utterance based on dialogue-specific characteristics—communicative acts and communicative events—using separate prompts following the role prompts and chain-of-thoughts methods; 2) We engaged multiple LLMs including GPT-4-turbo, GPT-4o, DeepSeek in collaborative code prediction; 3) We leveraged the interrelation between events and acts to implement consistency checking using GPT-4o. In particular, our contextual consistency checking provided a substantial accuracy improvement. We also found the accuracy of act predictions was consistently higher than that of event predictions. This study contributes a new methodological framework for enhancing the precision of automated coding of dialogue data as well as offers a scalable solution for addressing the contextual challenges inherent in dialogue analysis. 1 Introduction In recent years, large language models (LLMs) have gained considerable attention for their potential applications in qualitative research, particularly in deductive coding [ 1 ] . With their advanced text-processing capabilities, LLMs hold significant promise in automating and enhancing the analysis of complex text data. However, a critical limitation persists: the inability of LLMs to consistently understand and interpret context remains a major challenge [ 2 ] . Research has shown that LLMs often exhibit issues such as inconsistency [ 2 ] and contextual forgetfulness [ 3 ] , where they fail to maintain a coherent understanding of the data over time. This limitation becomes particularly pronounced and may introduce additional biases when analyzing dialogue data, such as interviews, which are inherently complex, context-dependent, and characterized by dynamic, interactive exchanges [ 4 ] . Dialogue, as a dynamic process, involves multiple layers of interaction and meaning-making, and its complexity is addressed through various analytical frameworks [ 23 ] .Understanding dialogue data is crucial for gaining insights into student engagement in learning processes, particularly in collaborative learning contexts [ 5 ] . Within the Ethnography of Communication, dialogue is categorized into distinct communicative components: Communicative Acts (CAs), Communicative Events (CEs), and Communicative Situations (CSs) [ 6 ] . Communicative Acts, such as questioning or responding, are the fundamental units of dialogue. These acts combine to form Communicative Events, which encompass coherent sequences involving participants, purposes, tasks, and themes. At the highest level, Communicative Situations provide the broader contextual and structural framework within which dialogue unfolds [ 6 ] . In dialogic teaching and learning environments, dialogue reflects this hierarchical structure, comprising multiple layers of interaction—each containing distinct events, and each event comprising various acts [ 7 ] . From the individual acts to the overarching situational context, each layer shapes the communication dynamics, influencing how meaning is constructed and exchanged throughout the learning process [ 6 ] . Effectively analyzing dialogue requires capturing these interconnected dimensions, which poses a significant challenge for large language models (LLMs). Currently, many coding frameworks either merge events and acts into a single code or only focus on events. In our study, we aim to address this limitation by separating events and acts as independent communicative components for prediction within each dialogue interaction. We can leverage the inherent complexity of dialogue and the interdependencies between different communicative components to assist LLMs in understanding the intricate nature of dialogue data, thereby reducing their cognitive burden. This study seeks to bridge these gaps by developing an LLM-assisted coding approach specifically designed for dialogue data. By harnessing advanced LLMs and capturing the multi-dimensional and dynamic characteristics of dialogue, we aim to improve coding accuracy and address the challenges LLMs encounter in interpreting complex dialogue contexts. Our primary research question is: How can LLM-assisted methods address the challenges of understanding complex dialogue data contexts and overcome LLMs’ limitations in maintaining contextual consistency? 2 Related Work In recent years, qualitative coding based on LLMs has yielded valuable findings and methodological insights, including various research frameworks that suggest LLMs hold potential for streamlining an automated qualitative data analysis [ 8 ] . Despite this promise, their application in the educational field remains limited, with human feedback still playing a critical role in the coding process [ 8 ] . Comparative studies have revealed that hybrid methods, which integrate LLMs at different stages of codebook development, tend to outperform fully automated approaches, as sole reliance on models like ChatGPT often fails to produce satisfactory results [ 9 ] . While LLMs have demonstrated potential in dialogue-related tasks, such as multi-turn dialogue understanding and spoken language comprehension, limitations remain [ 10 ] . Studies have indicated that although LLM-based coding techniques can achieve fair to substantial agreement with human raters in context-independent tasks, they face challenges in context-dependent coding dimensions [ 11 ] . Autonomous coding of interview data by models like ChatGPT and Llama-2 has further highlighted the difficulty of adapting to nuanced, context-specific tasks, which are essential for qualitative research due to the intricate and flexible interpretations required [ 4 ] . To address these limitations, recent advancements have focused on leveraging contextual knowledge more effectively [ 12 ] . Approaches integrating retrieval-augmented generation (RAG) workflows have demonstrated superior performance by enabling LLMs to handle contextual information more efficiently [ 13 ] . Furthermore, research has underscored the importance of combining LLMs with traditional natural language processing (NLP) techniques to enhance coding at multiple levels, such as sentence, slot, and semantic coding [ 14 ] . These studies emphasize that LLMs must go beyond simple keyword recognition to capture deeper linguistic elements, including assumptions, values, metaphors, and power dynamics [ 14 ] . However, many existing approaches to coding dialogue data rely on prediction methods that are not specifically tailored to dialogue’s unique attributes. This lack of specialization limits their ability to consistently address contextual shifts and maintain coherence in analysis. Manual coding, while effective, is resource-intensive and impractical for large-scale datasets. As a result, there is a need for more efficient methods that address both the contextual intricacies of dialogue and the limitations of LLMs. 3 Methodology 3.1 Learning Context and Data Collection In this study, data was collected from student discussions during a collaborative problem-solving task conducted in a laboratory setting. 24 participants worked in 8 groups with three students in each group to complete a set of multiple choice questions and a short essay related to two education theories: Bloom’s Taxonomy [ 21 ] and the ICAP Framework [ 22 ] . Each group was tasked with completing a set of multiple-choice questions, which assessed their understanding of the components of the two theories. In addition, each group was required to write a short essay exploring the application and understanding of these two theories within practical teaching scenarios.