EduCoder: An Open-Source Annotation System for Education Transcript Data
Guanzhong Pan, Mei Tan, Hyunji Nam, Lucía Langlois, James Malamut, Liliana Deonizio, Dorottya Demszky · 2025
Read EduCoder: An Open-Source Annotation System for Education Transcript Data by Guanzhong Pan, Mei Tan, Hyunji Nam, Lucía Langlois, James Malamut, Liliana Deonizio, Dorottya Demszky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05385.txt.
Introduction
Saad Ashraf, Dorottya Demszky, and their co-authors release an open tool for coding classroom talk. Their paper, from twenty twenty-five, presents EduCoder. Researchers label transcripts line by line to study teaching. General tools fit this job poorly, since class talk mixes teacher, student, and peer speech in complex ways.
EduCoder lets teams build shared codebooks from real data. It supports both fixed-choice and free-text labels. Lesson context, like the aim of the class, sits beside each line. Two coders' work can be viewed side by side to iron out gaps. The system is free and open for any team to use.
Catalog record
| CREATOR | Guanzhong Pan, Mei Tan, Hyunji Nam, Lucía Langlois, James Malamut, Liliana Deonizio, Dorottya Demszky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05385
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EduCoder: An Open-Source Annotation System for Education Transcript Data Guanzhong Pan, Mei Tan, Hyunji Nam, Lucía Langlois, James Malamut, Liliana Deonizio, Dorottya Demszky Abstract We introduce EduCoder , a domain-specialized tool designed to support utterance-level annotation of educational dialogue. While general-purpose text annotation tools for NLP and qualitative research are abound, few address the complexities of coding education dialogue transcripts—with diverse teacher-student and peer interactions. Common challenges include defining codebooks for complex pedagogical features, supporting both open-ended and categorical coding, and contextualizing utterances with external features, such as the lesson’s purpose and the pedagogical value of the instruction. EduCoder is designed to address these challenges by providing a platform for researchers and domain experts to collaboratively define complex codebooks based on observed data and incorporates both categorical and open-ended annotation types along with contextual materials. Additionally, it offers a side-by-side comparison of multiple annotators’ responses, allowing comparison and calibration of annotations with others to improve data reliability. The system is open-source 1 1 1 https://anonymous.4open.science/r/EduCoder-5878 with demo video available 2 2 2 https://www.youtube.com/watch?v=5YFR2seN0_k&t=4s . 1 Introduction Teacher-student and student-student dialogue play a critical role in learning, offering insight into instruction quality, student reasoning and understanding ( Mercer and Howe 2012 ; Michaels et al. 2008 ) . In recent years, the release of an increasing number of conversational datasets reflects growing interest in analyzing such dialogues ( Stasaski et al. 2020 ; Suresh et al. 2022 ; Demszky and Hill 2023 ; Wang et al. 2024 ; Macina et al. 2023 ) . Since manual analysis is labor-intensive and difficult to scale, there has been growing efforts to develop natural language processing (NLP) based measures of educationally relevant constructs in transcript data ( D’Mello and Graesser 2012 ; Jensen et al. 2021 ; Hunkins et al. 2022 ; Demszky et al. 2021 ; Alic et al. 2022 ; Suresh et al. 2018 ; Zhao et al. 2024 ) . However, progress in NLP model development for analyzing teachers’ and students’ talk moves hinges on rigorous and thoughtful human annotation, critical for model validation and training. Challenges & Motivation. Rigorous human annotation of educational dialogue is nontrivial, reconciling literature-based learning theories, practitioner expertise, linguistic features, and contextual information. It follows an iterative calibration process where researchers and practitioners collaboratively specify annotation codebook. In addition to conceptual factors, there are multiple practical factors making process quite challenging. First, no purpose-built, adaptive tool exists for annotating educational transcripts with iterative co-design and calibration support. Most existing platforms target general NLP taggings like named entity recognition (NER) or part-of-speech (POS) tagging, and lack support for annotating high-inference pedagogical features such as the discourse moves of teachers and students–and the latent cognitive processes they enact ( Nakayama et al. 2018 ; Honnibal and Montani 2017 ; Perry 2021 ) . Qualitative analysis platforms ATLAS.ti Scientific Software Development GmbH 2023 ; Dhakal 2022 support open coding but lack support for structured, utterance-level annotation and real-time analysis for collaborative calibration. As a result, many researchers rely on spreadsheets or ad hoc solutions, which are time-consuming to build, often clunky to use, and are poorly integrated with modern machine learning workflows. Second, while non-technical annotators—such as educators and qualitative researchers—may possess deep domain expertise, they often face significant barriers when using computational annotation tools . These challenges extend beyond a steep learning curve to include usability issues, interface complexity, and inefficiencies in accessing dispersed contextual information. Although conventional tools like spreadsheets or manual methods may be more accessible, they often lack the integrated features necessary for high-quality analysis—such as support for cross-annotator comparison, critical for effective annotator training and reliable results. Additionally, they typically do not provide a user-friendly interface that consolidates all relevant contextual materials into a single, centralized workspace for improving annotation efficiency and supporting annotator understanding. Third, there is growing interest in leveraging large language models (LLMs) as annotation tools to help meet the demand for high-quality labeled data. While LLMs have shown promising performance on tasks such as sentiment analysis, political stance detection, and educational content classification—sometimes achieving IRR comparable to that of humans in well-defined domains ( Bojić et al. 2025 ; Long et al. 2024 ) —their reliability remains less understood when applied to more complex or socially nuanced constructs. Recent work has proposed treating LLMs as additional “raters” to help quantify and control measurement error in human annotation processes Broska et al. 2025 ; Calderon et al. 2025 . Building on this line of research, we take it a step further: rather than treating LLMs as standalone annotators, we incorporate their outputs as one of several points of reference during human annotator calibration. This approach allows us to explore how LLMs might support, rather than replace, human judgment in complex annotation tasks. Our System EduCoder addresses these challenges through an intuitive interface designed for utterance-level annotation in educational dialogue analysis. The system supports: • Customizable Codebook: Researchers can upload their own codebook of features, allowing for tailored annotation that aligns with different research questions. • In-text Annotation & Contextual Materials: The tool features an intuitive, side-by-side interface that enables annotators to apply codes directly to individual transcript utterances. To facilitate contextually grounded and accurate annotation, the interface presents the full transcript alongside relevant contextual materials, including task instructions and screenshots of the questions students were engaging with. Codebook references are integrated directly into the interface to provide clear definitions of features and avoid confusion along with illustrative examples and non-examples. • Calibration Support: EduCoder offers built-in diagnostics for IRR ( Cohen 1960 ) , highlights disagreements to facilitate calibration, and enables comparison with other annotators to support iterative refinement of codebooks. In addition, the tool integrates annotations generated by LLMs, such as ChatGPT-4o ( OpenAI 2024 ) and Claude 4 Sonnet ( Anthropic 2025 ) , to facilitate human-AI collaboration in annotation. LLM annotations can serve as an immediate reference point for human coders and support calibration. By comparing their judgments with those of other humans and LLMs, annotators can identify discrepancies, refine interpretations, and enhance their understanding of the coding task. To demonstrate EduCoder ’s flexible design and ensure its accessibility, we provide an online playground 3 3 3 https://educoder.vercel.app/ with examples, including sample transcripts and a codebook, drawn from real-world classroom dialogue. We invite the community to use EduCoder , adapt our provided materials, and contribute to a growing ecosystem of tools for advancing educational dialogue research. Feature EduCoder INCEpTION Prodigy Doccano Label Studio LightTag POTATO AWOCATo EASE AutoDive+ Custom Codebook import ✓ ✓ Utterance-level annotation ✓ – – – – – – ✓ ✓ ✓ Real-time IRR & visual diff ✓ – – – – – Instructions & Contextual Materials ✓ ✓ ✓ ✓ LLM for Annotation ✓ – – – – – – – ✓ Usability for non-technical users ✓ – – ✓ ✓ ✓ ✓ ✓ ✓ ✓ Open-source / Free for research ✓ ✓ \times ✓ ✓ ✓ Free for Academia ✓ ✓ – License MIT Apache 2.0 Commercial MIT Apache 2.0 MIT Polyform Shield GPL GPL Request only Table 1: Comparison of EduCoder with widely used annotation tools across key capabilities relevant to educational dialogue annotation. A checkmark (✓) indicates the tool fully supports