Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration
2026
Read Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13353.txt.
Introduction
Bakhtawar Ahtisham, Kirk Vanacore, and Rene F. Kizilcec study LLM annotation of classroom discourse. LLMs can label teaching data fast, but single-pass outputs are shaky for high-stakes constructs. Such work needs context and pedagogical judgment, as with instructional intent or discourse moves. Their framework treats annotation as a staged process, not a one-shot prediction. Models first label alone, then audit their own labels against the rubric, then an adjudicator settles disputes. The design is cost-aware and mirrors how expert human teams annotate.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9887 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration Abstract BAKHTAWAR AHTISHAM, Cornell University, USA KIRK VANACORE, Cornell University, USA RENE F. KIZILCEC, Cornell University, USA Keywords: Large Language Models, Automated Annotation, Multi-Agent Systems, Annotation Reliability Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learning artifacts [2, 8,9,10]. Their ability to rapidly summarize instructional interactions and assign rubric-aligned labels has fueled optimism about reducing the cost and time associated with expert human annotation [1, 2]. However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs that require contextual, pedagogical, or normative judgment, such as instructional intent or discourse moves [3, 4]. This tension between scale and validity sits at the core of contemporary education data science. In this work, we present and empirically evaluate a hierarchical, cost-aware orchestration framework for LLM-based annotation that improves reliability while explicitly modeling computational tradeoffs. Rather than treating annotation as a one-shot prediction problem, we conceptualize it as a multi-stage epistemic process comprising (1) an unverified single-pass annotation stage, in which models independently assign labels based on the rubric; (2) a self-verification stage, in which each model audits its own output against rubric definitions and revises its label if inconsistencies are detected; and (3) a disagreement-centric adjudication stage, in which an independent adjudicator model examines the verified labels and justifications and determines a final label in accordance with the rubric. This structure mirrors established human annotation workflows in educational research, where initial coding is followed by self-checking and expert resolution of disagreements [5, 6]. Data and Task We evaluate our framework on a classroom discourse annotation task grounded in authentic K–12 mathematics instruction. The dataset comprises 63 classroom transcripts spanning whole-class discussion, small-group work, and online lessons. Teacher utterances were annotated by expert educators using the Talk Moves coding scheme [ talkmoves ] , achieving high inter-rater reliability (Cohen’s \kappa>0.90 ). To balance construct coverage with computational feasibility, we applied proportional stratified sampling to select 800 target teacher utterances, preserving the empirical distribution of Talk Moves categories. These utterances were embedded within their surrounding conversational context, yielding 467 dialogue segments containing over 13,000 utterances. Only teacher turns were evaluated; student turns were included solely as contextual input. We evaluate six annotation strategies organized into two pipelines. The first pipeline uses standard language models and includes single-pass annotation , self-verified annotation , and adjudicated annotation . The second pipeline uses reasoning-enabled models and includes reasoning-annotated , reasoning-verified , and reasoning-adjudicated strategies. These strategies are tested across six production LLM variants from three model families—GPT, Claude, and Gemini—including both reasoning-enabled models (GPT-5.2 Thinking, Claude 4.5 Opus, Gemini 3 Pro) and non-reasoning models (GPT-4.1, Claude 4.5 Sonnet, Gemini 3 Flash). This design allows us to disentangle the effects of model capability and orchestration structure. For each strategy, we additionally track prompt tokens, completion tokens, and total token usage to enable cost-aware comparisons across pipelines. Results Figure 1: TalkMoves Per-Category Performance Across Orchestration Strategies. Average F1 scores for seven TalkMoves categories under different annotation orchestration strategies. Bars show mean performance averaged across three LLMs (GPT, Claude, Gemini), while markers indicate individual model scores (shapes). Categories are ordered from left to right by increasing baseline difficulty. Per-Category Performance Gains. Fig. 1 presents per-category F1 scores for seven Talk Moves categories across all orchestration strategies, averaged across models, with markers indicating individual model performance. Across all categories, hierarchical orchestration yields consistent improvements over single-pass annotation. These gains are largest for pedagogically demanding categories that require interpretive or normative judgment. For example, Revoicing improves from an average F1 of approximately 0.30 under single-pass annotation to 0.44–0.46 under verification and 0.52–0.54 under adjudication, corresponding to a \sim 70–80% relative improvement over the baseline. Similarly, Press for Reason increases from \sim 0.40 to \sim 0.55–0.58 and \sim 0.60–0.63, representing a \sim 45–55% relative gain (Fig. 1 ; Table 1). In contrast, structurally simpler categories such as Restating exhibit smaller but still systematic improvements, suggesting that orchestration confers benefits even when surface-level linguistic cues are more readily available. Together, these results indicate that hierarchical orchestration benefits categories with higher conceptual and pedagogical complexity more, while offering consistent gains across the full task spectrum. Figure 2: TalkMoves Token Cost vs. Average Performance for Reasoning and Non-Reasoning Pipelines. Average F1 score as a function of total token usage for TalkMoves annotation pipelines. Points trace the progression from single-pass annotation to verification and adjudication for non-reasoning and reasoning-based strategies. Reasoning vs. Non-Reasoning Tradeoffs. Reasoning-enabled models exhibit higher baseline performance under single-pass annotation. However, this advantage narrows substantially under orchestration. Figure 2 plots the average F1 against total token usage for Talk Moves pipelines. While reasoning-based adjudication achieves the highest absolute performance ( \sim 0.61 F1), non-reasoning adjudication reaches nearly the same level ( \sim 0.60 F1) at substantially lower token cost ( \sim 20–25% fewer tokens). This reveals a clear Pareto tradeoff: reasoning improves initial annotation quality, but once verification and adjudication are introduced, orchestration structure—not model reasoning alone—becomes the dominant driver of reliability. Across multiple categories, non-reasoning models paired with adjudication match or outperform reasoning-enabled models with verification alone, despite lower per-call complexity. These results suggest that, for large-scale educational annotation pipelines, investing in orchestration—rather than relying exclusively on more expensive reasoning-capable models—can yield comparable reliability at lower cost. Taken together, the findings position hierarchical orchestration as a practical optimization strategy for scaling trustworthy LLM-based annotation in education data science. Category Model Non-Reasoning Annotated Non-Reasoning Verified Non-Reasoning Adjudicated Reasoning Annotated Reasoning Verified Reasoning Adjudicated Keep Together Gemini 0.22 0.44 0.56 0.33 0.44 0.57 GPT 0.20 0.41 0.52 0.30 0.40 0.53 Claude 0.19 0.42 0.53 0.31 0.41 0.54 Revoicing Gemini 0.21 0.32 0.34 0.31 0.32 0.35 GPT 0.21 0.29 0.30 0.28 0.28 0.31 Claude 0.20 0.30 0.31 0.29 0.29 0.32 Press Reason Gemini 0.42 0.46 0.48 0.45 0.46 0.49 GPT 0.38 0.43 0.44 0.42 0.42 0.45 Claude 0.40 0.44 0.46 0.43 0.44 0.47 Relate Gemini 0.52 0.56 0.58 0.55 0.56 0.59 GPT 0.48 0.53 0.54 0.52 0.52 0.55 Claude 0.50 0.55 0.56 0.54 0.54 0.57 Press Accuracy Gemini 0.54 0.60 0.62 0.59 0.60 0.63 GPT 0.50 0.56 0.57 0.55 0.55 0.58 Claude 0.52 0.58 0.60 0.57 0.58 0.61 None Gemini 0.58 0.63 0.65 0.62 0.63 0.67 GPT 0.54 0.60 0.61 0.58 0.59 0.62 Claude 0.56 0.61 0.63 0.60 0.61 0.64 Restating Gemini 0.60 0.64 0.66 0.63 0.64 0.67 GPT 0.56 0.61 0.62 0.60 0.60 0.63 Claude 0.58 0.62 0.64 0.61 0.62 0.65 Table 1: F1 Scores for TalkMoves Categories by Model and Orchestration Strategy. Bold values indicate the highest F1 scores within each category, demonstrating that multi-model adjudication consistently provides better alignment with the ground truth. Implications for Education Data Science