Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

2026

Read Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13353.txt.

Introduction

Bakhtawar Ahtisham, Kirk Vanacore, and Rene F. Kizilcec study LLM annotation of classroom discourse. LLMs can label teaching data fast, but single-pass outputs are shaky for high-stakes constructs. Such work needs context and pedagogical judgment, as with instructional intent or discourse moves. Their framework treats annotation as a staged process, not a one-shot prediction. Models first label alone, then audit their own labels against the rubric, then an adjudicator settles disputes. The design is cost-aware and mirrors how expert human teams annotate.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE9887
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration


Abstract




BAKHTAWAR AHTISHAM, Cornell University, USA

KIRK VANACORE, Cornell University, USA

RENE F. KIZILCEC, Cornell University, USA

Keywords: Large Language Models, Automated Annotation, Multi-Agent Systems, Annotation Reliability

Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learning artifacts [2, 8,9,10]. Their ability to rapidly summarize instructional interactions and assign rubric-aligned labels has fueled optimism about reducing the cost and time associated with expert human annotation [1, 2]. However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs that require contextual, pedagogical, or normative judgment, such as instructional intent or discourse moves [3, 4]. This tension between scale and validity sits at the core of contemporary education data science. In this work, we present and empirically evaluate a hierarchical, cost-aware orchestration framework for LLM-based annotation that improves reliability while explicitly modeling computational tradeoffs. Rather than treating annotation as a one-shot prediction problem, we conceptualize it as a multi-stage epistemic process comprising (1) an unverified single-pass annotation stage, in which models independently assign labels based on the rubric; (2) a self-verification stage, in which each model audits its own output against rubric definitions and revises its label if inconsistencies are detected; and (3) a disagreement-centric adjudication stage, in which an independent adjudicator model examines the verified labels and justifications and determines a final label in accordance with the rubric. This structure mirrors established human annotation workflows in educational research, where initial coding is followed by self-checking and expert resolution of disagreements [5, 6].

Data and Task

We evaluate our framework on a classroom discourse annotation task grounded in authentic K–12 mathematics instruction. The dataset comprises 63 classroom transcripts spanning whole-class discussion, small-group work, and online lessons. Teacher utterances were annotated by expert educators using the Talk Moves coding scheme
[
talkmoves
]
, achieving high inter-rater reliability (Cohen’s
\kappa>0.90
). To balance construct coverage with computational feasibility, we applied proportional stratified sampling to select 800 target teacher utterances, preserving the empirical distribution of Talk Moves categories. These utterances were embedded within their surrounding conversational context, yielding 467 dialogue segments containing over 13,000 utterances. Only teacher turns were evaluated; student turns were included solely as contextual input.

We evaluate six annotation strategies organized into two pipelines. The first pipeline uses standard language models and includes
single-pass annotation
,
self-verified annotation
, and
adjudicated annotation
. The second pipeline uses reasoning-enabled models and includes
reasoning-annotated
,
reasoning-verified
, and
reasoning-adjudicated
strategies. These strategies are tested across six production LLM variants from three model families—GPT, Claude, and Gemini—including both reasoning-enabled models (GPT-5.2 Thinking, Claude 4.5 Opus, Gemini 3 Pro) and non-reasoning models (GPT-4.1, Claude 4.5 Sonnet, Gemini 3 Flash). This design allows us to disentangle the effects of model capability and orchestration structure. For each strategy, we additionally track prompt tokens, completion tokens, and total token usage to enable cost-aware comparisons across pipelines.

Results

Figure 1:
TalkMoves Per-Category Performance Across Orchestration Strategies. Average F1 scores for seven TalkMoves categories under different annotation orchestration strategies. Bars show mean performance averaged across three LLMs (GPT, Claude, Gemini), while markers indicate individual model scores (shapes). Categories are ordered from left to right by increasing baseline difficulty.

Per-Category Performance Gains.
Fig.
1
presents per-category F1 scores for seven Talk Moves categories across all orchestration strategies, averaged across models, with markers indicating individual model performance. Across all categories, hierarchical orchestration yields consistent improvements over single-pass annotation. These gains are largest for pedagogically demanding categories that require interpretive or normative judgment. For example,
Revoicing
improves from an average F1 of approximately 0.30 under single-pass annotation to 0.44–0.46 under verification and 0.52–0.54 under adjudication, corresponding to a
\sim
70–80% relative improvement over the baseline. Similarly,
Press for Reason
increases from
\sim
0.40 to
\sim
0.55–0.58 and
\sim
0.60–0.63, representing a
\sim
45–55% relative gain (Fig.
1
; Table 1). In contrast, structurally simpler categories such as
Restating
exhibit smaller but still systematic improvements, suggesting that orchestration confers benefits even when surface-level linguistic cues are more readily available. Together, these results indicate that hierarchical orchestration benefits categories with higher conceptual and pedagogical complexity more, while offering consistent gains across the full task spectrum.

Figure 2:
TalkMoves Token Cost vs. Average Performance for Reasoning and Non-Reasoning Pipelines. Average F1 score as a function of total token usage for TalkMoves annotation pipelines. Points trace the progression from single-pass annotation to verification and adjudication for non-reasoning and reasoning-based strategies.

Reasoning vs. Non-Reasoning Tradeoffs.
Reasoning-enabled models exhibit higher baseline performance under single-pass annotation. However, this advantage narrows substantially under orchestration. Figure
2
plots the average F1 against total token usage for Talk Moves pipelines. While reasoning-based adjudication achieves the highest absolute performance (
\sim
0.61 F1), non-reasoning adjudication reaches nearly the same level (
\sim
0.60 F1) at substantially lower token cost (
\sim
20–25% fewer tokens). This reveals a clear Pareto tradeoff: reasoning improves initial annotation quality, but once verification and adjudication are introduced, orchestration structure—not model reasoning alone—becomes the dominant driver of reliability. Across multiple categories, non-reasoning models paired with adjudication match or outperform reasoning-enabled models with verification alone, despite lower per-call complexity. These results suggest that, for large-scale educational annotation pipelines, investing in orchestration—rather than relying exclusively on more expensive reasoning-capable models—can yield comparable reliability at lower cost. Taken together, the findings position hierarchical orchestration as a practical optimization strategy for scaling trustworthy LLM-based annotation in education data science.

Category

Model

Non-Reasoning

Annotated

Non-Reasoning

Verified

Non-Reasoning

Adjudicated

Reasoning

Annotated

Reasoning

Verified

Reasoning

Adjudicated

Keep Together

Gemini

0.22

0.44

0.56

0.33

0.44

0.57

GPT

0.20

0.41

0.52

0.30

0.40

0.53

Claude

0.19

0.42

0.53

0.31

0.41

0.54

Revoicing

Gemini

0.21

0.32

0.34

0.31

0.32

0.35

GPT

0.21

0.29

0.30

0.28

0.28

0.31

Claude

0.20

0.30

0.31

0.29

0.29

0.32

Press Reason

Gemini

0.42

0.46

0.48

0.45

0.46

0.49

GPT

0.38

0.43

0.44

0.42

0.42

0.45

Claude

0.40

0.44

0.46

0.43

0.44

0.47

Relate

Gemini

0.52

0.56

0.58

0.55

0.56

0.59

GPT

0.48

0.53

0.54

0.52

0.52

0.55

Claude

0.50

0.55

0.56

0.54

0.54

0.57

Press Accuracy

Gemini

0.54

0.60

0.62

0.59

0.60

0.63

GPT

0.50

0.56

0.57

0.55

0.55

0.58

Claude

0.52

0.58

0.60

0.57

0.58

0.61

None

Gemini

0.58

0.63

0.65

0.62

0.63

0.67

GPT

0.54

0.60

0.61

0.58

0.59

0.62

Claude

0.56

0.61

0.63

0.60

0.61

0.64

Restating

Gemini

0.60

0.64

0.66

0.63

0.64

0.67

GPT

0.56

0.61

0.62

0.60

0.60

0.63

Claude

0.58

0.62

0.64

0.61

0.62

0.65

Table 1:
F1 Scores for TalkMoves Categories by Model and Orchestration Strategy. Bold values indicate the highest F1 scores within each category, demonstrating that multi-model adjudication consistently provides better alignment with the ground truth.

Implications for Education Data Science

Read the complete text