From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis

Eason Chen Isabel Wang Nina Yuan, Sophia Judicke Kayla Beigh Xinyi Tang · 2026

Read From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis by Eason Chen Isabel Wang Nina Yuan, Sophia Judicke Kayla Beigh Xinyi Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27440.txt.

Introduction

Chen and co-authors use coding agents to label tutoring chats. Hand coding of student talk is slow and costly. In their loop, an agent tests a classifier on human labels. It studies each mismatch and drafts prompt fixes. A researcher reviews each change before use. Across 659 sessions, the best agent hit a kappa of 0.78. That score matches human agreement rates. Costs ran about five to eight dollars per agent. Extra rounds past the peak hurt scores, so held-out checks matter.

Catalog record

CREATOREason Chen Isabel Wang Nina Yuan, Sophia Judicke Kayla Beigh Xinyi Tang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52302
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis Eason Chen Isabel Wang Nina Yuan, Sophia Judicke Kayla Beigh Xinyi Tang Abstract Behavioral analysis of tutoring dialogues is essential for understanding student learning, yet manual coding remains a bottleneck. We present a methodology where LLM coding agents autonomously improve the prompts used by LLM classifiers to label educational dialogues. In each iteration, a coding agent runs the classifier against human-labeled validation data, analyzes disagreements, and proposes theory-grounded prompt modifications for researcher review. Applying this approach to 659 AI tutoring sessions across four experiments with three agents and three classifiers, 4-fold cross-validation on held-out data confirmed genuine improvement: the best agent achieved test κ = 0.78 \kappa=0.78 (SD = 0.08 =0.08 ), matching human inter-rater reliability ( κ = 0.78 \kappa=0.78 ), at a cost of approximately $5–8 per agent. While development-set performance reached κ = 0.91 \kappa=0.91 – 0.93 0.93 , the cross-validated results represent our primary generalization claim. The iterative process also surfaced an undocumented labeling pattern: human coders consistently treated expressions of confusion as engagement rather than disengagement. Continued iteration beyond the optimum led to regression, underscoring the need for held-out validation. We release all prompts, iteration logs, and data. keywords LLM labeling, behavioral analytics, prompt engineering, AI tutoring, educational data mining 1 Introduction Understanding student behavior in tutoring dialogues is fundamental to improving educational technology. Detecting patterns such as help-seeking, confusion, and disengagement enables researchers to evaluate tutoring effectiveness and design adaptive interventions [ 2 , 1 ] . Yet behavioral labeling remains a costly bottleneck: trained coders must manually analyze hundreds of sessions, a process requiring weeks of effort and limiting the scale of educational dialogue research. Large language models (LLMs) offer a promising solution, achieving strong performance on qualitative coding tasks [ 6 , 5 ] . Recent work has increasingly adopted LLM-assisted and hybrid annotation pipelines for educational data [ 6 , 16 ] , reducing but not eliminating the need for human expertise in prompt development. However, developing effective prompts still relies heavily on researcher effort: researchers iterate through trial-and-error, analyze errors, and refine instructions over days or weeks. This bottleneck limits the scalability and reproducibility of LLM-assisted educational research. We present an alternative approach where LLM coding agents serve not as tools to be configured, but as research teammates that autonomously iterate on labeling prompts through systematic error analysis. Applying this methodology to behavioral labeling in AI tutoring dialogues, we address practical questions: How many iterations are needed? How much does it cost? Do different agents produce comparable results? And critically, when should iteration stop? Experiment 1 explored this collaborative workflow. A researcher worked with Claude Code over 21 iterations, improving overall from \kappa=0.77 to \kappa=0.78 , with Follow-up Type improving from \kappa=0.59 to \kappa=0.68 , at a total cost of $22. Beyond the quantitative improvement, this process yielded a pedagogical insight: the agent’s systematic error analysis revealed that human coders consistently labeled expressions of confusion (“I don’t know,” “I’m stuck”) as engagement rather than disengagement, a pattern not explicitly documented in our original codebook. This raised our core R esearch Q uestion: Can LLM agents reliably improve prompts for educational content labeling tasks? Experiment 2 validated the methodology’s generalizability. We assigned the same labeling task to three independent coding agents ( OpenAI Codex , Claude Code , and Gemini ), each starting from scratch with a minimal baseline prompt. Starting from baseline \kappa\approx 0.70 , after approximately 10 iterations ( \sim $5 per agent), all three agents improved prompts substantially. While development-set performance reached \kappa=0.91 – 0.93 , our primary evaluation uses held-out cross-validation: 4-fold cross-validation confirmed genuine improvement, with the best agent achieving held-out test \kappa=0.78 (SD =0.08 ), matching human inter-rater reliability ( \kappa=0.78 ). Importantly, agents independently converged on similar disambiguation rules, suggesting the methodology identifies genuine patterns rather than memorizing specific examples. We also investigated continued iteration beyond the optimal point. When we extended Codex from v7 ( \kappa=0.93 ) through v10, performance regressed to \kappa=0.91 , demonstrating that more iterations do not always yield improvements. Experiments 3–4 examined whether performance differences stemmed from a “same-family advantage,” where agents perform better when the classifier shares their architecture. When we switched to Claude Opus 4.5 as classifier (Experiment 3), Claude Code achieved \kappa=0.91 . When we further tested Gemini 3 Pro as classifier (Experiment 4), Claude Code achieved its best performance ( \kappa=0.89 ), while Gemini Agent reached \kappa=0.92 with its same-family Gemini classifier. A key finding emerged: same-family advantages manifest primarily as faster convergence rather than higher performance ceilings, with GPT classifier achieving the highest final performance across agents. Our contributions span methodology, empirical findings, and transparency. We introduce a collaborative workflow where LLM coding agents autonomously execute develop-test-analyze cycles while researchers provide theoretical oversight. We provide empirical evidence that iterative refinement enables substantial performance gains, with Codex and Gemini achieving the highest performance ( \kappa=0.93 ). Through systematic comparison of three classifiers ( GPT , Claude , Gemini ), we reveal that classifier choice affects both performance ceiling and convergence speed, with same-family classifiers enabling faster iteration while GPT classifier achieved the highest final performance. Our analysis of iteration dynamics demonstrates that continued iteration can lead to regression, not just diminishing returns, underscoring the importance of using separate validation sets to select the best-performing iteration, as in standard machine learning practice. Along with the paper, we released documentation of iteration logs, costs, and failures to enable replication. Finally, we show how systematic disagreement analysis can surface implicit labeling criteria that human coders apply but have not explicitly documented. 2 Related Work Educational Dialogue Analysis. Dialogue-based tutoring systems have a rich research tradition, from early systems like CIRCSIM-Tutor and AutoTutor to modern LLM-powered chatbots. Analyzing student behavior in these dialogues reveals learning processes invisible in outcome measures alone. Baker et al. [ 2 ] pioneered automated detection of “gaming the system,” while Aleven et al. [ 1 ] distinguished productive help-seeking from answer-seeking behavior. Recent work has extended behavioral detection to confusion, disengagement, and self-regulated learning patterns [ 20 ] . However, behavioral labeling at scale remains labor-intensive: most studies rely on small, manually-coded samples that limit generalizability. Our work addresses this bottleneck by enabling scalable, reliable behavioral coding through LLM agent collaboration. LLMs for Qualitative Coding. LLMs have shown promise for qualitative analysis in education [ 6 , 5 ] , including classroom discussion assessment [ 16 ] and self-regulated learning detection [ 20 ] . Recent work has compared LLM and human coding performance [ 12 , 7 ] , finding that task-specific finetuning often outperforms in-context learning for nuanced educational coding tasks. However, most studies lack documentation of iterative prompt development. Stamper et al. [ 15 ] advocate grounding LLM-based analysis in theoretical frameworks; our methodology embeds help-seeking theory [ 1 ] throughout the iteration process. Automated Prompt Optimization. Several methods automate prompt improvement through algorithmic search. APE [ 21 ] generates and selects instructions via LLM proposals, DSPy [ 10 ] compiles declarative programs into optimized prompts, and OPRO [ 18 ] uses LLMs as optimizers guided by past prompt-score pairs. Our approach shares the

Read the complete text