Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education

James Edgell, Wm. Matthew Kennedy, Isaac Pattis, Ben Knight, Danielle Carvalho, Elizabeth Wonnacott · 2026

Read Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education by James Edgell, Wm. Matthew Kennedy, Isaac Pattis, Ben Knight, Danielle Carvalho, Elizabeth Wonnacott (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20088.txt.

Introduction

Edgell and co-authors set out how to judge AI tools for second language education. Fast uptake of language models has outrun fair tests of teaching value. Past evaluations stayed narrow and task-specific. Their L2-Bench effort grounds scoring in a validated designer construct for learning experiences. The method joins pedagogical theory with sociotechnical evaluation practice. A layered taxonomy organizes over 1,000 real task-response pairs with expert rubric scores. A pilot check with 39 raters found the tasks authentic. Criteria scores were lower, and rater agreement was poor throughout.

Catalog record

CREATORJames Edgell, Wm. Matthew Kennedy, Isaac Pattis, Ben Knight, Danielle Carvalho, Elizabeth Wonnacott
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83723
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20088

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education James Edgell, Wm. Matthew Kennedy, Isaac Pattis, Ben Knight, Danielle Carvalho, Elizabeth Wonnacott Abstract The rapid adoption of large language models in AI-powered language education has created an urgent need for evaluations that assess pedagogical effectiveness, particularly in language learning—one of the most common LLM use cases ( 38 ; 13 ) . With only narrowly defined task-specific evaluations of AI system capabilities in second language (L2) education existing in the literature, we require more holistic approaches in this AI for education space. To address this gap, we introduce L2-Bench, a novel evaluation benchmark grounded in a validated “language learning experience designer” construct to assess AI capabilities across L2 education contexts. Our methodology integrates pedagogical theory, sociotechnical AI evaluation methods, and operationalizes a hierarchical taxonomy to structure an expert-curated dataset of over 1,000 authentic rubric-scored task-response pairs with measurement and scoring pipeline. We report the results of a pilot validation exercise (N = 39) on an initial sample of our dataset (tasks were validated as authentic [M = 4.23/5], but criteria scores were lower [M = 3.94], with universally poor inter-annotator agreement despite good internal consistency), alongside the experimental design for our follow-up practitioner data validation study as we iterate and scale to the full dataset. Ultimately, this research not only offers methodological lessons towards a more context-specific AI evaluations ecosystem, but also works towards better design of reproducible evaluations for AI systems deployed to educational contexts. 1 Introduction Access to language education is a human right ( 45 ) . Large Language Models (LLMs) are rapidly being integrated into language learning products used by millions of learners worldwide. However, no widely recognised benchmarks exist for evaluating AI capabilities in education beyond narrow lesson generation ( 12 ) or knowledge of pedagogical concepts ( 25 ) . Although important aspects, learning experience design requires knowing when and how to use these capabilities in practice, reflecting the diverse educational scenarios encountered in real-world settings. The absence of comprehensive evaluation frameworks for AI in education creates several critical challenges. Educators and institutions lack systematic methods to assess AI capabilities for specific teaching scenarios, leading to uninformed adoption decisions. Product developers cannot rigorously validate their AI implementations against pedagogical best practices. Most importantly, the lack of standardized evaluation impedes the development of more effective AI-powered educational systems. In this paper, we introduce L2-Bench (see Appendix C.1 for glossary): an evaluation benchmark assessing AI performance across diverse tasks comprising second language “learning experience design”. We discuss our iterative efforts to develop and validate a methodology for building L2-Bench, and in doing so, we make two contributions: (1) We define a hierarchical competency taxonomy grounded in established pedagogical frameworks - emphasising learning design as both a major real-world use case ( 38 ; 13 ) , and a necessary condition for any application of AI to educational spaces; and (2) In the interests of open science, we report the results (positive and negative) of our efforts to validate our core technical components: taxonomy, measures, and an initial sample (325 items) of our planned 1,300 item dataset. 2 Related work The speed and scale of AI system deployment have outpaced the development of evaluation methodologies for novel technologies and real-world contexts ( 18 ; 7 ; 33 ; 34 ) , prompting calls for greater rigor, systematization, and the incorporation of social scientific methods ( 9 ; 44 ; 28 ) . This evaluations crisis is acute in education, where no generally accepted holistic evaluations for AI in education exist despite rapid adoption ( 16 ; 13 ) . While some evaluations assess performance in highly rules-based domains (e.g., mathematics or computer science), these are not readily transferable to more open-ended educational settings. Conversely, benchmarks claiming to measure broad constructs such as knowledge or reasoning risk importing inappropriate measurement techniques into educational contexts. These limitations create conditions for substantial, compounding risk ( 6 ; 21 ) . Evaluations for AI in education are only beginning to emerge. A growing body of scholarship suggests AI use may yield marginal learning benefits but at a cost to engagement ( 30 ; 27 ) . Google DeepMind’s LearnLM team has produced a series of evaluations during the development of a tutorial chatbot ( 19 ) and, in partnership with Eedi, conducted a small (N = 165) RCT reporting a 5.5% improvement in independent problem solving over human tutoring alone (93.6% probability of a genuine improvement) ( 39 ) . However, these evaluations remain proprietary and focus narrowly on tutorial interaction, privileging instruction-following over holistic pedagogical adaptivity ( 24 ) . Other efforts are noteworthy but limited. 46 ’s general-purpose AI education evaluation lacks grounding in widely accepted pedagogical frameworks. Oak National Academy released a benchmark dataset for AI-generated educational content safety, though its scope is restricted to safety concerns ( 11 ) . 21 ’s taxonomy of AI harms in education advances context-specific evaluation but has yet to be widely operationalized. 36 qualitative analysis of Khanmigo using 10 ’s CALL framework offers useful insights but relies on personal experience and thus remains anecdotal. 3 Building L2-Bench 3.1 Theory and design A gap remains. Current evaluations insufficiently cover educational contexts, particularly second-language learning, and frequently misgeneralize tutorial interaction to group instruction. We propose what we understand to be the first AI evaluation methodology for language learning and among the first holistic evaluations of AI in educational contexts. Context-specificity is key. Language education is different from most other areas of education: it requires learners to acquire more implicit knowledge and proceduralised skills than other subjects; it comprises both the target of learning and the means of learning; it is heavily influenced by affective factors (motivation, identity, anxiety, confidence, and willingness to communicate) ( 29 ) ; and it is fundamentally shaped by each learner’s own social and cultural experience ( 31 ) . As a result, language is never “solved” ( 15 ) . Furthermore, classroom learning is not the aggregation of individual interactions; knowledge is produced through social interaction ( 5 ; 20 ) . Because AI systems shape learning both directly and indirectly through instructional materials, evaluation must extend beyond subject-matter knowledge to capture learning experience design ( 22 ; 21 ) . Indeed, our evaluation is more interested in assessing LLM capabilities in designing experiences that promote the “doing,” not diagramming, of language ( 35 ; 3 ) . As the contours of language learning are unique, we aim to produce evaluation artefacts that are each representative of the peculiar “vernacular” ( 20 ) of language learning. We draw upon three frameworks common to UK and EU language learning design: the Council of Europe’s Common European Framework of Reference for Languages (CEFR) ( 14 ) , the Eaquals Framework for Language Teacher Training and Development ( 17 ) and the British Council’s Continuing Professional Development Framework ( 8 ) . We also engage experts in a series of practitioner validation exercises. Although one weakness of this approach is that non-European language pedagogy principles were not reflected, we believe our method allows alternative frameworks to be easily utilized. Also, our benchmark initially intends to assess model

Read the complete text