SciEval: A Benchmark for Automatic Evaluation of K–12 Science Instructional Materials
Zhaohui Li, Peng He, Zhiyuan Chen, Honglu Liu, Zeyuan Wang, Tingting Li, Jinjun Xiong · 2026
Read SciEval: A Benchmark for Automatic Evaluation of K–12 Science Instructional Materials by Zhaohui Li, Peng He, Zhiyuan Chen, Honglu Liu, Zeyuan Wang, Tingting Li, Jinjun Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25472.txt.
Introduction
Li and six colleagues put forth SciEval, a benchmark for automatic evaluation of K-12 science instructional materials. Reviewing materials is slow and hard to scale, yet educators increasingly use AI to make them. The authors frame Automatic Instructional Materials Evaluation, or AIME, as a task predicting scores and evidence from a rubric. They curate SciEval: 273 materials across 13 criteria, 3,549 ratings, using the EQuIP rubric. They test GPT, Gemini, Llama, and Qwen and find none strong. Fine-tuning Qwen3 gains up to 11 percent on held-out tests.
Catalog record
| CREATOR | Zhaohui Li, Peng He, Zhiyuan Chen, Honglu Liu, Zeyuan Wang, Tingting Li, Jinjun Xiong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36314 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SciEval: A Benchmark for Automatic Evaluation of K–12 Science Instructional Materials Zhaohui Li, Peng He, Zhiyuan Chen, Honglu Liu, Zeyuan Wang, Tingting Li, Jinjun Xiong Abstract The need to evaluate instructional materials for K–12 science education has become increasingly important, as more educators use generative AI to create instructional materials. However, the review of instructional materials is time-consuming, expertise-intensive, and difficult to scale, motivating interest in automated evaluation approaches. While large language models (LLMs) have shown strong performance on general evaluation tasks, their performance and reliability on instructional materials remain unclear. To address this gap, we formulate Automatic Instructional Materials Evaluation (AIME) as a generative AI task that predicts scores and evidence using the rubric designed by the educator. We create a benchmark dataset and develop baseline models for AIME. First, we curate the first AIME dataset, SciEval , consisting of instructional materials annotated with pedagogy-aligned evaluation scores and evidence-based rationales. Expert annotations achieve high inter-rater reliability, resulting in a dataset of 273 lesson-level instructional materials evaluated across 13 criteria (N=3549) using the EQuIP rubric. Second, we test mainstream LLMs (GPT, Gemini, Llama, and Qwen) on SciEval and find that none achieve strong performance. Then we fine-tune Qwen3 on SciEval . Results on a held-out test set show that domain-aligned fine-tuning can achieve up to 11% performance gains, highlighting the importance of domain-specific fine-tuning for AIME and facilitating the use of LLMs in other educational tasks. 1 Introduction High-quality instructional materials are central to achieving equitable, standards-aligned K-12 science education since they provide benefits to students’ learning and teacher professional learning [ 17 , 3 , 5 , 22 ] . Recently, an increasing number of educators have begun using generative AI tools, such as ChatGPT, to create instructional materials, including lesson plans, assessment questions, and classroom activities. This phenomenon brings the need for automated evaluation of instructional materials, since manual evaluation is time-consuming and lacks reasoning details, with bias and errors, and limits scalability [ 19 , 4 , 11 ] . Recent research found that LLMs can be used for instructional material generation and evaluation, but LLMs struggle with pedagogical criteria, such as the depth of pedagogical reasoning and the coherence of instructional decisions [ 12 , 18 ] . In addition, most existing AI research focuses on content generation with human evaluation, but pays limited attention to systematically evaluating reliability and alignment with an official pedagogical rubric [ 36 , 31 , 37 , 28 ] . Figure 1 : The left summarizes the EQuIP rubric criteria, and the right shows the corresponding lesson components, illustrating how rubric criteria align with instructional content for evaluation. Thus, we propose a new interdisciplinary task, Automatic Instructional Materials Evaluation (AIME), for both education and computer science communities. As illustrated in Figure 1 , AIME aims to evaluate instructional materials by producing an overall quality score accompanied by evidence for every rubric dimension, which requires an algorithm to connect EQuIP [ 2 ] rubric criteria with the relevant content in instructional materials automatically. AIME benefits educators by reducing the assessment burden and has the potential to provide timely and detailed feedback. For AI research, AIME shifts the formulation of educational data evaluation from a classification problem that gives a score to a task that uses LLMs to generate feedback with evidence, which is more helpful and advanced in AI for Education [ 35 , 1 , 6 ] . The key challenge of AIME is how to extract the evidence related to a rubric criterion in a long context. Instructional materials are often presented in long PDF file, averaging 25 pages and about 6,000 words in our data. This causes the inherently long sequence modeling difficulty, which often leads to performance degradation in transformer-based LLMs [ 21 ] . Furthermore, there is no reliable publicly available dataset for model training, particularly for LLM fine-tuning approaches that perform best on other educational tasks. To address these challenges, we created the first AIME dataset for K-12 science education, named SciEval, from validated online open-source organizations, including OpenSciEd, BSCS, CREATE for STEM Institute, and the K-12 Alliance at WestEd with the NGSS Quality Badge. Annotations were produced by two trained science education researchers using a multi-round training, calibration, and adjudication process guided by senior experts, resulting in high inter-rater reliability (89.7% of agreement). The final dataset contains 273 lesson-level instructional materials, yielding 3,549 criterion-level annotation scores with evidence-based rationales. Building on SciEval, we conduct a three-step study to benchmark AIME task. First, we co-designed and examined three structured prompts 1 with science education researchers. We found that the simplified prompt achieves the highest performance. Second, we benchmarked both commercial models and open-source instruction-tuned LLMs, and found that Qwen3-4B-Instruct [ 33 ] achieved the strongest performance. Finally, we apply data augmentation on training data and fine-tune Qwen3-4B-Instruct with LoRA-based supervised fine-tuning, and techniques [ 9 , 15 ] . Results show fine-tuning improves performance on the score, while evidence grounding remains at the same level and motivates further work on long-context retrieval and evidence evaluation. Our contributions are as follows. (1) We propose AIME, which evaluates instructional materials to support the safe and sustainable use of AI in education, and present SciEval, the first large-scale benchmark dataset for this task. (2) We benchmark mainstream commercial and open-source LLMs on SciEval. (3) We design a task-specific LLM fine-tuning approach with data augmentation for AIME, enabling stronger domain adaptation and improved performance that can be generalized to other educational tasks. 2 Related works Science educators in the United States have a strong consensus around A Framework for K–12 Science Education [ 23 ] and the Next Generation Science Standards (NGSS) [ 25 ] . The EQuIP rubric provides structured criteria for assessing alignment to NGSS [ 2 ] . Research has examined the design and implementation of NGSS-aligned materials and the practical challenges teachers face in using them effectively [ 29 ] . Studies have highlighted difficulties in applying these evaluation frameworks reliably across settings [ 26 ] . However, most of this work has focused on human-led processes and has not yet explored automated AI evaluation. Several recent studies have explored LLMs for lesson plan research, but most focus on content creation rather than pedagogy-aligned evaluation. EduGPT developed a generation model based on prompt engineering on top of GPT-4 and evaluated on three self-defined dimensions: Granularity, Accuracy, and Structure [ 34 ] . ELF proposes a GPT Framework to generate teacher–student dialogical content to support classroom instruction [ 31 ] . Their evaluation relies on perplexity-based statistical metrics and human questionnaires on two self-collected datasets. LessonPlanLM [ 36 ] defines three core features of high-quality lesson plans and fine-tunes a Qwen2-based model using over 100,000 self-collected plans. While this work advances large-scale lesson plan generation, the proposed features are not aligned with educational standards, and the dataset is not designed for automated evaluation. Similarly, LANG [ 28 ] collects 10,000 high-quality lesson plans and integrates RAG with LLMs to improve accuracy and reduce hallucinations. Those approaches emphasize generation quality rather than evaluation reliability. Hauk’s work [ 12 ] reports that LLMs can approximate human judgments on some criteria, but struggle with higher-order pedagogical reasoning, such as