Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics
Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi · 2026
Read Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics by Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26607.txt.
Introduction
Bhusal and three colleagues tested varied language models for automating secondary mathematics competency assessment in Nepal. A human-in-the-loop frame scored models against faculty ground truth on Grade 10 optional mathematics across four topics and four skills. Gemini-based mixture-of-experts models reached fair agreement, while the larger Orion model showed no agreement at all. The architecture-compatibility gap suggests instruction-following outweighs raw parameter scale in rubric-bound tasks. Models remain helpers, not ready for autonomous certification.
Catalog record
| CREATOR | Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38942 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26607
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi Abstract As Competency-Based Education (CBE) is gaining traction around the world, the shift from marks-based assessment to qualitative competency mapping is a manual challenge for educators. This paper tackles the bottleneck issue by suggesting a “Human-in-the-Loop” benchmarking framework to assess the effectiveness of multiple LLMs in automating secondary-level mathematics assessment. Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation. The multi-provider ensemble, consisted of open-weight models—Eagle (Llama 3.1-8B) and Orion (Llama 3.3-70B) and proprietary frontier models Nova (Gemini 2.5 Flash) and Lyra (Gemini 3 Pro), was benchmarked against a ground truth defined by two senior mathematics faculty members ( κ w = 0.8652 \kappa_{w}=0.8652 ). The findings show a marked “Architecture-compatibility gap”. Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved “Fair Agreement” ( κ w ≈ 0.38 \kappa_{w}\approx 0.38 ), the larger Orion (70B) model exhibited ‘No Agreement’ ( κ w = − 0.0261 \kappa_{w}=-0.0261 ), suggesting that architectural compliance with instruction constraints outweighs the scale of raw parameters in rubric-constrained tasks. We conclude that while LLMs are not yet suitable for autonomous certification, they provide high-value assistive support for preliminary evidence extraction within a “Human-in-the-Loop” framework. 1 Introduction Assessing students’ performance has been a crucial part of academics historically. Traditionally, subjects such as mathematics were assessed based on students’ marks and scores instead of competencies. This displays a significant gap in the assessment system, as marks alone might not be the best metric to measure student understanding of specific topics. Competencies are defined as skills and knowledge areas required to demonstrate proficiency in a subject. A better example can be: to excel in AI, a scholar needs to have knowledge in domains like: theory, mathematics, applications and ethics. These form four core competencies in AI. A clear assessment of a student’s level in the competencies required to excel in the subject provides a more scientific, rich, and tangible analysis of where a student’s strengths lie and where weaknesses exist. This is where Competency-Based Education (CBE) plays a crucial role in addressing limitations of traditional assessment practices [ 1 ] . With the current reasoning capacity of Large Language Models (LLMs), free open-source versions and proprietary models can be leveraged to assess student performance beyond the quantitative window, including qualitative aspects, analogous to human supervision. LLMs can reason over open-ended answers; however, no accurate comparison of LLMs vs. expert teachers has yet been conducted at the competency-rubric level, especially in school mathematics. The Global South’s academic diaspora is particularly behind in this aspect, with many remaining gaps in integrating modern academic practices and bridging the digital divide [ 9 ] . In this study, we benchmark four LLMs with a human-in-the-loop approach to determine which models can best assess students based on the Grade 10 Optional Mathematics Syllabus of Nepal [ 2 ] . The findings can later be leveraged to develop an autonomous assessment system to assist humans in evaluating students in a more efficient and scientific manner(Fig. 1 ). Following are the contributions of the paper: 1. A competency-aligned Grade 10 Math assessment framework covering topics such as Matrix, Coordinate Geometry, Trigonometry, and Functions 2. An assessment of cross-competencies that fit overall mathematics levels and topics globally: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation 3. A human–LLM comparative evaluation protocol involving two senior mathematics faculty members engaged in competency, rubric, and question design, alongside the competency mapping of students 4. An empirical analysis across various model scales and architectures. 1. Rubric Construction Expert-designed competency framework 2. Assessment Design 16 open-ended math questions 3. Student Responses 33 Grade 10 handwritten scripts 4. Expert Annotation Double-blind grading \kappa_{w}=0.865 5. LLM Evaluation Rubric-based competency prediction Human Ground Truth Construction Figure 1: Overview of the benchmark construction and evaluation pipeline. Human experts design the rubric, create the assessment, and produce the ground-truth annotations before LLMs are evaluated on the same rubric-based competency prediction task. The rest of this paper is organized as follows: Section 2 discusses Related Works; Section 3 describes the Experiment Design; Section 4 explains the Methodology; Section 5 provides Results and Discussions; Section 6 presents the Conclusions; and finally, Section 7 provides Limitations and Future Recommendations 2 Related Works The integration of Artificial Intelligence (AI) into Competency-Based Education (CBE) has transitioned from theoretical exploration to practical implementation. Recent literature (2023–2025) highlights significant advances in Large Language Models (LLMs) for grading and rubric generation. 2.1 Global Trends and the Shift to Orchestration Research shows an increase in the use of AI for competency evaluation. Competency-Based Education (CBE) has been widely adopted as a paradigm in which evaluation is heavily dependent on various sources of evidence, such as project portfolios, rubrics, and mentor evaluations, rather than traditional tests [ 6 ] . A bibliometric study by [ 12 ] shows that scientific output in this area has increased at an average annual rate of 8.43%, due to the need to scale personalized education. Likewise, in medical education, [ 3 ] found a peak in AI-related publications in 2024, in which there was a shift from basic knowledge validation to the evaluation of sophisticated clinical reasoning skills via Generative AI. Current models suggest that AI should not only grade but also direct the learning process. [ 8 ] presented the concept of the “Adaptive-Autonomy Curve,” suggesting that AI systems should automatically decrease scaffolding and increase autonomy as competency is verified. At the same time, [ 10 ] presented the AICoLED framework, which employed an Input-Process-Output model to translate digital competencies. The framework used for this study required additional validation because it depended on small expert consensus testing instead of large student evaluation tests. Existing studies mainly investigate higher education and professional fields with limited empirical research on school-level mathematics assessment. 2.2 Advancements in Automated Assessment Tools Technological innovation has directed its efforts toward developing automation solutions for assessment tasks that require extensive manual labor. [ 13 ] proved that Large Language Models (LLMs) could attain a 99.03% grading accuracy rate compared to human teachers using a structured prompt approach. However, this high accuracy may not translate to secondary education evaluation processes that involve open-ended, complex responses. The SmartRubrics system which [ 7 ] described introduced methods that use learning outcomes from syllabus documents to produce assessment matrices. 2.3 Identified Gaps: The ”Black Box" and Validation Crisis Despite these advancements, the reviewed literature identifies three critical limitations hindering the trusted deployment of AI in professional certification: 1. The ”Black Box" Problem: [ 14 ] and [ 4 ] point out that the evaluation of AI systems is often a “black box" problem, in which the underlying logic for a grade is not transparent. This makes it difficult to trust the system, as it may be biased. [ 5 ] further points out that teacher adoption is statistically linked to their confidence in AI ethics; without transparent logic, adoption will be low. 2. Technical Limitations in Context: Although LLMs are highly effective, they are not suited for large contexts. [ 13