MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors

Baraa Hikal, Mohamed Basem, Islam Oshallah, Ali Hamdi · 2025

Read MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors by Baraa Hikal, Mohamed Basem, Islam Oshallah, Ali Hamdi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18549.txt.

Introduction

Judging whether an AI tutor's reply actually helps a student takes more than one score. Baraa Hikal and three co-authors enter a 2025 shared task on grading tutor responses along four lines: spotting the student's mistake, locating it, giving guidance, and suggesting a doable next step. Their system fine-tunes one language model for all four tracks through a single training pipeline.

At judging time, they add a twist: an ensemble step that listens for disagreement among model votes, which improves coverage of rare labels. The system placed first in Providing Guidance, third in Actionability, and fourth in both mistake tracks. The authors present the work as evidence that disagreement-aware tuning makes educational judging sturdier.

Catalog record

CREATORBaraa Hikal, Mohamed Basem, Islam Oshallah, Ali Hamdi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18549

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors Baraa Hikal, Mohamed Basem, Islam Oshallah, Ali Hamdi Abstract We present MSA-MathEval , our submission to the BEA 2025 Shared Task on evaluating AI tutor responses across four instructional dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. Our approach uses a unified training pipeline to fine-tune a single instruction-tuned language model across all tracks, without any task-specific architecture modifications. To improve prediction reliability, we introduce a disagreement-aware ensemble inference strategy that enhances coverage of minority labels. Our system achieves strong performance across all tracks, ranking 1 st in Providing Guidance, 3 rd in Actionability, and 4 th in both Mistake Identification and Mistake Location. These results demonstrate the effectiveness of scalable instruction tuning and disagreement-driven modeling for robust, multi-dimensional evaluation of LLMs as educational tutors. 1 Introduction Large language models (LLMs) are increasingly used in educational applications, acting as AI tutors that engage students in natural language. However, effective tutoring goes beyond producing correct answers. AI tutors must recognize student mistakes, explain misconceptions, provide constructive guidance, and suggest actionable next steps. Evaluating such complex pedagogical behavior remains a key challenge—especially at scale. Prior work in intelligent tutoring systems (ITS) emphasized these goals long before the advent of LLMs. For example, AutoTutor used natural language processing (NLP) and dialogue-based feedback to improve learning outcomes across domains ( Nye et al. 2014 ) . Later, metrics such as conversational uptake were proposed to capture tutor responsiveness and its link to instructional quality ( Demszky et al. 2021 ) . With the rise of instruction-tuned LLMs, evaluation frameworks have emerged to assess their pedagogical abilities. Tack and Piech ( Tack and Piech 2022 ) proposed the AI Teacher Test to evaluate whether model responses demonstrate student understanding and helpfulness. Subsequent work introduced fine-grained rubrics, including coherence, correctness, targetedness, and actionability ( Macina et al. 2023 ; Daheim et al. 2024 ; Wang et al. 2024 ) . Building on these efforts, the BEA 2025 Shared Task introduces MRBench , a pedagogically motivated benchmark for evaluating AI-generated tutor responses in math dialogues ( Kochmar et al. 2025 ) . While BEA 2023 emphasized response generation, BEA 2025 shifts toward assessing feedback quality across four instructional dimensions derived from educational science. In this work, we present MSA-MathEval , a unified system that addresses all four tracks using a single fine-tuned model and consistent training pipeline. We fine-tune the open-weight Mathstral-7B-v0.1 —an instruction-tuned LLM specialized for mathematical reasoning—using parameter-efficient LoRA adapters. To improve prediction reliability, we incorporate ensemble-based inference that combines model disagreement and uncertainty estimation. Figure 1: Overview of our unified MSA-MathEval framework for the BEA 2025 Shared Task. The pipeline includes preprocessing, LoRA-based fine-tuning of Mathstral-7B-v0.1, and disagreement-aware ensemble inference. Our contributions are as follows: • We design a unified training pipeline for all four BEA 2025 tracks, using LoRA-based fine-tuning of Mathstral-7B-v0.1 with no track-specific architecture changes. • We propose an ensemble-based inference strategy leveraging model disagreement and uncertainty for robust prediction. • We achieve top-tier performance across all tracks, including first place in Providing Guidance. 2 Related Work Evaluating the pedagogical capabilities of AI tutors builds upon long-standing research in intelligent tutoring systems (ITS) and more recent advances in large language models (LLMs). Early ITS such as AutoTutor emphasized the importance of natural language dialogue in promoting student learning through error remediation and scaffolding ( Nye et al. 2014 ) . These systems often relied on rule-based or statistical NLP methods to assess learner inputs and generate appropriate tutor responses. The emergence of instruction-tuned LLMs has prompted a shift toward more scalable methods for modeling tutoring behavior. Tack and Piech ( Tack and Piech 2022 ) proposed the AI Teacher Test to benchmark LLM outputs on criteria such as helpfulness and pedagogical appropriateness. Macina et al. ( Macina et al. 2023 ) and Daheim et al. ( Daheim et al. 2024 ) introduced fine-grained rubrics for LLM tutoring quality in mathematical dialogue, including dimensions such as targetedness, coherence, and actionability. In terms of modeling strategies, prior systems have explored both classification and ranking approaches for feedback generation. Daheim et al. used multi-aspect annotation schemes to evaluate feedback informativeness, while Wang et al. ( Wang et al. 2024 ) proposed a bridging rubric for LLM feedback grounded in human tutor behavior. These studies highlight the need for systems that go beyond correctness to capture richer instructional attributes. Compared to these approaches, our work introduces a unified training and inference framework across multiple feedback dimensions, leveraging ensemble disagreement and uncertainty estimation for prediction stability. Unlike previous models with track-specific adaptations or rule-based post-processing, we apply a consistent architecture based on the Mathstral-7B-v0.1 model across all tasks. This allows us to assess the generalizability of instruction-tuned mathematical LLMs in a multi-dimensional educational evaluation setting. 3 Method Our approach, MSA-MathEval , applies a unified framework across all four tracks in the BEA 2025 Shared Task. We build on the instruction-tuned Mathstral-7B-v0.1 model and leverage parameter-efficient fine-tuning (LoRA) along with ensemble-based inference to enhance prediction robustness. The methodology consists of the following stages: dataset preprocessing, model selection, fine-tuning strategy, and ensemble-based inference (see Figure 1 ). 3.1 Preprocessing The original dataset consists of nested JSON files, where each dialogue contains multiple tutor responses annotated across four pedagogical dimensions. To facilitate instruction-based fine-tuning, we transformed the data into four track-specific JSONL files. Each file includes a flattened dialogue, a natural language evaluation prompt, and a categorical label from three possible options: Yes , To some extent , or No . Each training instance was structured as a two-turn dialogue following the chat schema used by instruction-tuned language models. Specifically: • user : This field contains a complete, track-specific prompt with explicit evaluation criteria, followed by the dialogue context and tutor response to be evaluated. • assistant : This field contains the gold label corresponding to the tutor response—one of "Yes" , "To some extent" , or "No" —as annotated in the development set. The system role was omitted to reduce token overhead and focus the model on the input–output mapping relevant to each multi-class classification task. Track 1 – Mistake Identification TASK DEFINITION: You are an expert evaluator of AI tutor responses. Your task is to determine whether the tutor’s response accurately identifies a mistake in the student’s reasoning or solution. EVALUATION CRITERIA: 1."Yes"– The tutor accurately identifies a mistake in the student’s response. 2."To some extent"– The tutor shows some awareness, but it is ambiguous or uncertain. 3."No"– The tutor fails to identify or misunderstands the mistake. Track 2 – Mistake Location TASK DEFINITION: You are an expert evaluator of AI tutor responses. Your task is to determine whether the tutor’s response accurately points to a genuine mistake and its location in the student’s response. EVALUATION CRITERIA: 1."Yes"– The tutor clearly points to the exact location of the mistake. 2."To some extent"– The tutor refers to a mistake but is vague or indirect. 3."No"– The tutor provides no indication of where the mistake occurred. Track 3 – Providing Guidance TASK DEFINITION: You are an expert evaluator of AI tutor responses. Your task is to determine whether the tutor’s response

Read the complete text