RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation?

Santiago Góngora † \dagger, Ignacio Sastre † \dagger, Santiago Robaina, Luis Chiruzzo, Aiala Rosá · 2025

Read RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation? by Santiago Góngora † \dagger, Ignacio Sastre † \dagger, Santiago Robaina, Luis Chiruzzo, Aiala Rosá (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11243.txt.

Introduction

Santiago Góngora and a team of co-authors test how far small language models can go in grading tutor responses, in two thousand twenty-five. Their team joined a shared task on evaluating artificial intelligence tutors. They limited themselves to models under one billion parameters, to mirror labs in the Global South where computing power is costly.

Across five tracks, their models stayed within a modest gap of the winning teams. The smallest gap was about six points and the largest about thirteen, on the organizers' strict score. All their models can run on a cheap graphics card or even without one. The authors conclude that small models are competitive for tutor evaluation under tight budgets.

Catalog record

CREATORSantiago Góngora † \dagger, Ignacio Sastre † \dagger, Santiago Robaina, Luis Chiruzzo, Aiala Rosá
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11243

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RETUYT-INCO at BEA 2025 Shared Task: How Far Can Lightweight Models Go in AI-powered Tutor Evaluation? Santiago Góngora † \dagger, Ignacio Sastre † \dagger, Santiago Robaina, Luis Chiruzzo, Aiala Rosá Abstract In this paper, we present the RETUYT-INCO participation at the BEA 2025 shared task. Our participation was characterized by the decision of using relatively small models, with fewer than 1B parameters. This self-imposed restriction tries to represent the conditions in which many research labs or institutions are in the Global South, where computational power is not easily accessible due to its prohibitive cost. Even under this restrictive self-imposed setting, our models managed to stay competitive with the rest of teams that participated in the shared task. According to the e ​ x ​ a ​ c ​ t ​ F 1 exact\ F_{1} scores published by the organizers, the performance gaps between our models and the winners were as follows: 6.46 6.46 in Track 1 ; 10.24 10.24 in Track 2 ; 7.85 7.85 in Track 3 ; 9.56 9.56 in Track 4 ; and 13.13 13.13 in Track 5 . Considering that the minimum difference with a winner team is 6.46 6.46 points — and the maximum difference is 13.13 13.13 — according to the e ​ x ​ a ​ c ​ t ​ F 1 exact\ F_{1} score, we find that models with a size smaller than 1B parameters are competitive for these tasks, all of which can be run on computers with a low-budget GPU or even without a GPU. 1 Introduction † † \dagger These (corresponding) authors contributed equally to this work: {sgongora,isastre}@fing.edu.uy The remarkable advances in the development of Large Language Models (LLMs) in recent years have turned Natural Language Processing into a discipline with great potential for application in different domains, and Education is not the exception ( Ignat et al. 2024 ) . However, these technological advances are not affordable to everyone. The cost of closed models — which are the most powerful and are typically considered the State-of-the-art in NLP — and the expensive infrastructure required to use large open models, coupled with negative effects on the environment, make research on other methods still essential. Our RETUYT-INCO team, as a research lab from South America, is no exception to this reality. Naturally, we are concerned about these issues and, consequently, we have focused on experimenting with open models in recent editions of the BEA shared tasks. For the 2023 shared task, consisting in generating teacher responses in educational dialogues ( Tack et al. 2023 ) , we participated using open models, obtaining competitive results ( Baladón et al. 2023 ) . One of the highlights of our participation was the “Hello” baseline , a simple strategy we followed which achieved remarkable results, unveiling the fragility of BERTScore ( Zhang et al. 2020 ) . More recently, for the 2024 BEA shared task, consisting in performing simplification experiments for different languages ( Shardlow et al. 2024 ) , we mainly focused on fine-tuning BERT and Mistral models (i.e., open models), even using synthetic data in some cases ( Sastre et al. 2024 ) . Figure 1: Class balance in our train and dev sets. The columns are coupled according to classes. In this paper, we present the RETUYT-INCO participation in the five tracks of the BEA 2025 Shared Task: Pedagogical Ability Assessment of AI-powered Tutors Kochmar et al. 2025 . This year, in addition to maintaining our restriction of working with open models, we challenged ourselves with an extra restriction: to experiment only with language models of fewer than a billion parameters and classical machine learning (ML) approaches. We will call these lightweight models, as they have to be small enough to run on a low-end GPU or with no GPU at all. This restriction is related to the situation many research labs face every day in the Global South: the lack of minimum resources to run what other regions consider small models (7B parameters or more). In our case, we have limited access to a national computing cluster, which we can use to fine-tune LLMs up to 7B parameters, but we do not have resources to host the fine-tuned LLMs and use them in real applications. Moreover, this is not the only motivation for this self-imposed restriction, as one of the research lines of our lab is the application of NLP tools to aid teachers in rural areas ( Chiruzzo et al. 2022 ; Rosá et al. 2025 ) . In such contexts it is very unlikely to use state-of-the-art LLMs, due to the impossibility of using them through APIs (since children’s privacy is key, sending private data to third-party servers is not an option), and the prohibitive cost of installing capable GPUs in trustworthy servers. Overall, throughout this paper, we will try to answer the general research question that motivated our participation: What is the performance gap between lightweight models and those state-of-the-art models, which would naturally have a better chance of winning the competition? . 2 Dataset For this edition, the dataset consists of 300 conversations Maurya et al. 2025 . Each dialogue is composed of interactions between a teacher and a math student. In the final turn of each dialogue the student shows clear confusion about a concept, and the dataset includes potential tutor responses intended to help the student. These responses — some of them generated by seven LLM-based tutors and others written by human tutors — are also evaluated by human evaluators (using Yes , No or To some extent ) according to four dimensions of interest that coincide with the four proposed tracks in the shared-task: mistake identification , mistake location , providing guidance and actionability . Due to the lack of a specific development set, during the first month we split the official dataset published by the organizers into two parts: 80% We decided to do this split focusing on the conversations — and not on the responses — trying to ensure that each conversation and all its responses stayed either in the train or the dev set. As a consequence, our train-dev split may not preserve the class balance of the original set. Figure 1 shows the class balance for each dimension in our train and dev partition. 3 Considered approaches For our experiments we considered classical ML classification algorithms, BERT-based approaches and fine-tuning a small autoregressive language model. Since all the tracks in the shared task are classification problems, many of the models we considered were used in more than one track. All of them were trained (or fine-tuned) using our train set, running on GoogleColab 1 1 1 https://colab.research.google.com/ or a national computational cluster (see Section 3.4.1 ). At the end of this section — in Subsection 3.5 — we will show the results we obtained when evaluating them on the dev set. 3.1 Preliminary experiments To gain a greater understanding of how challenging the tracks were, we performed four preliminary experiments with increasing degree of complexity. The first and most basic one consisted in answering always yes , what naturally degraded the F_{1}\ macro score, since the No and To some extent classes were never chosen. Then, we tried

Read the complete text