Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs Conference: The 40th ACM/SIGAPP Symposium on Applied Computing; March 31-April 4, 2025; Catania, Italy The 40th ACM/SIGAPP Symposium on Applied Computing (SAC ’25), March 31-April 4, 2025, Catania, Italy DOI: 10.1145/3672608.3

Bianca Raimondi, Saverio Giallorenzo, Maurizio Gabbrielli · 2025

Read Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs Conference: The 40th ACM/SIGAPP Symposium on Applied Computing; March 31-April 4, 2025; Catania, Italy The 40th ACM/SIGAPP Symposium on Applied Computing (SAC ’25), March 31-April 4, 2025, Catania, Italy DOI: 10.1145/3672608.3 by Bianca Raimondi, Saverio Giallorenzo, Maurizio Gabbrielli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05891.txt.

Introduction

Raimondi, Giallorenzo, and Gabbrielli ask in twenty twenty-five if small language models can answer course questions well. They test models on quiz items from a class on programming languages. The set has one hundred sixty-two multiple choice questions, and the team shares it in public.

They try three sizes of the LLaMA-2 model, from small to large. They tune the small models on parts of the course textbook. They also shrink the models to use less memory and power. The tuned small models beat the large generic ones on the quiz. The result suggests teachers can get good answers with cheap tools and their own course text.

Catalog record

CREATORBianca Raimondi, Saverio Giallorenzo, Maurizio Gabbrielli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-05891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs Conference: The 40th ACM/SIGAPP Symposium on Applied Computing; March 31-April 4, 2025; Catania, Italy The 40th ACM/SIGAPP Symposium on Applied Computing (SAC ’25), March 31-April 4, 2025, Catania, Italy DOI: 10.1145/3672608.3707732 ISBN: 979-8-4007-0629-5/25/03 CCS: Applied computing Interactive learning environments CCS: Computing methodologies Natural language generation Bianca Raimondi, Saverio Giallorenzo, Maurizio Gabbrielli Abstract . In education, the capability of generating human-like text of Large Language Models (LLMs) inspired work on how they can increase the efficiency of learning and teaching. We study the affordability of these models for educators and students by investigating how LLMs answer multiple-choice questions (MCQs) with respect to hardware constraints and refinement techniques. We explore this space by using generic pre-trained LLMs (the 7B, 13B, and 70B variants of LLaMA-2) to answer 162 undergraduate-level MCQs from a course on Programming Languages (PL)—the MCQ dataset is a contribution of this work, which we make publicly available. Specifically, we dissect how different factors, such as using readily-available material—(parts of) the course’s textbook—for fine-tuning and quantisation (to decrease resource usage) can change the accuracy of the responses. The main takeaway is that smaller textbook-based fine-tuned models outperform generic larger ones (whose pre-training requires conspicuous resources), making the usage of LLMs for answering MCQs resource- and material-wise affordable. † † cc-license: CC-BY 1. Introduction Large Language Models (LLMs) ( Zhao et al. 2023 ) have recently attracted attention for their capability to handle complex tasks ( Zhang et al. 2023a ) . In education, LLMs can help students and teachers reduce the effort involved in learning and teaching ( Dijkstra et al. 2022 ; Rasul et al. 2023 ; Extance 2023 ) . In this paper, we focus on studying how LLMs answer multiple-choice questions (MCQs) regarding a specific academic topic (bachelor-level programming languages) with respect to hardware constraints and refinement techniques. 1.1. Related Work Recent evidence shows how ascertaining MCQs is challenging for LLMs and requires ad-hoc prompting to obtain good results ( Robinson et al. 2023 ) . Here, we look at work close to ours, focussed on the domain of question-answering for education. Nori et al. ( Nori et al. 2023 ) evaluate GPT-4 on medical competency examinations including both QA and MCQs and Silva et al. ( Silva et al. 2023 ) conduct a study on MCQs in the field of agriculture. In both works, the authors show that using specific prompt engineering techniques increases the number of correct answers. Other works aim to test LLMs such as LLaMA and GPT to answer MCQs. In particular, some works ( Tran et al. 2023 ; Savelka et al. 2023 ; Newton and Xiromeriti 2023 ) show that GPT performs poorly in correctly answering Computer Science MCQs. These results align with the accuracy profiles we see in ours. Several other investigations ( Pezeshkpour and Hruschka 2023 ; Zheng et al. 2023 ; Zhang et al. 2023b ) have assessed the ability of LLMs like LLaMA and GPT to handle MCQs. In particular, Zhang et al. 2023b show a higher proficiency of LLMs in STEM than social sciences and humanities, highlighting the importance of optimising their knowledge structures. Unfortunately, Zhang et al. 2023b conducted their experiments on three closed-source LLMs, which did not allow them to refine their models and further study their optimisation. In order to improve the performance of these models, Bucher and Martini ( Juan José Bucher and Martini 2024 ) tried to refine some open source models in the field of text classification. The authors showed how this type of refinement, carried out on smaller models, can be used to obtain a resource with a higher precision than the larger ones. In our contribution, we provide further insight on and address some limitations of the cited work by 1) experimenting with open-source LLMs, also comparing different versions thereof (in terms of the number of parameters) and 2) investigating how one can refine the knowledge structure of LLMs, via fine-tuning. 1.2. Proposed Research Given the recent, spreading success of LLMs in many assistive tasks, researchers have been exploring their potential in education. Specifically, LLMs offer opportunities such as helping students generate ideas, summarise literature, and create personalised learning experiences. Moreover, they can assist lecturers by designing quizzes, providing feedback on materials and exams, and generating educational content tailored to student needs. However, this new technology comes with challenges, such as the acceptance of AI feedback and the need for lecturers to integrate these tools into their teaching. For instance, to address concerns about accuracy and trust, researchers advocate ( Extance 2023 ) that universities coordinate AI-generated and human feedback to ensure reliable information. We appraise our work in this context, aiming to find ways to improve the accuracy of the responses of such models, so that they could be used in real educational environments, thereby reducing the problem of hallucinations. Specifically, we investigate the affordability of using LLMs to answer MCQs on a common item of undergraduate Computer Science curricula: Programming Languages (PL). We study the accuracy of a publicly available, generic model, LLaMA-2, in responding 162 MCQs on different subjects of a PL course—we make the dataset of questions publicly available. In the case of PL, LLMs need to manipulate linguistic structures that, while different from natural language, are analogous to the latter and should theoretically be within their capabilities. However, the domain of PL has technical specificities, similar to other STEM disciplines, requiring dedicated reasoning skills that the current LLMs may lack. Hence, broadening our view, we see this work contributing to the general body of knowledge focused on improving task-specific LLM accuracy. To build our study, the first question we ask is to what extent one can use the considered, generic models ‘as-is’ to accomplish the task of ascertaining our MCQ questionnaire. In answering this question, we dissect how different factors, such as quantisation (to decrease resource usage), can change the accuracy of the responses. Following these results, we investigate the affordability of increasing the accuracy of the model. Given the work of Bucher and Martini ( Juan José Bucher and Martini 2024 ) , we wonder whether fine-tuning could not only improve performance in tasks such as classification (where models can be tested using classic metrics such as accuracy, precision and recall), but also be able to acquire more specific knowledge (or at least refine their handling of the lexicon) in a given subject area. This aspect can be particularly helpful in the field of Education, where the use of smaller models, and thus reduced resources, can encourage schools to adopt such technology. Namely, we refine the model by fine-tuning it with material readily available to the educator—the textbook used to teach the course—and report on the hardware requirements and accuracy increases. To further shed light on how the different parameters of the fine-tuning routine and how the material from the textbook influences the answers of the refined model, we conduct a correlation study that clarifies the relationship between these variables and the increases in accuracy. The LLaMA-2 model comes in four sizes—defined by their number of parameters; empirically, ceteris paribus , the greater the number of parameters afforded by a model the more

Read the complete text