Detection of Disengagement from Voluntary Quizzes: An Explainable Machine Learning Approach in Higher Distance Education

Behnam Parsaeifard, Christof Imhof, Tansu Pancar, Ioan-Sorin Comsa, Martin Hlosta · 2025

Read Detection of Disengagement from Voluntary Quizzes: An Explainable Machine Learning Approach in Higher Distance Education by Behnam Parsaeifard, Christof Imhof, Tansu Pancar, Ioan-Sorin Comsa, Martin Hlosta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02681.txt.

Introduction

Behnam Parsaeifard and co-authors build a tool that spots distance students drifting away from optional quizzes, in a 2025 paper. Quitting small tasks can be an early sign of dropping out for good. The team drew on quiz logs from 42 courses over four terms at a distance university. They picked the log fields that best flagged fading effort.

Eight learning methods were trained and compared on the logs. The best reached a balanced accuracy of about 91 percent and caught most drifting students. A method called SHAP then showed which signals drove each flag, so staff can see why a student was marked. The authors close with ideas for quick, well-timed nudges that could pull students back before they quit.

Catalog record

CREATORBehnam Parsaeifard, Christof Imhof, Tansu Pancar, Ioan-Sorin Comsa, Martin Hlosta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02681

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detection of Disengagement from Voluntary Quizzes: An Explainable Machine Learning Approach in Higher Distance Education Thanks: B. Parsaeifard, C. Imhof, T. Pancar, I.-S. Comsa, M. Hlosta, and P. Bergamin are with the Institute for Research in Open-, Distance- and eLearning, Swiss Distance University of Applied Sciences, Brig, CH-3900, Switzerland (e-mail addresses: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]). N. Bergamin (e-mail address: [email protected]) is with Department of Informatics, Swiss Distance University of Applied Sciences, Brig, CH-3900, Switzerland. P. Bergamin is also with the North-West University, Potchefstroom, 2531, South Africa. Behnam Parsaeifard, Christof Imhof, Tansu Pancar, Ioan-Sorin Comsa, Martin Hlosta Abstract Students disengaging from their tasks can have serious long-term consequences, including academic drop-out. This is particularly relevant for students in distance education. One way to measure the level of disengagement in distance education is to observe participation in non-mandatory exercises in different online courses. In this paper, we detect student disengagement in the non-mandatory quizzes of 42 courses in four semesters from a distance-based university. We carefully identified the most informative student log data that could be extracted and processed from Moodle. Then, eight machine learning algorithms were trained and compared to obtain the highest possible prediction accuracy. Using the SHAP method, we developed an explainable machine learning framework that allows practitioners to better understand the decisions of the trained algorithm. The experimental results show a balanced accuracy of 91%, where about 85% of disengaged students were correctly detected. On top of the highly predictive performance and explainable framework, we provide a discussion on how to design a timely intervention to minimise disengagement from voluntary tasks in online learning. I Introduction The advent of distance education has made learning more flexible than ever before. Instead of having to attend classes and solve tasks at specific time, students are granted more freedom in choosing when to engage with their academic workload. This flexibility attracts many non-traditional student groups to higher education, including students that are employed outside of their studies, either fully or part-time. While deadlines are still set in place, students are responsible themselves for planning and time management, especially as far as non-mandatory tasks and exercises are concerned. This freedom can also lead to satisficing behaviour, meaning students only do the bare minimum to pass their courses (see e.g., [ 1 , 2 ] ). It can be assumed that such minimalist strategies are quite prevalent due to students being engaged in duties outside their studies (for example work, family and leisure) [ 3 ] . The COVID-19 pandemic is thought to have fostered this kind of behaviour even more [ 4 ] . Non-completion of voluntary tasks, such as optional quizzes, is a form of behavioural disengagement strongly linked to academic drop-out or attrition [ 5 , 6 , 7 , 8 ] . Voluntary quizzes are critical predictors of later performance (e.g., summative examinations) and play a significant role in academic success [ 9 , 10 , 11 ] . Drop-out rates in distance education are notably higher compared to traditional classrooms, particularly among fully-employed students [ 12 , 13 ] . Early detection of disengagement enables timely interventions and minimise negative impacts. An automated detection system can process students’ interactions with the Learning Management System (LMS) in real-time to predict disengagement behaviours based on log data accumulated throughout the semester. For effective interventions, it is crucial to identify the specific factors contributing to disengagement, such as low participation, inconsistent activities, or irregular access to course materials. These insights enable tailored support, helping students and teachers understand the causes and take corrective actions. This paper proposes a framework that utilises Machine Learning (ML) to detect student disengagement and employs an explainable approach to interpret the model’s decisions, enabling the design of targeted interventions. Our primary focus is on quiz non-completion as an indicator of disengagement, particularly in voluntary tasks, as completing these tasks is closely linked to improved academic performance. The study sample includes adaptive courses where quiz difficulty is tailored to student performance, chosen for their consistent number of quizzes compared to other courses. To address these objectives, we formulate the following research questions: • RQ1: What data can be used to detect disengagement from voluntary tasks in online courses? • RQ2: Which ML algorithms provide the highest predictive performance in detecting disengagement? • RQ3: How might the explainability of ML decisions be interpreted in the context of potential interventions? • RQ4: How can intervention strategies be aligned with detected behavioral patterns to inform targeted and timely support actions? To answer these research questions, the following contributions are proposed in this paper: • Data from four academic semesters at a distance university of applied sciences, encompassing 42 online courses, was collected. In these courses, adaptive tasks with varying difficulty levels were recommended to students based on their performance. ( RQ1 ) • Eight ML models are implemented, configured, trained, and evaluated in a comparative analysis, using various performance metrics. The model with the highest predictive accuracy is selected for further investigation to enhance the explainability and interpretation of its predictions. ( RQ2 ) • SHapley Additive exPlanations (SHAP) were used to analyze why certain students are at risk of disengaging from voluntary tasks. This analysis highlights key predictors, grouped into three behavioural patterns: erratic, delayed, and irregular. Disengagement predictions are further categorised into low (L), medium (M), and high (H) risk levels, enabling teachers to initiate personalised, dialogue-based interventions tailored to each risk category. ( RQ3 ) • Based on a structured literature review, intervention strategies were identified and mapped to the behavioural patterns derived from SHAP explainability. For each risk level (L, M, H) specific, evidence-based interventions are proposed. Additionally, the study outlines recommended timing for intervention deployment, grounded in the model’s detection outputs and the interpretability of behavioral indicators. ( RQ4 ) The remainder of the paper is organised as follows: Section II presents the related work on student disengagement and prediction models, in distance education. Section III outlines the methods and tools used for data collection and processing, along with a description of the proposed explainable ML-based framework. Section IV presents the findings and addresses the research questions. Section V offers practical insights for teachers to initiate tailored, personalised dialogue-based interventions. Finally, the paper concludes with Section VI . II Related Work II-A Background on Disengagement in Distance Education Engagement is a construct often encountered in education. It is described as “energy in action, the connection between person and activity” (see [ 14 ] , p.1) and reflects a person’s active involvement in a task or activity [ 15 ] . Engagement has been linked to academic performance, persistence, satisfaction, and retention, and is acknowledged to play a significant role in learning, with some researchers going as far as to call it “the holy grail of learning” (see [ 16 ] , p.1). Engagement tends to be thought of as behavioural, emotional, and cognitive manifestations of motivation [ 17 , 18 ] and, in contrast to motivation, cannot be separated from its context since an individual is always engaged in something, e.g. a task, an activity or a relationship [ 19 ] . Currently, no definition of engagement has been universally accepted and there

Read the complete text