Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces
Jakob Schwerter 1,2, Loreen Sabel 2, Judith Bose 2, Matthew L. Bernacki 3, Di Xu 4, Marko Schmellenkamp 5, Thomas Zeume 5, Philipp Doebler 2 · 2026
Read Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces by Jakob Schwerter 1,2, Loreen Sabel 2, Judith Bose 2, Matthew L. Bernacki 3, Di Xu 4, Marko Schmellenkamp 5, Thomas Zeume 5, Philipp Doebler 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22812.txt.
Introduction
Schwerter and seven colleagues test whether prediction models built from digital learning traces travel across courses, using weekly self-regulated-learning indicators from three theory-heavy computer science courses at two universities. Early warning proves possible, with time management, effort control, and steady engagement standing out as predictors. Random Forest fits its own course best, Elastic Net travels better, and accuracy falls when the share of at-risk students differs between institutions.
Catalog record
| CREATOR | Jakob Schwerter 1,2, Loreen Sabel 2, Judith Bose 2, Matthew L. Bernacki 3, Di Xu 4, Marko Schmellenkamp 5, Thomas Zeume 5, Philipp Doebler 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96629 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22812
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces Jakob Schwerter 1,2, Loreen Sabel 2, Judith Bose 2, Matthew L. Bernacki 3, Di Xu 4, Marko Schmellenkamp 5, Thomas Zeume 5, Philipp Doebler 2 Abstract STEM dropout rates remain high at universities, particularly in computer science programs with theory-intensive courses. Digital learning environments now capture rich behavioral data that could help identify struggling students early, yet the generalizability of data-driven prediction models across courses and institutions remains uncertain. Guided by self-regulated learning (SRL) theory, this study analyzed multimodal digital-trace data from three undergraduate theoretical computer science courses ( N 1 = 137 N_{1}=137 , N 2 = 104 N_{2}=104 , N 3 = 148 N_{3}=148 ) at two universities. Weekly SRL-aligned digital-trace indicators were modeled using Elastic Net, Random Forest, and XGBoost to evaluate predictive performance over time and across settings, and model calibration both within and across courses. Early prediction of at-risk students was feasible, with SRL-related behaviors such as time management, effort regulation, and sustained engagement emerging as key predictors. While Random Forest achieved the highest in-sample accuracy, Elastic Net generalized more robustly across contexts. Out-of-sample accuracy and calibration declined between institutions with different base rates, underscoring the contextual nature of predictive analytics in higher education. These findings suggest that digital learning traces enable early identification of at-risk students within courses, but generalizing predictive models beyond their original context requires caution, particularly if the at-risk rates differ between contexts. Cross-Course Generalizability \authorsaffiliations 1 Hector Research Institute of Education Sciences and Psychology, University of Tübingen 2 TU Dortmund University 3 University of North-Carolina, Chapel Hill 4 University of California, Irvine 5 Ruhr University Bochum \authornote Correspondence concerning this manuscript should be directed to Jakob Schwerter, University of Tübingen, Hector Research Institute of Education Sciences and Psychology (HIB), Europastr. 6, 72072 Tübingen, Germany. Email: [email protected] \addORCIDlink Jakob Schwerter0000-0001-5818-2431 \addORCIDlink Loreen Sabel0000-0002-9832-8842 \addORCIDlink Matthew L. Bernacki0000-0003-1279-2829 \addORCIDlink Di Xu0000-0002-8160-4783 \addORCIDlink Marko Schmellenkamp0000-0003-3966-6590 \addORCIDlink Thomas Zeume0000-0002-5186-7507 \addORCIDlink Philipp Doebler0000-0002-2946-8526 Funding : This work was supported by “From Prediction to Agile Interventions in the Social Sciences (FAIR, PROFILNRW-2020-068)”, by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), grant 448468041 on “Formal Foundations for Teaching Support Systems in Theoretical Computer Science”, and by Intramural Funding of the “Trustworthy Data Science and Security” at TU Dortmund. The sole responsibility for the content of this publication lies with the authors. Declarations : All authors certify that they have no affiliations with or involvement in any organization or entity with any financial or non-financial interest in the subject matter or materials discussed in this manuscript. The authors have no financial or proprietary interests in any material discussed in this article. Declaration of generative AI and AI-assisted technologies in the writing process : During the preparation of this work, the authors used GPT-5 and DeepL for language-focused editing of human-written text. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication. Conflict of Interest : The authors have no conflicts of interest to declare. Ethical Approval : Approval was granted by the Ethics Committee of TU Dortmund University. keywords Self-Regulated Learning, Digital Trace Data, Learning Analytics, Out-of-sample Generalization, Predictive Modeling, Probability Calibration, STEM Education Higher education faces declining enrollment and growing dropout concerns, while also grappling with a student population that is becoming increasingly diverse in academic preparation and background ( 33 ; 55 ; 56 ) . These challenges are particularly pronounced in science, technology, engineering, and mathematics (STEM) disciplines, where maintaining persistence through degree completion has proven especially difficult. Across OECD countries, STEM completion rates average roughly 68 percent—well below those in health, welfare, or social sciences programs ( 57 ) . Programs in information and communication technology and engineering disciplines consistently exhibit the lowest completion rates, indicating that this pattern reflects a systemic rather than institution-specific issue. One commonly cited reason is that students in these fields encounter theory-intensive coursework early in their studies, and many students struggle to manage the substantial workload and sustain effective study habits ( 77 ) . When these difficulties accumulate ( 9 ) , withdrawal becomes more likely, which leads to important consequences for students, institutions, and society ( 70 ; 21 ) . These challenges have prompted instructors and institutions to look for ways to better support students’ day-to-day learning processes, particularly in the large and demanding courses where many students struggle. One promising approach stems from the widespread integration of digital learning environments into college instruction. These environments structure opportunities for regular practice, provide immediate feedback, and make students’ engagement patterns visible in ways that traditional settings do not ( 32 ) . A growing number of studies consistently show that active participation in digital learning platforms is associated with better performance and persistence rates across a range of higher education settings, including, for example business-administration courses ( 72 ; 26 ) , social-science statistics courses ( 74 ) , psychology statistics courses ( 35 ; 36 ) , and undergraduate science courses ( 52 ) , but evidence remains sparse for computer science—despite high dropout rates and distinct cognitive demands in theory-driven coursework. These findings suggest that digital trace data capture meaningful aspects of how students navigate complex learning tasks, enabling early-warning models that flag risk weeks before exam outcomes are available ( 6 ; 80 ; 84 ) . Self-regulated learning (SRL; see 20 ; 30 ) offers a robust theoretical framework for interpreting these digital traces and explaining why early learning behaviors predict later achievement ( 6 ) . SRL describes learners’ ability to direct their cognition, motivation, and behavior toward achieving academic goals. As students transition into college-level STEM coursework, these regulatory processes become particularly consequential, because students must manage demanding tasks with less external structure than they experienced in secondary school ( 21 ; 87 ) . Predictive learning analytics translates these SRL-aligned digital traces into early-warning models intended to identify students who may be at risk of failure ( 75 ) . A growing body of research demonstrates that exam performance can often be predicted within the first weeks of a semester ( 1 ; 7 ; 12 ) . When accurate, these early predictions create opportunities for instructors to reach out to struggling students, offer timely feedback, or connect them with additional resources before disengagement becomes more difficult to address ( 12 ) . As a result, predictive analytics and early-warning systems have been adopted widely across higher education. In the United States, roughly one-third of higher-education institutions—approximately 1,400 colleges and universities—now use early-warning or predictive analytics systems, representing a market exceeding $500 million ( 2 ) . In Germany the state of North Rhine-Westphalia has committed nearly €4 million (2024–2026) to support the development of AI-based learning analytics initiatives ( 53 ) . Despite this momentum, however, important questions remain about the conditions under which prediction models provide reliable information and the extent to which they can be used beyond the specific courses or institutions in which they were developed. On the one hand, the rapid adoption of early-warning systems across institutions underscores the importance of generalizability, since colleges often hope to reuse models across semesters or