Accurate Predictions in Education with Discrete Variational Inference
Tom Quilter, Anastasija Ilick, Karen Poon, Richard Turner · 2025
Read Accurate Predictions in Education with Discrete Variational Inference by Tom Quilter, Anastasija Ilick, Karen Poon, Richard Turner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23484.txt.
Introduction
Guessing whether a student will answer an exam question right is a core skill for any AI tutor. Tom Quilter and 3 co-authors work on formal mathematics exams, where good training data have been scarce. They release the largest open set yet of marked formal math exam responses. Their model builds on item response theory, a classic way to link student skill and question difficulty.
Their best tool is a new discrete variational inference method, built for settings with little data. It beat classic item response and matrix methods, and pushed accuracy past eighty percent. One surprise: a single skill value per student was enough to reach top accuracy, even when topic-level skill profiles were added.
Catalog record
| CREATOR | Tom Quilter, Anastasija Ilick, Karen Poon, Richard Turner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23484
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accurate Predictions in Education with Discrete Variational Inference Tom Quilter, Anastasija Ilick, Karen Poon, Richard Turner Abstract One of the largest drivers of social inequality is unequal access to personal tutoring, with wealthier individuals able to afford it, while the majority cannot. Affordable, effective AI tutors offer a scalable solution. We focus on adaptive learning, predicting whether a student will answer a question correctly, a key component of any effective tutoring system. Yet many platforms struggle to achieve high prediction accuracy, especially in data-sparse settings. To address this, we release the largest open dataset of professionally marked formal mathematics exam responses to date. We introduce a probabilistic modelling framework rooted in Item Response Theory (IRT) that achieves over 80 percent accuracy, setting a new benchmark for mathematics prediction accuracy of formal exam papers. Extending this, our collaborative filtering models incorporate topic-level skill profiles, but reveal a surprising and educationally significant finding, a single latent ability parameter alone is needed to achieve the maximum predictive accuracy. Our main contribution though is deriving and implementing a novel discrete variational inference framework, achieving our highest prediction accuracy in low-data settings and outperforming all classical IRT and matrix factorisation baselines. Introduction Everyone remembers an amazing teacher from their childhood. Such individuals possess a passion for their subject and the ability to convey complex ideas. However, one of the most important and sometimes overlooked skills that incredible teachers have is to be able to present the student’s work at exactly the right level. This principle is echoed in Duolingo’s design philosophy, where the platform aims for learners to answer around 80 percent of questions correctly. This level of difficulty is challenging enough to sustain engagement, but not so difficult as to become discouraging, occupying the so-called Goldilocks zone of learning ( ( Bjork 1994 ) , ( Csikszentmihalyi 1990 ) ). AI personal tutors hold the promise of helping to redress educational inequality, an inequality driven in large measure by unequal access to high quality personal tutoring. Although wealthier families can afford such individualised support, the majority of students cannot, exacerbating long-standing social divides. For AI tutoring platforms to succeed, their effectiveness will in part depend on whether they can deliver questions at just the right level to keep students engaged and stretched, just as the very best human tutors do. Our aim in this paper is simply to predict whether a student will get an unseen question correct, given their previous question answers, other students’ responses, and meta-data. We hereby release the largest open benchmark of marked mathematics exam paper responses to date, and create models to predict a student’s question success. Moreover, educational platforms regularly have the problem of limited data, limited by the number of students for new platforms and for all platforms limited by the number of questions brand new students have answered. We produce novel methods to predict accurately in these limited data scenarios. A natural place to start is a model incorporating the ability of the student and the difficulty of a question, a two-parameter model. We achieve remarkable accuracy with this simple ability-difficulty model. So much so that when we introduce a more complex interaction model allowing for students’ individual strengths and weaknesses in certain topics of mathematics, such as algebra, data and geometry, we are unable to beat it. This suggests a potentially important result for education: that a single latent ability parameter may be the main factor driving a student’s success on exam questions, and perhaps the only one needed. In our search for factors that improve prediction accuracy, we find some benefit in modelling shared topic-level strengths and weaknesses across students in the same high school class, often taught by a single teacher. We also uncover interesting interpretability results, showing that the model implicitly discovers meaningful question types such as algebra or geometry as a byproduct of its optimisation for predictive accuracy. A major challenge for new educational platforms is the cold-start problem, having too few students on the system to make accurate predictions. It is in this low data regime that we make our most significant breakthrough. We derive a novel discrete variational inference framework, placing Gaussian priors over latent ability variables, which outperforms all our other models when student data is limited. For completeness, we also address the challenge of predicting performance for new students who have only attempted a small number of questions. We apply pool-based active learning to achieve significantly higher predictive accuracy than random selection and we also identify which questions are most informative to present, guiding personalised assessment. The main contributions of our work include: • When data are scarce our novel Variational Inference for discrete data yields improved accuracy Placing uncertainty over all latent parameters delivers our most important results. • Open-sourcing a New Large-Scale High Quality Educational Dataset We release the largest open dataset of professionally marked formal mathematics exam responses to date, providing a valuable benchmark. • Ability is all you need: Remarkably we discover that a single parameter of student ability cannot be beaten by more complex models incorporating individual topic strengths and weaknesses. Related Work Item Response Theory (IRT) in Education. IRT offers a probabilistic framework for modeling the probability that a learner answers an item correctly as a function of latent traits. The simplest variant—the Rasch 1PL model—uses a single student‐ability and item‐difficulty parameter with a logistic link ( Rasch 1960 ) . Classic extensions add item discrimination and guessing parameters ( Birnbaum 1968 ) . Foundational psychometric texts firmly established IRT for large‑scale assessment ( Hambleton, Swaminathan, and Rogers 1991 ) . Early educational technology work showed its practical value: Chen and Huang 2005 used IRT to adapt e‑learning content difficulty and demonstrated improved engagement. Modern computerised adaptive testing likewise selects questions in real time based on IRT estimates to maximise information gain. Collaborative Filtering and Matrix Factorisation. Viewing students as “users” and questions as “items” yields a natural analogy to recommender systems. Matrix‑factorization (MF) methods learn low‑rank embeddings for students and items and, with a logistic link, are algebraically similar to Rasch. MF has repeatedly outperformed feature‑engineered approaches on benchmarks such as the KDD Cup 2010; e.g. Thai-Nghe et al. 2011 reported lower prediction error than per‑skill models. MF also generalises to grade prediction across courses ( Polyzou and Karypis 2017 ) . Its flexibility allows side features or temporal components via factorization machines or tensor factorization. Notably, Vie, Kashima, and Yamada 2019 introduced Knowledge Tracing Machines , unifying IRT, logistic regression and performance‑factor models in a factorization‑machine framework that achieved state‑of‑the‑art accuracy while remaining interpretable. Knowledge Tracing and Sequential Models. Standard IRT and MF treat ability as static, whereas knowledge tracing (KT) models its temporal evolution. Bayesian Knowledge Tracing ( Corbett and Anderson 1994 ) and performance‑factor models are long‑standing baselines. Deep KT began with recurrent networks (DKT) ( Piech et al. 2015 ) and now includes memory networks ( Zhang et al. 2017 ) and attention mechanisms ( Ghosh and Heffernan 2020 ) . However, simpler latent‑factor methods can rival deep KT when tuned well; Vie, Kashima, and Yamada 2019 outperformed DKT on Duolingo data. The NeurIPS 2020 Education Challenge attracted hybrid solutions mixing sequential models with meta‑learning and