A Constraints-Based Approach to Fully Interpretable Neural Networks for Detecting Learner Behaviors
Juan D. Pinto Luc Paquette · 2025
Read A Constraints-Based Approach to Fully Interpretable Neural Networks for Detecting Learner Behaviors by Juan D. Pinto Luc Paquette (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20055.txt.
Introduction
Juan Pinto and Luc Paquette build a neural net in a 2025 paper that teachers can fully read and trust. Smart models now flag how learners act in class software, but their inner works are hard to explain. Post-hoc tales about a model can stray from what it truly learned.
Their design bakes limits into the net as it learns, so its parts keep clear sense. The facts pulled out for tales match the whole of what the model knows about the act in question. Tales built from them stay true and still read well to staff. Schools gain a flag tool whose calls they can defend to students and parents.
Catalog record
| CREATOR | Juan D. Pinto Luc Paquette |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Constraints-Based Approach to Fully Interpretable Neural Networks for Detecting Learner Behaviors Juan D. Pinto Luc Paquette Abstract The increasing use of complex machine learning models in education has led to concerns about their interpretability, which in turn has spurred interest in developing explainability techniques that are both faithful to the model’s inner workings and intelligible to human end-users. In this paper, we describe a novel approach to creating a neural-network-based behavior detection model that is interpretable by design. Our model is fully interpretable, meaning that the parameters we extract for our explanations have a clear interpretation, fully capture the model’s learned knowledge about the learner behavior of interest, and can be used to create explanations that are both faithful and intelligible. We achieve this by implementing a series of constraints to the model that both simplify its inference process and bring it closer to a human conception of the task at hand. We train the model to detect gaming-the-system behavior, evaluate its performance on this task, and compare its learned patterns to those identified by human experts. Our results show that the model is successfully able to learn patterns indicative of gaming-the-system behavior while providing evidence for fully interpretable explanations. We discuss the implications of our approach and suggest ways to evaluate explainability using a human-grounded approach. keywords Explainable AI, model transparency, interpretable neural networks 1 Introduction The field of education, as with many other areas of research, has continued to see an increasing use of complex machine learning models as these have become more powerful and versatile over the years. However, as these models have grown in complexity, they have also become more difficult to interpret, leading to concerns around fairness, accountability, and trust [ 8 ] , while also obscuring pedagogical insights that could improve learning outcomes among students. There has been significant interest in tackling these issues within the eXplainable AI (XAI) community, with a growing group of educational data mining (EDM) researchers focusing on the implications and possible solutions to the problems arising from highly complex models (as evidenced by workshops specializing on XAI in education [ 21 ] ). There is growing awareness of the inherent limitations of post-hoc explainability techniques (i.e. generating explanations, typically feature importance measures, based on analyses of a model’s inputs and outputs) which may often make them unsuitable for use in educational settings. In this paper, we describe a novel approach to training a neural-network-based behavior detection model (more specifically, a convolutional neural network) that is interpretable by design [ 28 ] . Our model is fully interpretable , by which we mean that the parameters we extract for our explanations (1) have a clear interpretation, (2) fully capture the model’s learned knowledge about the learner behavior of interest, and (3) can be used to create explanations that are both faithful and intelligible. We achieve this by implementing a series of constraints to the model that both simplify its inference process and bring it closer to a human conception of the task at hand. We specifically focus on the detection of gaming-the-system behavior, a type of learner disengagement with an educational task. We chose this due to the existence of previous expert-based features and models [ 17 ] that serve as a baseline of both accuracy and interpretability. We also chose to use a deep learning model due to their rising popularity among EDM researchers and the difficulty of faithfully interpreting their decision-making process [ 19 ] . Our research is guided by the following questions: 1. Can we train a convolutional neural network to detect gaming-the-system behavior with convolutional filters acting as explicit behavioral patterns? 2. Can we alter the model’s architecture so that the patterns are the only learnable parameters? 3. How can we create a differentiable perfect matching function that allows the model to definitively indicate a match or non-match based on the presence of any exact filter match to the input? 4. How do the patterns learned by our model compare to those identified by human experts? 5. How can we evaluate the interpretability of our model’s explanations? 2 Background 2.1 Intrinsically interpretable models The literature abounds with descriptions of the differences between models that are intrinsically interpretable and those that are opaque (often referred to as “black-box models”) [ 11 , 10 ] . Yet fundamentally, any model’s interpretability must be judged by the interpretability of the explanations derived from it [ 23 , 20 ] . These explanations can be derived from one of two sources: the model’s inner workings themselves (e.g. its parameters or gradients) or via post-hoc techniques that use simplified approximations based on the model’s inputs and outputs alone (such as LIME [ 22 ] or SHAP [ 12 ] ). With this understanding, [ 23 ] has proposed defining an explanation as the output of an interpretation function being performed on a piece of evidence. Evidence, in this context, refers to direct aspects of the model’s workings, which can be extracted from any combination of its parameters, gradients, inputs, and outputs. A piece of evidence in itself carries no semantic significance—this can only be added by an interpretation function. The interpretation function describes how the model makes use of the evidence. By this definition, an explanation is therefore an inference made from interpreting specific evidence. An important concept that [ 23 ] introduce is that of explanatory potential . This is the extent to which a specific set of evidences accounts for the whole of the model’s predictions. In other words, using a subset of a model’s parameters to create an explanation may be sufficient to explain a portion of the transformations that the input goes through to reach the output, say 70%, but it may not provide a complete picture. Of course, a model does not typically use each parameter equally when making predictions, so the explanatory potential of each individual piece of evidence (along with their interactions) may vary. It should be noted that explanatory potential as defined by [ 23 ] pertains only to a set of evidences. It is the upper limit of the extent to which an explanation derived from it can account for the model’s predictions, but an explanation also depends on an accurate interpretation function. Interpretations can be particularly challenging to identify given sufficiently complex transformations. Furthermore, even when derived from sound interpretations and a set of evidences with high explanatory potential, an explanation may still prove to be too complex for a target audience to understand. Thus why creating useful explanations must ultimately be a human-centered endeavor. With these considerations in mind, the question then arises of how to evaluate not just an explanation’s potential explainability, but its actual usefulness in practice. 2.2 Evaluating explainability Predictive and inferential models are typically evaluated primarily based on the accuracy of their outputs. Many robust methods for measuring different aspects of this accuracy have been devised and validated. It has become clear that no single accuracy metric captures the full complexity of a model’s performance, and so a variety of metrics are used to provide a more complete picture [ 3 , 18 ] . When it comes to evaluating explainability, however, there is