Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models

Dominik Glandorf, Fares Fawzi, Tanja Käser · 2026

Read Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models by Dominik Glandorf, Fares Fawzi, Tanja Käser (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04482.txt.

Introduction

Glandorf, Fawzi and Käser build a scalable pipeline predicting learner video-control use — watching, pausing, skipping and rewinding — from video content alone. The pipeline embeds short segments with multimodal large language models and trains a classifier to spot fine interaction peaks. From multimedia learning theory, the team codes features with GPT-5 and interprets predictions with concept vectors. Tests on 77 million control events from 66 courses show embedding-based classifiers predict peaks and generalize to unseen fields. They encode theory-relevant concepts, supporting pre-screening of educational video design.

Catalog record

CREATORDominik Glandorf, Fares Fawzi, Tanja Käser
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40528
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04482

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models Dominik Glandorf, Fares Fawzi, Tanja Käser Abstract Learners’ use of video controls in educational videos provides implicit signals of cognitive processing and instructional design quality, yet the lack of scalable and explainable predictive models limits instructors’ ability to anticipate such behavior before deployment. We propose a scalable, interpretable pipeline for predicting population-level watching, pausing, skipping, and rewinding behavior as proxies for cognitive load from video content alone. Our approach leverages multimodal large language models (MLLMs) to compute embeddings of short video segments and trains a neural classifier to identify temporally fine-grained interaction peaks. Drawing from multimedia learning theory on instructional design for optimal cognitive load, we code features of the video segments using GPT-5 and employ them as a basis for interpreting model predictions via concept activation vectors. We evaluate our pipeline on 77 million video control events from 66 online courses. Our findings demonstrate that classifiers based on MLLM embeddings reliably predict interaction peaks, generalize to unseen academic fields, and encode interpretable, theory-relevant instructional concepts. Overall, our results show the feasibility of cost-efficient, interpretable pre-screening of educational video design and open new opportunities to empirically examine multimedia learning theory at scale. 1 Introduction The rise of digital and asynchronous learning has made educational videos a ubiquitous teaching medium [ 23 ] . Millions of learners engage with video-based online learning environments such as massive open online courses (MOOCs), university-run flipped classrooms, and even general-purpose video platforms [ 22 , 21 ] . Unlike live lectures, instructors receive no direct feedback on which moments challenge learners or impose unnecessary cognitive load. Tools that identify such moments can therefore support instructional video design and iterative improvement [ 6 ] . The Cognitive Theory of Multimedia Learning (CTML) [ 18 ] is a common framework for optimizing instructional video design. It highlights cognitive load as a central variable and specifies multimedia features , such as visual complexity, redundancy, human presence, information presentation modalities, and segmentation, that affect cognitive processing demands, and it formulates principles for effective multimedia instruction. Empirical studies have related variation in these features within instructional videos to learner interaction behavior as a proxy for cognitive load. They indicate that learners pause and rewind at moments of increased processing difficulty or conceptual and visual boundaries [ 20 , 13 ] and that higher textual and graphical complexity is associated with increased pausing, rewinding, and dropout [ 11 , 3 ] . These analyses typically rely on manual annotation of multimedia features, which is labor-intensive and difficult to scale, especially at the temporal resolution of individual video moments. Moreover, they are primarily descriptive: while they identify associations between CTML features and learner behavior, they do not provide predictive models that can estimate how learners will interact with new instructional videos prior to deployment. In parallel, a large body of work has focused on predicting learning outcomes and engagement from learners’ video interaction data. Most prior studies rely on post-hoc outcome prediction (e.g., [ 15 ] ), with comparatively few predicting intermediate outcomes during a course or video interactions [ 19 , 27 ] . The features used in these models range from aggregated video control statistics (e.g., number of clicks or average pause duration; e.g., [ 15 ] ) to more fine-grained interaction patterns (e.g., [ 1 , 17 ] ), but are generally content-agnostic. Work that explicitly incorporates content typically relies on high-level course metadata [ 27 ] or treats videos as monolithic units rather than sequences of moments [ 2 , 31 , 26 , 28 , 24 ] . Even studies that model positions or sequences of activities within videos do not explicitly link theory-driven multimedia features to learner behavior [ 5 , 25 , 16 , 17 ] . As a result, predictive models of learner-video interactions that operate at moment-level granularity, use raw video content, and are grounded in multimedia learning theory remain largely unexplored. Most institutions do not systematically map large-scale video-interaction data to CTML features, nor do they reach sufficient sample sizes to draw timely conclusions about instructional quality [ 16 ] . This gap highlights the need for a scalable, explainable approach to address the cold-start problem in behavioral feedback and support data-driven video design. In this paper, we propose a scalable and explainable pipeline for predicting design-related peaks in learner-video interactions from video content alone. Our approach uses multimodal large language models (MLLMs) to compute video embeddings and trains a neural classifier to predict moment-to-moment watching, pausing, rewinding, and skipping behavior based on video content alone. To interpret these predictions, we draw on multimedia learning theory by coding CTML features with GPT-5 and applying Testing with Concept Activation Vectors (TCAV) to assess how strongly these features are encoded in the embeddings and influence model predictions. Accordingly, we address three research questions: (RQ1) to what extent moment-to-moment learner-video interactions can be predicted from video embeddings; (RQ2) whether these predictions can be explained using AI-coded CTML features; and (RQ3) how sensitive embedding-based predictions are to such features. We evaluate our approach on a multi-year dataset of over 77 million video interactions across 1,641 videos from 66 courses. Our results show that embedding-based models achieve strong predictive performance, generalize to unseen academic fields, and encode interpretable, theory-relevant CTML features. While CTML features alone are less predictive than embeddings, the TCAV analysis demonstrates that embeddings meaningfully capture multimedia learning concepts at low computational cost. In summary, leveraging recent advances in video understanding, our approach enables scalable, CTML-grounded prediction of moment-to-moment learner-video interactions, enabling cost-efficient, interpretable pre-screening of instructional video design and large-scale empirical investigation of multimedia learning theory. We publicly release the models and coding scheme, including prompts, along with the code and supplementary material 1 1 1 https://github.com/epfl-ml4ed/video-interaction-prediction-AIED26 . 2 Method Figure 1: Pipeline for predicting learners’ interactions with online learning videos and explaining predictions from multimedia learning theory. We propose a multi-step pipeline for predicting aggregated learners’ video interaction behavior (pausing, rewinding, watching, and skipping) on unseen instructional videos in an interpretable manner (see Fig. 1 ). First, we aggregate learner-video interactions into behavioral signals over the video timeline. Next, we employ MLLMs to produce embeddings of video content and train a neural network to predict these behavioral signals from the content alone. Finally, to explain the model’s predictions, we use GPT-5 to encode video content with CTML-based features and apply TCAV for interpretation. 2.1 Context and Data We evaluated our approach on a large-scale dataset of learner clickstreams collected between 2013 and 2025 from a video-based learning platform supporting MOOCs and flipped classroom courses. The platform offers courses from a technical university to both worldwide online learners and enrolled on-campus students. The courses span Science, Technology, Engineering, and Mathematics (STEM) and related fields, and many were adapted from traditional lectures and redesigned for web-based instruction. The institutional ethics review board approved the study (Approval No. 000612). We included courses with at least one hour of video material in which at least 200 active learners interacted, yielding 174 runs of 66 distinct courses (67% English, 33% local language) from 11 fields. The thresholds were chosen to balance sufficient signal quality of interaction behavior

Read the complete text