ViBED-Net: Video Based Engagement Detection Network Using Face-Aware and Scene-Aware Spatiotemporal Cues

Prateek Gothwal, Deeptimaan Banaerjee, Ashis Kumer Biswas · 2025

Read ViBED-Net: Video Based Engagement Detection Network Using Face-Aware and Scene-Aware Spatiotemporal Cues by Prateek Gothwal, Deeptimaan Banaerjee, Ashis Kumer Biswas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18016.txt.

Introduction

Prateek Gothwal, Deeptimaan Banerjee, and Ashis Kumer Biswas judge student engagement from class videos using both face and scene. Online lessons need such feedback to improve outcomes and tailor teaching. Their ViBED-Net reads facial crops and full frames side by side, pulling features with a strong image network.

Two ways of tracking change over time are compared, memory cells and transformer encoders. Extra training data for rare engagement levels helps balance the scores. The memory-cell version led the tested variants on a large e-learning benchmark. Teachers and platform designers get a concrete design for reading attention without asking students directly.

Catalog record

CREATORPrateek Gothwal, Deeptimaan Banaerjee, Ashis Kumer Biswas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ViBED-Net: Video Based Engagement Detection Network Using Face-Aware and Scene-Aware Spatiotemporal Cues Thanks: This paper is currently under review at a publisher. Prateek Gothwal, Deeptimaan Banaerjee, Ashis Kumer Biswas Abstract Engagement detection in online learning environments is vital for improving student outcomes and personalizing instruction. We present ViBED-Net (Video-Based Engagement Detection Network), a novel deep learning framework designed to assess student engagement from video data using a dual-stream architecture. ViBED-Net captures both facial expressions and full-scene context by processing facial crops and entire video frames through EfficientNetV2 for spatial feature extraction. These features are then analyzed over time using two temporal modeling strategies: Long Short-Term Memory (LSTM) networks and Transformer encoders. Our model is evaluated on the DAiSEE dataset, a large-scale benchmark for affective state recognition in e-learning. To enhance performance on underrepresented engagement classes, we apply targeted data augmentation techniques. Among the tested variants, ViBED-Net with LSTM achieves 73.43% accuracy, outperforming existing state-of-the-art approaches. ViBED-Net demonstrates that combining face-aware and scene-aware spatiotemporal cues significantly improves engagement detection accuracy. Its modular design allows flexibility for application across education, user experience research, and content personalization. This work advances video-based affective computing by offering a scalable, high-performing solution for real-world engagement analysis. The source code for this project is available on https://github.com/prateek-gothwal/ViBED-Net . Keywords Student Engagement Detection, EfficientNet, Spatiotemporal Modeling, Dual-Stream Architecture 1 Introduction Detecting student engagement in online learning environments is a critical yet challenging task in the field of educational technology. As virtual classrooms become increasingly prevalent, the ability to monitor and respond to students’ attentiveness in real time has become essential for maintaining instructional quality and personalized learning. Engagement detection from video data, in particular, presents a rich but underexplored opportunity—not only in education, but across domains such as entertainment, marketing, and gaming, where understanding user attention and affective responses can provide actionable insights. For instance, content creators can leverage engagement detection to identify the most captivating segments of their films or games, enabling targeted improvements and audience personalization [ 10 ; 14 ] . The emergence of datasets such as DAiSEE [ 3 ] and CMOSE [ 15 ] has facilitated progress in this area by providing annotated video data for training and evaluating models. DAiSEE, in particular, has become a benchmark for developing affective state recognition systems in educational settings. Among the recent advancements, the method proposed by Malekshahi et al. [ 6 ] currently achieves state-of-the-art performance on the DAiSEE dataset by leveraging a generalizable deep learning framework capable of capturing both spatial and temporal cues from video data. This work surpasses earlier hybrid architectures such as EfficientNetB7 combined with TCN, LSTM, and Bi-LSTM [ 8 ] , demonstrating enhanced generalization and accuracy in learner engagement detection. In this paper, we present a novel two-stream architecture called ViBED-Net (Video Based Engagement Detection Network) that combines EfficientNetV2 with temporal modeling modules—Long Short-Term Memory (LSTM) networks and Transformer encoders—to detect student engagement from video with state-of-the-art performance on the DAiSEE dataset. Our approach processes both the facial region and the full video frame simultaneously, capturing fine-grained emotional cues and contextual scene information. The EfficientNetV2 backbone ensures efficient and high-resolution spatial feature extraction, while the LSTM and Transformer modules model temporal dependencies crucial for understanding engagement over time. This dual-stream and multi-temporal modeling strategy significantly outperforms existing benchmarks, demonstrating the effectiveness and flexibility of ViBED-Net for real-world engagement analysis applications. By addressing a crucial gap in video-based affective computing, our work contributes a powerful, efficient, and extensible framework for multi-domain engagement analysis, with promising applications in online learning analytics, user experience research, and human-computer interaction. 2 Literature Review Student engagement detection has gained substantial traction in educational technology, especially with the widespread adoption of online learning platforms. A significant contribution to this domain is the DAiSEE dataset, which provides labeled videos capturing affective states such as engagement, boredom, confusion, and excitement in e-learning settings. As one of the earliest datasets of its kind, DAiSEE has served as a benchmark for numerous deep learning approaches targeting video-based engagement recognition [ 3 ] . Building on this foundation, Selim et al. introduced a hybrid model combining EfficientNetB7 with Temporal Convolutional Networks (TCN), LSTM, and Bi-LSTM for engagement detection. Their approach, evaluated on the DAiSEE dataset, demonstrated strong performance by leveraging both spatial and temporal modeling capabilities [ 8 ] . More recently, Malekshahi et al. proposed a general deep learning model that achieved state-of-the-art accuracy on DAiSEE by capturing engagement-related spatiotemporal patterns in a scalable and modular manner [ 6 ] . Another influential work, Do I Have Your Attention? , introduced a large-scale engagement dataset and established baseline models. The authors further validated their approach on DAiSEE, providing insights into cross-dataset generalization and the robustness of attention-based engagement predictors [ 10 ] . In parallel, the CMOSE dataset emerged as a high-quality, multimodal resource that addresses limitations in labeling granularity and modality diversity. By incorporating eye gaze, posture, and facial expression modalities with synchronized high-quality labels, CMOSE enables more holistic modeling of student engagement [ 15 ] . Collectively, these studies underscore the importance of both spatial and temporal modeling in engagement prediction. However, many prior architectures focus primarily on either facial expressions or coarse global features, often neglecting the combined value of both streams. Our proposed model, ViBED-Net, addresses this gap through a dual-stream architecture that integrates EfficientNetV2 with both LSTM and Transformer-based temporal encoders. This allows it to simultaneously capture facial micro-expressions and full-scene context, achieving superior performance on DAiSEE and offering broader applicability across domains such as content creation, marketing analytics, and user experience research. 3 Datasets 3.1 DAiSEE The DAiSEE (Dataset for Affective States in E-Environments) dataset is a large-scale video corpus curated to support research on affective computing in e-learning environments. It comprises 9,068 video snippets, each approximately 10 seconds long, collected from 112 participants in naturalistic online learning conditions. Each clip is annotated for four affective states—engaged, bored, confused, and excited—at four ordinal intensity levels: very low, low, high, and very high. Annotations were obtained via crowdsourcing and refined using majority voting to ensure label quality. The dataset includes diverse variations in lighting, facial occlusion, and head pose, making it suitable for training models that generalize to real-world conditions. Due to its scale, label richness, and ecological validity, DAiSEE has become a benchmark for video-based engagement detection in academic settings [ 3 ] . Figure 1: Frames from DAiSEE dataset 3.2 DAiSEE The DAiSEE (Dataset for Affective States in E-Environments) dataset is a large-scale video corpus curated to support research on affective computing in e-learning environments. It comprises 9,068 video snippets, each approximately 10 seconds long, collected from 112 participants in naturalistic online learning conditions. Each clip is annotated for four affective states—engaged, bored, confused, and excited—at four ordinal intensity levels: very low, low, high, and very high. Annotations were obtained via crowdsourcing and refined using majority voting to ensure label quality. The dataset includes diverse variations in lighting, facial occlusion, and head pose, making it suitable for training models that generalize to real-world conditions. Due to its scale, label richness, and ecological validity, DAiSEE has become a benchmark for video-based engagement detection in academic settings [ 3 ] . 4 Proposed Methodology In

Read the complete text