Intelligent emotion and behavior recognition for online learners using deep affective computing Authors: Priya K, Pradeep G, Ravimaran S Published: 2026 ABSTRACT AbstractTeachers in online classes often have limited visibility into how students are feeling and whether they are staying engaged. This paper presents a practical affective-computing framework that analyses webcam video to recognize both facial emotions and learning-centered behavioral states during live virtual sessions. Student faces are first detected in each frame, then encoded using a convolutional network with an attention module to emphasize informative facial regions. To capture how affect evolves over time, the extracted features are grouped into fixed 16-frame sequences and modelled with a bidirectional GRU to predict four behavioral states: engaged, confused, frustrated and disengaged. We evaluate emotion recognition on the FER2013 benchmark under the standard Train/Public Test/Private Test split, and we evaluate behavior recognition on a publicly available online-learning facial expression dataset (EmoDetect, Kaggle) used for the behavior-recognition experiments from 120 learners (800 annotated temporal segments; inter-annotator agreement κ≈0.75). The proposed model reaches 94.5% accuracy on FER2013 Private Test and achieves 86.9% accuracy with a macro-F1 of 85.4% on the four-class behavior task, improving over facial-expression-only baselines. Profiling further shows that the full pipeline can run at around 25 frames per second on a mid-range GPU, indicating feasibility for real-time monitoring. These results suggest that combining attention-based facial representations with temporal modelling can provide instructors with clearer, webcam-observable indicators of learners’ affect and engagement during online teaching.Keywords: Affective computing, Emotion recognition, Behavior recognition, Student engagement, Spatio-temporal modelling, Online learningSubject terms: Mathematics and computing, Psychology, Psychology Introduction Online learning has reshaped everyday teaching and learning, and video-conferencing platforms such as Zoom, Google Meet and Microsoft Teams are now central to many courses1. Although these tools broaden access and offer flexibility, they also reduce the visibility of behavioral and emotional cues that instructors routinely use to gauge how students are coping with the lesson2. In face-to-face classrooms, teachers can often detect confusion, frustration, boredom, or engagement at a glance and adjust explanations, pacing, or support accordingly. These decisions are informed not only by what students say, but also by nonverbal signals-facial expressions, gaze direction, head movements and other attentive behaviors-that indicate whether learners are following the content3. In virtual settings, however, such cues may be incomplete or unreliable because of low-quality webcams, unstable lighting, limited screen space, and the small grid-view windows in which students appear. As a result, instructors may find it difficult to assess understanding during the session and to identify students who need timely help before disengagement occurs4. Affective computing, and in particular facial-expression analysis, has been explored as a practical way to recover some of the learner cues that are less visible in virtual classrooms. Facial Expression Recognition (FER) systems apply computer vision and deep learning to infer affective states from facial movements and appearance patterns5. However, much of the FER literature is built around a small set of basic emotion labels-such as happiness, sadness, anger, fear, and surprise-typically predicted at the level of individual frames. Although these categories provide a general indication of affect, they do not adequately describe the learning-related states that matter during instruction. In online classes, students often exhibit learning-centered behavioral states, including confusion, frustration, sustained mental effort, and disengagement, which can strongly influence comprehension and attention6. Many current FER pipelines overlook these states because they treat expressions as isolated snapshots rather than modelling how affective cues persist, fluctuate, and transition over time within a learning episode. Capturing these temporal dynamics is therefore important for moving from generic emotion recognition to actionable indicators of learner engagement. Additional challenges stem from the characteristics of online video streams. Students appear under highly variable lighting, low-resolution webcams, cluttered or changing backgrounds, and grid-style layouts where many participants are displayed simultaneously7. Methods developed using high-quality, single-face images often degrade in this setting, because faces may be small, partially occluded, or distorted by compression and network artefacts. These conditions can reduce detection reliability and make frame-level predictions unstable, especially when learners shift posture, look away from the screen, or when the platform dynamically resizes video tiles. This motivates the need for robust, context-aware models tailored to online learning, capable of handling multi-student scenes and leveraging temporal information to stabilise predictions and better reflect learning-centered behavioral dynamics8. Deep-learning methods now make it practical to move beyond single-frame emotion labels and towards more reliable analysis in real classroom video. Convolutional networks can extract facial representations that remain informative under noise and compression, attention mechanisms can focus the model on regions that carry expression cues, and sequence models can summarise short-term dynamics rather than treating each frame independently. When these components are combined, the model can use both what is expressed on the face and how that signal changes over time, which is important for learning-centered behavioral states that develop across a short interval rather than appearing as a single snapshot. This spatial-temporal modelling typically produces more stable predictions in online-class conditions than frame-only FER pipelines9. To address these gaps, we develop a deep affective-computing framework intended for online teaching scenarios. The approach estimates facial affect from individual video frames and then uses temporal context to infer learning-centered behavioral states-engagement, confusion, frustration, and disengagement-from short sequences of webcam footage10. An attention-augmented CNN is used to emphasize informative facial regions and produce robust feature representations, while a temporal behavior module aggregates features across successive frames to reduce the instability typical of frame-only FER outputs11. The pipeline is further designed for realistic virtual classrooms by incorporating multi-student face detection that remains effective under low-resolution cameras, compression artefacts, variable lighting, and grid-view layouts12. This study examines whether a single deep-learning pipeline can track students’ affect and learning-centered behavior from ordinary webcam video during live online classes. The goal is to use immediate facial cues together with short temporal context to infer behavioral states-engagement, confusion, frustration, and disengagement-more reliably as learners progress through instructional content. Understanding learners’ emotional and behavioral states can help instructors make better decisions during teaching, especially in online classes where many in-person cues are difficult to observe. The proposed framework is designed to provide timely, webcam-based indicators that can help educators notice learners who may be struggling, track observable engagement signals, and adjust explanations or pacing during live sessions when appropriate13. More broadly, systems that analyse affect and behavior in real time may support the development of emotionally aware digital learning environments by enabling earlier intervention and more personalized support, provided that privacy, consent, and responsible use are clearly addressed14. The main contributions of this study are as follows: A unified deep affective-computing framework is presented to perform frame-level emotion recognition and sequence-level learning-centered behavior classification in real time from webcam video in online classes. A unified deep affective-computing framework is presented to perform frame-level emotion recognition and sequence-level learning-centered behavior classification in real time from webcam video in online classes. An attention-enhanced CNN is formulated to extract fine-grained facial representations under realistic virtual-classroom conditions, including low resolution, variable illumination, compression artefacts, and grid-view layouts. An attention-enhanced CNN is formulated to extract fine-grained facial representations under realistic virtual-classroom conditions, including low resolution, variable illumination, compression artefacts, and grid-view layouts. A temporal modelling component is incorporated to interpret short-term affective dynamics and infer four behavioral states: engaged, confused, frustrated, and disengaged. A temporal modelling component is incorporated to interpret short-term affective dynamics and infer four behavioral states: engaged, confused, frustrated, and disengaged. A multi-student face detection and per-student analysis pipeline is implemented to support reliable inference in multi-participant video streams typical of virtual classrooms. A multi-student face detection and per-student analysis pipeline is implemented to support reliable inference in multi-participant video streams typical of virtual classrooms. Comprehensive evaluation is reported using FER2013 and a the publicly available EmoDetect dataset, including quantitative comparisons and ablations that demonstrate advantages over frame-only FER baselines. Comprehensive evaluation is reported using FER2013 and a the publicly available EmoDetect dataset, including quantitative comparisons and ablations that demonstrate advantages over frame-only FER baselines. The remainder of this paper is organized as follows. Section 2 summarizes related work in affective computing, with emphasis on facial expression recognition and learning-centered behavior analysis for online education. Section 3 describes the proposed framework, including the multi-student face detection stage, the attention-enhanced emotion recognition network, and the temporal module used for behavior inference. Section 4 presents the datasets, preprocessing pipeline, and experimental protocol used for training and evaluation. Section 5 reports the results, compares the proposed approach with existing methods, and discusses the behavioral patterns observed in the online-class setting. Finally, Section 6 concludes the study and highlights directions for future work toward more advanced, emotionally intelligent online learning systems. Related works Recent research has increasingly focused on understanding student emotions and behavior in online learning environments, especially as virtual classrooms have become widely adopted. Early studies primarily used conventional CNN-based Facial Expression Recognition (FER) models trained on benchmark datasets such as FER2013 and RAF-DB. While these systems recognized basic emotions effectively, they were not specifically developed for online educational settings and often struggled with real-world constraints such as low video resolution and multi-student grid views. Theoretical grounding and construct operationalisation Student engagement is widely conceptualised in educational psychology as a multidimensional construct that includes behavioral, emotional, and cognitive components rather than a single facially observable state15,16. The ICAP framework further connects learners’ overt activity to the depth of cognitive engagement by distinguishing Interactive, Constructive, Active, and Passive modes17. In webcam-based online classes, however, the available signals are largely limited to facial behavior and coarse attentional indicators (e.g., on-screen gaze/orientation). Accordingly, we operationalise engagement in this work as webcam-observable engagement indicators that primarily reflect emotional engagement and aspects of behavioral/attentional engagement, and we do not claim direct measurement of deeper cognitive or interactive engagement processes. Confusion and frustration are treated as learning-related achievement emotions. Under the Control–Value Theory of Achievement Emotions, such emotions arise from learners’ appraisals of control (e.g., perceived competence) and value (task importance), and can influence persistence and strategy use during learning18. In parallel, Cognitive Load Theory provides a complementary explanation: demanding tasks may elicit confusion/frustration when working-memory demands become high relative to available resources19. Finally, Self-Determination Theory motivates the instructional relevance of monitoring these indicators20, although SDT needs are not measured directly in the present vision-only setup. This framing clarifies that the proposed system predicts operational, observable affective–attentional states aligned with established learning theories and informs the segment-level annotation codebook used for behavior labels. Beyond basic FER benchmarks, several datasets focus specifically on engagement and learning-centered affect in e-learning settings. DAiSEE provides in-the-wild e-learning videos annotated for boredom, confusion, engagement, and frustration, supporting sequence-based engagement analysis21. An additional Indian-student affective database captures faces along with hand gestures and body postures for both e-learning and classroom environments, using structured annotation procedures22. These resources motivate the need for practical models that operate under webcam-quality constraints and clearly reported evaluation protocols. Prior work on affective dynamics during learning commonly analyses a compact set of learning-centered states such as engagement/flow, confusion, frustration, and boredom, including their transitions during problem solving and tutoring interactions23,24. In parallel, disengagement/off-task behavior is an established construct in learning analytics and has been operationalised as task-unrelated activity or withdrawal23. These empirical standards motivate the use of a concise behavior taxonomy that remains interpretable and instructionally actionable in webcam-based settings. Salloum et al.25,26 presented a refined CNN-based FER model trained on FER2013 for classifying seven basic emotions. Their study achieved high accuracy and demonstrated that CNNs can support emotion-aware personalized teaching. However, their work focused only on basic emotions and relied on generic datasets rather than learning-centered affect, limiting its applicability to online classroom behavior understanding. Rahman et al.27 introduced the EmoDetect dataset, one of the first learning-centered affective databases designed specifically for online learning scenarios. Their dataset included real student facial videos annotated for frustration and non-frustration during online tasks. Although EmoDetect contributed significantly to dataset development for educational emotion research, the authors mainly addressed data collection and manual coding, without proposing a complete deep-learning model for real-time deployment. Aly et al.28 reported an emotion-recognition model built on a ResNet-50 backbone enhanced with a Convolutional Block Attention Module (CBAM) to emphasize informative facial regions. Their implementation improves fine-grained facial feature representation and was presented in the context of an online learning support application. Nevertheless, the approach remains centred on emotion classification and does not incorporate an explicit behavior-recognition layer or temporal modelling to capture how affective cues evolve across an online session. Zhang et al.29 proposed a spatio-temporal model that integrates a ResNet50 feature extractor with CBAM attention and a Temporal Convolutional Network (TCN) to capture facial-expression dynamics in online-learning video. Their study explicitly considers practical challenges such as illumination variability, partial occlusions, and temporal dependency between frames, and it demonstrates that temporal modelling can improve the robustness of continuous emotion estimation. However, the focus remains on tracking emotion progression, and the framework does not extend to learning-centered behavioral constructs-such as confusion episodes, disengagement, or indicators of sustained cognitive effort-that are typically required for instructional interpretation. Wang et al.30,31 introduced SFER-YOLOv5, a lightweight facial-expression recognition approach intended for multi-student online classroom video. The model combines modifications such as soft-NMS, coordinate attention, and revised loss formulations to improve efficiency and robustness under low-resolution streams and grid-view layouts. While the study reports engagement-related outputs using broad affective groupings (e.g., positive, neutral, negative), the resulting engagement assessment remains coarse. In particular, it does not separate finer learning-centered behavioral states-such as sustained frustration, confusion episodes, or indicators of prolonged mental effort-that are important for instructional interpretation Table 1. Table 1Summary of recent related works.AuthorModelPerformanceAdvantagesLimitationsResearch gapZhao et al.32Multi-branch vision transformer (MB-ViT) for student affect predictionAccuracy: 92.8%, F1: 91.3%Captures global and local facial features; robust to lighting variationNo behavior modelling; focused only on static framesLack of temporal modelling and multi-student detection for online classroomsKhan et al.33Lightweight CNN with Attention for FER in low-resolution webcamsAccuracy: 90.2%Optimised for low-quality video input; real-time friendlyLimited to 7 basic emotions; no learning-centered affect states (confusion, frustration)Need dynamic behavior classification suited for online learningJain et al.34Hybrid ResNet-GRU model for classroom engagement trackingEngagement F1: 88.7%Temporal modelling improves engagement detectionDataset small; no multi-student grid-view handlingRequires robust multi-student tracking + behavioral sequence modellingWahyuni et al.35Facial-gaze fusion model for online learner attention estimationAccuracy: 89.4%Combines eye-gaze + FER for better attention trackingHigh sensitivity to head pose changes; gaze estimation unstable in webcam videosMissing deep temporal learning for behavior dynamicsPark et al.36Ensemble CNN-LSTM model for recognising cognitive load from facial cuesAccuracy: 91.5%Captures temporal variations linked to mental effortCannot recognize disengagement; requires controlled lightingNeeds a unified affect-behavior framework for real-time online education Summary of recent related works. These studies collectively highlight the growing interest in applying affective computing to virtual learning, but they also reveal important limitations. Most models still rely on basic emotions rather than learning-specific affective states. Several works lack temporal modelling to interpret how emotions evolve throughout learning activities. Engagement analysis remains coarse, and few systems integrate facial detection, emotion recognition, temporal behavior analysis, and multi-student processing into a unified, real-time framework designed specifically for online classrooms. Proposed methodology The proposed methodology introduces an intelligent deep affective computing framework designed to recognize both emotional states and learning-centered behavioral patterns of students in real time during online classes. The system integrates multi-student face detection, attention-enhanced facial emotion recognition, and temporal behavior modelling into a single end-to-end. The overall workflow is illustrated conceptually as a series of interconnected modules that operate sequentially to extract, analyse, and interpret affective signals from online classroom video streams Table 1. System overview The proposed framework operates through three integrated modules that work together to analyse student’s affective states in real time during online classes. First, the multi-student face detection module identifies and extracts all visible student faces from each video frame, even under challenging virtual classroom conditions such as low-resolution webcams, varied lighting, and grid-style layouts. A lightweight YOLO-based face detector is used to localise student faces efficiently in multi-participant online-class video. The detected regions are cropped, aligned, and normalized before being processed by an attention-enhanced CNN for emotion recognition. The attention mechanism emphasizes informative facial regions (for example, around the eyes and mouth) so that subtle expression cues are retained in the resulting frame-level feature representation. These features are then passed to a temporal behavior module, which models short sequences using a Temporal Convolutional Network (TCN) or a bidirectional GRU to capture how affective cues evolve across consecutive frames. Incorporating temporal context enables the inference of learning-centered behavioral states-including engagement, confusion, frustration, and disengagement-under common virtual-classroom conditions21,22. Together, the detection, emotion-encoding, and temporal modelling components form an integrated pipeline for continuous monitoring of students’ affective and behavioral patterns in online learning environments. The overall workflow of the proposed deep affective-computing framework is illustrated in Fig 1. Overview of the proposed deep affective-computing framework for recognising learners’ emotions and learning-centered behavioral states in online classes. A live virtual-classroom video stream is first sampled into frames, and multiple learners are detected in the scene. Detected face regions are then cropped and aligned, with basic quality normalisation applied when needed to improve robustness under low resolution and variable lighting. The processed faces are passed to the emotion-recognition network to produce frame-level affective representations. These representations are subsequently aggregated over short temporal windows by the behavior module to infer learning-centered states over time. These pre-processed faces are then analysed using an attention-enhanced CNN–CBAM model to extract expressive facial features and classify frame-level emotions. A temporal modelling layer (TCN/Bi-GRU) observes how these emotions evolve over time to identify higher-level behaviors such as engagement, confusion, frustration, or disengagement. Consistent with multidimensional engagement definitions and ICAP, these behavior labels are treated as webcam-observable engagement indicators rather than direct measures of latent cognitive engagement Fredricks et al.15,16,17. The outputs are visualised through an analytics layer that provides per-student summaries and instructor-support insights, while an evaluation and explainability module reports performance and highlights important facial regions. A dedicated training unit continuously improves the model using FER datasets and classroom data. Dataset description and model formulation The proposed affective computing framework requires two categories of data: (i) facial expression images for emotion recognition, and (ii) continuous video sequences for learning-centered behavior analysis. To achieve this, the system uses a combination of publicly available facial expression datasets for pre-training and a publicly available online-learning facial expression dataset (EmoDetect, Kaggle) for behavior-recognition experiments. This hybrid dataset strategy ensures that the model learns general emotional features while also adapting to the unique challenges of real online classroom environments. The public FER datasets-such as FER2013, and AffectNet-provide large, diverse collections of annotated facial expressions. These datasets are used to initialise the CNN with strong generalisable emotional features. Formally, each FER dataset provides input samples. The following equations are included to standardise notation and to describe the implemented pipeline clearly for reproducibility. They formalise commonly used deep-learning steps (feature extraction, temporal aggregation, and softmax classification) and are not intended to provide new theoretical guarantees. The main contribution of this study lies in the end-to-end system integration and the carefully reported empirical evaluation on the two tasks. where denotes a facial image and represents the emotion label across K emotion classes. The CNN learns a mapping which transforms each image into a d-dimensional emotional feature vector. These pre-trained weights serve as the foundation for recognizing emotions in online classroom video frames. To adapt the model for learning-related behaviour detection, a publicly available online-learning facial expression dataset was used. The dataset contains student facial videos and images recorded in online-learning settings and offers useful affective cues for analysing behaviour in realistic virtual classroom conditions. For this study, the data were arranged into fixed temporal segments and processed according to the experimental protocol described below. where each segment contains a sequence of cropped face frames with m consecutive frames per segment. In the final experimental configuration, each temporal segment is formed using m = 16 consecutive frames. A short pilot check was conducted using segment lengths in the 16–32 frame range, and m = 16 was selected for reporting because it provides sufficient short-term behavioral context while still enabling timely behavior updates during online inference. Each segment is manually annotated with behavior labels such as engaged, confused, frustrated, or disengaged, generating supervised pairs where denotes the segment-level behavior label assigned through the manual annotation protocol, using webcam-observable affective and attentional cues within the window. The label is not assumed to be a direct function of any single frame-level emotion. Behavior classes are defined as segment-level learning indicators inferred from temporal dynamics and coarse attentional cues, and they are not interpreted as a one-to-one mapping of the frame-level basic emotion categories. The methodological pipeline is theoretically grounded in a spatio-temporal representation learning paradigm. First, each frame is passed through the CNN encoder, generating Resulting in a feature set for segment : This forms the input to the temporal modelling module-either a Temporal Convolutional Network (TCN) or a Bi-directional GRU (Bi-GRU)-which learns the mapping Thereby mapping temporal dynamics of face-derived feature embeddings to segment-level behavior indicators in the proposed taxonomy. This connectivity allows the framework to combine instantaneous facial affect with temporal dynamics to model webcam-observable affective–attentional engagement indicators over time. To improve robustness, all data undergo preprocessing including face alignment, brightness and contrast standardisation, motion-blur compensation, and temporal smoothing. Data augmentation is used during training to increase variability and improve generalisation under typical online-class conditions. The applied transformations include random cropping, small rotations, horizontal shifts, brightness jitter, and mild Gaussian noise. This is especially relevant for behavior recognition, where certain classes may be less frequent (for example, frustration), and augmentation helps reduce overfitting to majority categories. Overall, the dataset strategy combines the breadth of standard FER benchmarks with the ecological validity of real online-classroom recordings, enabling both controlled benchmarking and evaluation under realistic deployment conditions. Mathematically structured data flows ensure a unified spatio-temporal learning architecture capable of identifying both instantaneous emotional cues and complex behavioral states, enabling reliable real-time affective analysis in modern virtual learning environments. Preprocessing Let the online classroom video be represented as a sequence of frames. where denotes the frame in the video and T is the total number of frames in a given session. In each frame , there may be visible students. The multi-student face detection module operates on every frame and produces a set of face bounding boxes. where denotes the region of interest (ROI) corresponding to the student’s face in frame t. Each detected face region is cropped and resized to a fixed spatial resolution H × W, generating a normalized face image Standard preprocessing is then applied, including intensity normalisation, optional histogram equalisation, and light data augmentation such as random shifts, small rotations, and controlled brightness changes. These operations increase robustness against typical online classroom variations such as illumination changes, low-quality webcams, and slight head movements. For behavior modelling, consecutive frames are grouped into temporal segments of fixed length m. A fixed temporal segment length of m = 16 frames is used for all reported behavior-recognition experiments. This setting matches typical online video sampling rates (10–15 FPS) and captures short behavioral transitions while keeping inference delay low. Temporal window lengths up to 32 frames were explored during pilot experiments; however, under the same real-time latency budget, extending the window did not deliver consistent performance gains. Therefore, m = 16 is retained as the final configuration to balance temporal context with timely behavior updates. For a given student, a temporal segment can be defined as where denotes the temporal segment for that student, is the starting frame index for the segment, and m is the number of frames in the segment. Each segment is associated with a behavior label where This representation establishes a direct connection to the proposed workflow: frames are captured from the online video stream, faces are extracted by the multi-student detection module, preprocessed and augmented, and then organized into temporal segments. These segments act as input sequences for the temporal behavior modelling module, thereby linking low-level visual observations to high-level behavioral states. Emotion feature extraction with attention-enhanced CNN The emotion recognition component is built on a deep convolutional neural network (CNN) augmented with attention mechanisms to focus on the most informative facial regions. Given a preprocessed face image the CNN encoder produces a high-level feature tensor where C is the number of feature channels, H′ and W′ are the reduced spatial dimensions, and θ denotes the learnable CNN parameters. To enhance discriminative power, a Convolutional Block Attention Module (CBAM) is applied on First, channel attention is computed using both global average pooling and global max pooling across spatial dimensions. The pooled descriptors are passed through a small shared multilayer perceptron (MLP), and their outputs are combined and activated (e.g., by a sigmoid function). The channel-refined feature map is then given by where ⊗ denotes element-wise multiplication. Next, spatial attention is obtained by applying average pooling and max pooling across channels to , concatenating the resulting maps, and passing them through a 7×7 convolution followed by a sigmoid activation. The spatially refined feature map becomes A global pooling operation (e.g., global average pooling) is then applied to to obtain a fixed-length feature vector which serves as the frame-level emotional embedding for student i at time t. A fully connected layer followed by softmax activation predicts the emotion probability distribution: where and are learnable parameters, and the entries of correspond to basic and learning-related emotion categories (e.g., happy, neutral, confused, frustrated). From a theoretical perspective, this attention-enhanced CNN defines a mapping from raw pixel space to a compact affective embedding space, while CBAM selectively amplifies channels and spatial locations most relevant for emotion discrimination (e.g., eye region for confusion, mouth region for frustration). The resulting vector becomes the interface between instantaneous facial emotion analysis and the subsequent temporal behavior modelling stage. Temporal behavior modelling with TCN/Bi-GRU While captures the emotional state at a single time point, In the present study, emotion and behavior are handled as related but distinct concepts and are analysed at different time scales. Emotions refer to instantaneous (frame-level) affective states inferred from facial actions, whereas behavior labels refer to short-duration (segment-level) learning indicators that summarise sustained attention and affect patterns within a time window. The temporal module processes per-frame feature embeddings and learns the association between their temporal evolution and the annotated segment label; therefore, emotion persistence is treated only as a supporting indicator and is not assumed to determine behavior in a fixed manner. As the input is limited to webcam face video, additional behavioral evidence such as posture, interaction logs (mouse/keyboard), and task-performance measures is outside the scope of the present dataset and remains a suitable direction for future extension.For each temporal segment of length m, the corresponding frame-level features are stacked into a sequence This formulation summarizes the implemented temporal modelling pipeline to improve clarity and support reproducibility. m denotes the number of frames per temporal segment (fixed to m=16 in all reported experiments), and d denotes the per-frame feature dimension produced by the backbone network (fixed by the architecture, i.e., d=[128]) The objective of the temporal module is to model the dynamics within and infer the behavior label Two alternative architectures are considered for temporal modelling: a Temporal Convolutional Network (TCN) and a Bidirectional Gated Recurrent Unit (Bi-GRU). TCN-based modelling In the TCN configuration, a stack of 1-D causal convolutions with dilations is applied along the temporal dimension. This allows the network to capture long-range dependencies while maintaining temporal order The TCN. Transformation can be expressed as where ϕ denotes the temporal convolution parameters and denotes the temporal convolution parameters and is a compact representation summarising the emotional evolution within segment . Bi-GRU-based modelling In the Bi-GRU configuration, the feature sequence is processed by forward and backward GRU cells. For each time step t in the segment, the forward and backward hidden states are updated as and the combined representation at time t is . An aggregation function (e.g., average pooling, max pooling, or using the last hidden state) is then applied over the segment: In both TCN and Bi-GRU cases, encodes the temporal dynamics of face-derived feature embeddings (affect-related and coarse attentional cues) observed within the segment.. A fully connected layer with softmax activation converts this into a behavior probability distribution: where and map the temporal representation to the behavior classes in Conceptually, this stage aggregates temporal patterns in face-derived embeddings to estimate segment-level behavior indicators aligned with the annotation codebook. For example, a sustained pattern of neutral and mild confusion may be classified as confused, whereas stable positive affect with attentive gaze may be labelled as engaged. Joint learning objective The proposed framework is trained in a multi-task learning setting, jointly optimising both frame-level emotion recognition and segment-level behavior classification. Let denote the cross-entropy loss for emotion prediction and denote the cross-entropy loss for behavior prediction. For a batch of facial frames with ground truth emotion labels the emotion loss is defined as where c indexes the emotion classes, is the predicted probability of class c, and 1{.}is the indicator function. For a batch of temporal segments with ground truth behavior labels the behavior loss is given by where j indexes the behavior classes and is the predicted probability of behavior class j. The overall training objective is a weighted sum of these two losses: where α and β are hyperparameters controlling the relative importance of emotion and behavior learning. By jointly minimising L, the CNN is encouraged to produce emotion features that are not only accurate for instantaneous expression recognition but also informative and stable for downstream behavior modelling. This joint learning strategy tightly couples low-level affect recognition with high-level learning-centered behavior analysis, resulting in a coherent and robust affective computing framework for online classroom environments. Algorithm 1 Training procedure for the proposed affective computing framework. Algorithm 2 Real-time emotion and behavior inference. Both algorithms outline the complete functioning of the proposed system. Algorithm 1 explains the training process, where the CNN is first pretrained on a large FER dataset and then fine-tuned using classroom video data after detecting faces, forming temporal segments, and assigning behavior labels. The temporal model (TCN/Bi-GRU) is then trained to model how face-derived feature embeddings evolve across frames to support segment-level behavior inference, and finally both emotion and behavior prediction losses are jointly optimised. Algorithm 2 describes how the trained system works in real time during an online class: frames are captured continuously, student faces are detected and preprocessed, and the CNN predicts the emotion for each frame. These predictions are collected in a sliding window, and once enough frames are available, the temporal model analyses the sequence to determine the current behavior state-such as engagement, confusion, frustration, or disengagement-with updates happening continuously throughout the session. Result and discussion This section describes the implementation environment, datasets, preprocessing pipeline, and experimental setup used to evaluate the proposed intelligent affective computing framework for online learners. The details are organized into system configuration, dataset description, and preprocessing architecture so that the experimental results in the subsequent subsections can be interpreted clearly from a practical deployment perspective. System configuration and implementation environment The proposed framework was implemented using Python and deep learning libraries to support real-time processing of online classroom video streams. All experiments were conducted on a workstation equipped with an Intel Core i7/Ryzen 7-class CPU, 32 GB RAM, and an NVIDIA RTX 3060/3080 GPU with at least 8–10 GB VRAM, running Windows 11/Ubuntu 22.04. GPU acceleration was essential to handle multi-student face detection, CNN inference, and temporal sequence modelling and runtime profiling results are reported in Section 4.1.1 to quantify FPS, latency, and compute usage. On the software side, the models were developed using PyTorch for deep learning, OpenCV for video capture and frame manipulation, and scikit-learn for computing evaluation metrics such as precision, recall, F1-score, and confusion matrices. Jupyter/Google Colab was used during prototyping for rapid experimentation and visualisation, whereas a standalone Python script was used for real-time inference. This configuration ensures that the proposed system can be reproduced on a typical mid-range research workstation or modern cloud GPU instance, making deployment feasible in institutional settings. Runtime profiling and real-time feasibility To support the claim of real-time suitability, runtime profiling is reported under the same hardware and software configuration described in Section 4.1. Because the framework operates at two stages, latency is measured separately for frame-level emotion inference and segment-level behavior inference, where the behavior module processes a short temporal window of consecutive frames. Timing results are obtained with batch size = 1 and include the complete inference pipeline, including preprocessing and the model forward pass. In addition to latency, the achieved throughput (frames per second, FPS) is reported under the same settings. To clarify deployment requirements, peak GPU memory usage and average CPU utilisation during inference are also documented. Together, these measurements provide reproducible evidence for interactive performance. Table 2 summarizes the practical inference cost of the proposed method in terms of latency, throughput, and hardware usage. The reported values indicate the per-frame time for emotion inference, the per-segment time for behavior inference (16-frame window), the achieved FPS during inference, and the corresponding peak GPU memory and average CPU utilisation, providing quantitative support for the real-time feasibility claim. Table 2Runtime and resource profiling for the proposed inference pipeline.MetricValueEmotion inference latency≈ 7 ms/frameBehavior inference latency≈ 15 ms/segment (segment length = 16 frames)Throughput≈ 25 FPSPeak GPU memory≈ 3.5 GBCPU utilisation≈ 20% Runtime and resource profiling for the proposed inference pipeline. Dataset description The proposed framework requires datasets that capture both general emotional expressions and learning-centered behaviors commonly exhibited during online classes. To achieve this, a hybrid dataset strategy was adopted, combining. (i)A widely used benchmark facial expression dataset for pretraining the CNN-CBAM emotion encoder, and. A widely used benchmark facial expression dataset for pretraining the CNN-CBAM emotion encoder, and. (ii)An online-learning-specific facial expression dataset containing videos and images of students participating in real virtual classrooms. This combination ensures that the model first learns generalisable facial emotion features and then adapts to the subtle affective variations present in online learning environments. An online-learning-specific facial expression dataset containing videos and images of students participating in real virtual classrooms. This combination ensures that the model first learns generalisable facial emotion features and then adapts to the subtle affective variations present in online learning environments. Public facial expression dataset (FER2013) The FER2013 dataset is a large-scale facial emotion dataset originally released as part of the Facial Expression Recognition Challenge. In the context of this study, FER2013 was used to pretrain the attention-enhanced CNN backbone. Pretraining on a large and diverse dataset enables the CNN to learn generalisable low-level and mid-level facial feature representations. This step is particularly important because it stabilises training when transferring to smaller, domain-specific datasets. The extracted features serve as an initialisation that captures emotional cues such as eyebrow contraction (anger), nose wrinkling (disgust), widened eyes (fear/surprise), and lip curvature (happiness/sadness). This improves the downstream performance when the model is fine-tuned using online-learning-specific data. Figure 2 shows representative samples from the FER2013 emotion categories. The dataset exhibits substantial variation in pose and alignment, including near-frontal faces as well as partially rotated views, with noticeable differences in tilt and horizontal/vertical offsets across images.This variability makes the dataset particularly valuable for real-world applications, as it reflects the natural pose differences commonly observed in unconstrained environments such as online classrooms or everyday interactions. Number of images per emotion category in the FER-2013 dataset. The FER-2013 dataset, obtained from the Kaggle repository (https://www.kaggle.com/), was employed as one of the primary benchmark datasets in this study. It contains a total of 35,887 grayscale facial images, each with a spatial resolution of 48 × 48 pixels. In the Kaggle release used in this study, FER2013 is provided in three predefined splits: Train (28,709 images), PublicTest (3,589 images), and PrivateTest (3,589 images); Table 3 reports the class-wise distribution across these splits. Table 3Class-wise distribution of FER2013 samples across the predefined train, publictest, and privatetest splits (7 emotion categories).ClassTrainPublic testPrivate testAngry3995467491Disgust4365655Fear4097496528Happy7215895879Sad4830653594Surprise3171415416Neutral4965607626Total2870935893589 Class-wise distribution of FER2013 samples across the predefined train, publictest, and privatetest splits (7 emotion categories). Temporal segment length m was tuned in the range 16–32; m = 16 is used for all final experiments reported in this paper. Table 3 reports the number of images available for each emotion class in the standard FER2013 splits. The distribution is clearly imbalanced (for example, Disgust has far fewer samples than Happy and Neutral), therefore macro-averaged metrics are reported in addition to accuracy to reflect class-balanced performance. This Fig 3 compares the number of images per emotion class in each FER2013 split. It shows that the Train split contains the majority of samples for all classes, while PublicTest and PrivateTest have smaller but consistent counts, confirming the corrected split totals used in Table 3. Class-wise distribution of FER2013 samples across Train, PublicTest, and PrivateTest splits. This Table 4 lists the fixed hyperparameter values used to generate all final reported results, distinguishing the final configuration from any tuning ranges discussed elsewhere. Table 4Final experimental hyperparameter settings used for all reported results (m=16, d=128, stride = 16, sampling 10–15 FPS).SettingFinal valueTemporal window length ((m))16 framesStride16 framesSampling rate10–15 FPSFeature dimension ((d))128Behavior classes4 (engaged, confused, frustrated, disengaged)FER2013 classes7 (angry, disgust, fear, happy, sad, surprise, neutral) Final experimental hyperparameter settings used for all reported results (m=16, d=128, stride = 16, sampling 10–15 FPS). Online learning/engagement dataset To examine the effectiveness of the proposed temporal behavior modelling under practical e-learning conditions, a publicly available online-learning facial expression dataset (EmoDetect, Kaggle) was used for the behavior-recognition experiments. This dataset is used exclusively for the four-class behavior recognition task-engaged, confused, frustrated, and disengaged-because these learning-centered states are best captured through short time-based observations rather than single images. In parallel, FER2013 is used only as a standard public benchmark to validate the proposed feature learning and attention mechanism for facial emotion recognition. For transparency and scientific clarity, the revised manuscript reports the dataset collection setting, temporal segmentation strategy, annotation procedure, inter-annotator agreement, and class-wise sample distribution in the next subsection, so that the behavior results can be interpreted fairly and reproduced by other researchers27. (Kaggle Online Engagement Dataset, 2023). https://www.kaggle.com/datasets/mistresearch/students-facial-expressions-in-online-learning-v1. FER2013 (Kaggle) is used only as a standard 7-class facial emotion recognition benchmark, following the official Train/PublicTest/PrivateTest split and evaluation protocol. The Student Engagement Prediction Dataset (Kaggle) and EmoDetect (Kaggle) are cited as publicly available reference resources to support transparency and to guide reproducible experimental design; however, they are not used for the final 4-class behavior recognition results reported in this study. All behavior-recognition experiments and conclusions are based exclusively on the publicly available EmoDetect dataset accessed through Kaggle, and the manuscript reports FER2013 and behavior results in separate sections, tables, and figures to prevent any mixing of datasets, splits, or evaluation metrics. The EmoDetect-type datasets include labels such as frustrated vs. non-frustrated, whereas student engagement datasets include engaged and not engaged classes. These classes reflect behavioral cues important for understanding learning performance, such as gaze direction, attentional focus, facial tension, and micro-expressions of confusion. Because these datasets capture facial behavior during actual pedagogical activities (e.g., listening to lectures, reading text, solving problems), they provide crucial context-specific affective information that benchmark FER datasets do not contain. Within this research, the online learning dataset was used to fine-tune the pretrained CNN and to train the temporal behavior model (TCN/Bi-GRU). Unlike FER2013, which emphasizes static emotions, the online learning dataset captures dynamic, subtle emotional transitions that are tightly connected to learning challenges. This makes it suitable for behavior recognition tasks such as identifying engagement, confusion, frustration, and disengagement. In this work, the behavior recognition performance is evaluated using. The publicly available EmoDetect dataset described in the preceding subsection, since behavior labels such as engaged, confused, frustrated, and disengaged require time-based annotations collected in the actual learning context. Publicly available datasets are referred to only for supporting purposes, such as benchmarking the emotion recognition component or pretraining the feature extractor, where standard datasets are appropriate. These public datasets are therefore not considered replacements for behavior-level ground truth, and the reported behavior results should be interpreted strictly with respect to the annotated EmoDetect online-learning dataset used in this study. Public online-learning dataset (EmoDetect): dataset characteristics and annotation The publicly available EmoDetect dataset comprises webcam-based online classroom recordings collected under realistic e-learning conditions, where factors such as lighting, background clutter, and camera quality naturally vary across participants. To enable temporal behavior analysis, each recording is partitioned into fixed-length time segments using a consistent windowing rule. Each segment is then labelled according to a predefined annotation protocol into one of four learning-centered behavior categories-engaged, confused, frustrated, or disengaged. This segment-wise representation supports the proposed temporal modelling by capturing short-term changes in facial cues that are commonly observed during online learning. Table 5 summarizes the online-learning dataset used for behavior recognition, including the number of learners, recording setup, temporal segmentation (16-frame window/stride), total labelled segments (800), approximate labelled duration, and the annotation reliability process (two annotators with adjudication and Cohen’s κ). Table 5Summary of the online-learning behavior dataset, including participants, recording settings, temporal segmentation, labelled samples, and annotation agreement.ItemValueTotal participants120 learnersTotal recording sessions120 session-level recordingsRaw recording duration70–90 hours total (≈35–45 min/session)Video settingsWebcam e-learning recordings; frames resized to model input; 10–15 FPSBehavior classesEngaged, confused, frustrated, disengagedSegment window/stride16 frames/16 framesTotal annotated segments800 temporal segmentsTotal labelled (segment) duration≈14.2–21.3 minutes (800×16 frames at 10–15 FPS)Annotators2 annotators + 1 adjudicator (protocol-based labelling)Inter-annotator agreementCohen’s κ ≈ 0.75 (computed on ≈20% overlap)Disagreement handlingDiscussion; remaining conflicts adjudicatedClass distribution (800)Engaged ≈320 (40%); Confused ≈200 (25%); Frustrated ≈160 (20%); Disengaged ≈120 (15%) Summary of the online-learning behavior dataset, including participants, recording settings, temporal segmentation, labelled samples, and annotation agreement. The online-learning behavior dataset comprises 800 annotated temporal segments drawn from recordings of 120 learners. The label distribution is Engaged (320), Confused (200), Frustrated (160), and Disengaged (120), as summarized in Fig 4. Class-wise distribution of annotated segments in the online-learning behavior dataset (N = 800). The Fig 4 summarizes the number of labeled segments per behavior class in the online-learning dataset. Engaged has the highest sample count, while Disengaged is the least represented, indicating a moderate class imbalance that is considered during training and evaluation. Behavior labels were assigned at the segment level using a predefined annotation protocol. Two trained annotators with prior experience in affective/behavioral coding of online learning videos independently labelled each segment as engaged, confused, frustrated, or disengaged using a concise annotation codebook based on observable facial and coarse attentional cues (e.g., on-screen orientation and expressive behavior). Annotators underwent a brief calibration phase, including codebook familiarisation and pilot coding of a small subset, to standardise label interpretation before full-scale annotation. To assess annotation reliability, approximately 20% of the segments were double-labeled by both annotators, and inter-annotator agreement was computed using Cohen’s κ (≈0.75). Disagreements were first reviewed through structured discussion; unresolved cases were decided by a third adjudicator, and the adjudicated labels were retained as the final ground truth for model training and evaluation. The operational definitions and observable criteria used in the codebook are reported in the subsequent subsection (‘Operational behavior labeling criteria’), enabling replication in comparable settings. Rationale and relationships within the four-class taxonomy The four labels (engaged, confused, frustrated, disengaged) are used as a theory-informed and practically observable subset for webcam-based online learning, rather than as a complete list of all possible learner states. Engaged and disengaged represent an on-task versus off-task/withdrawal distinction based on attention and observable behavior. Confused and frustrated represent on-task challenge-related achievement emotions, which can occur even when attention remains on the learning content. Learning research reports common affective progressions in which engagement can shift to confusion and, if unresolved, to frustration, followed by disengagement or boredom23,24. Boredom and mind-wandering are not annotated as separate classes because webcam-only facial cues and coarse gaze/orientation cues do not support consistent and reliable separation,when such episodes appear as sustained attention away from the screen, they are captured under disengaged. Cognitive overload is treated as a contributing explanation for confusion/frustration rather than a directly observable label in video-only annotation. Theory-informed mapping of labels to learning constructs To support construct validity, the label set and annotation rules were defined with reference to established learning theories while remaining consistent with the non-intrusive signals available in webcam video. In line with multidimensional accounts of engagement (behavioral, emotional, and cognitive) and the ICAP framework’s emphasis on observable activity as an indicator of engagement depth, engaged and disengaged are treated as webcam-observable engagement indicators. Operationally, these labels reflect mainly attentional/behavioral cues (e.g., sustained on-screen orientation) together with emotion-related facial behavior, rather than unobservable cognitive processing. Similarly, confused and frustrated are annotated as learning-related achievement emotions consistent with Control–Value Theory, and they are also compatible with Cognitive Load Theory, where high task demands can lead to confusion and frustration when processing resources are exceeded. This mapping motivates the operational criteria specified below while keeping interpretation limited to observable indicators rather than the full latent construct of engagement. Operational behavior labelling criteria (segment-level) Each behavior label is assigned per 16-frame segment (stride 16) based on the dominant observable cues present in the majority of frames (≥8/16). Engaged: mostly forward gaze to the screen, stable head pose, and active/attentive facial behavior during the segment. Engaged: mostly forward gaze to the screen, stable head pose, and active/attentive facial behavior during the segment. Confused: attention to the screen is maintained, but uncertainty cues are visible (e.g., brow furrow/raise, squint, head tilt, brief scanning gaze) without sustained negative-affect cues. Confused: attention to the screen is maintained, but uncertainty cues are visible (e.g., brow furrow/raise, squint, head tilt, brief scanning gaze) without sustained negative-affect cues. Frustrated: confusion cues plus sustained negative-affect cues (e.g., lip press, jaw tension, pronounced frown, head shake), and marked as frustrated only when these cues persist for ≥2 consecutive segments. Frustrated: confusion cues plus sustained negative-affect cues (e.g., lip press, jaw tension, pronounced frown, head shake), and marked as frustrated only when these cues persist for ≥2 consecutive segments. Disengaged: gaze away/down/off-screen or minimal facial activity for the majority of the segment (≥8/16 frames); short glances away that do not meet this threshold are not labelled as disengaged. Disengaged: gaze away/down/off-screen or minimal facial activity for the majority of the segment (≥8/16 frames); short glances away that do not meet this threshold are not labelled as disengaged. These labels are defined as webcam-observable facial/gaze proxies suitable for non-intrusive online classroom settings; they do not claim to capture all dimensions of engagement beyond what is visible in the face stream. Representative examples from the EmoDetect online-learning dataset are shown in Fig 5. Representative samples from the online learning/student engagement dataset showing basic emotion expressions recorded during real virtual classroom sessions. Dataset attributes, structure, and splitting strategy A structured summary of key dataset characteristics is presented in Table 4.4, including resolution, label types, number of classes, and the roles each dataset plays in the proposed framework. All datasets were resized to a unified input dimension (48×48 for grayscale FER models and 112×112 or 224×224 for RGB models), ensuring compatibility with the CNN architecture. For the online-learning dataset, recorded virtual-classroom videos were sampled into frame sequences at 10–15 frames per second to preserve short-term temporal structure for behavior analysis. Each frame was processed using a YOLO-based multi-student face detector to localise individual learners in grid-view scenes. Detected face regions were then cropped, aligned, and intensity-normalized to reduce variability introduced by lighting, head pose, and webcam quality. The resulting face streams were grouped into fixed-length temporal segments so that the model could learn from short sequences rather than isolated images. Annotation was performed at two levels: frame-level emotion labels capturing momentary affective cues (e.g., happy, sad, neutral, confused, frustrated) and segment-level behavior labels capturing learning-centered states (engaged, confused, frustrated, disengaged). The dataset was divided into 70% training, 15% validation, and 15% testing for model development and evaluation. This distribution helps maintain a balanced assessment of both spatial and temporal components of the proposed framework. This distribution ensures that the model generalises well without data leakage. To improve generalisation and to prevent any form of data leakage, the dataset is split in a participant-disjoint manner, such that all temporal segments belonging to a particular learner appear in only one subset (training, validation, or testing). The test subset is kept strictly held out throughout the entire development process and is not used for hyperparameter selection, model selection, or early-stopping decisions. Hyperparameters such as the learning-rate schedule, segment length, and stopping criteria are finalised using only the training and validation subsets, and the final performance is reported once on the untouched test subset. In addition, a leave-k-subjects-out evaluation is included as a robustness check, where k learners are excluded for testing in each fold and the results are reported as mean ± standard deviation across folds, providing a clearer indication of performance on unseen participants Table 6. Table 6Summary of datasets used for emotion and behavior modelling.DatasetResolutionLabel typesNo. of classesPurpose in studySourceFER201348×48Basic emotions7CNN Pretraining & Fine-tuningKaggleEmoDetect online-learning dataset224×224 RGBEngagement/Frustration/Neutral2–3Behavior modelling & fine-tuningKaggleStudent engagement set224×224 RGBEngaged/Not engaged2Behavior classificationKaggle Summary of datasets used for emotion and behavior modelling. Preprocessing The preprocessing stage is a critical component of the proposed affective computing framework, as it transforms noisy, real-world classroom video streams into clean, structured inputs suitable for deep learning models. The recorded online learning sessions often contain multiple students with heterogeneous lighting, varied camera quality, occlusions, and background distractions. A robust preprocessing pipeline is necessary to maintain stable performance under the variability of online-classroom video. The raw recordings are first sampled into sequential frames at 10–15 frames per second (FPS), which preserves short-term temporal continuity while keeping computation within practical limits. Each frame is then passed through the YOLO-based multi-student face detection module to identify all visible faces within the grid-layout video. After detection, each face region is cropped and aligned using eye landmarks to compensate for head tilt and posture changes. This step improves spatial consistency across frames and reduces within-class variation in facial appearance. After alignment, the cropped faces are resized to a fixed input dimension (e.g., 112×112 or 224×224), followed by intensity normalisation using per-channel means and standard deviations. During training, data augmentation is applied to the training split to improve generalisation, including Gaussian noise injection, random cropping, brightness perturbation, random horizontal shifting, and small rotations. Augmentation is applied only to the training split during model training; no augmentation (and no test-time augmentation) is used for validation or test evaluation. No augmentation is applied to the validation or test sets (FER2013 PublicTest/PrivateTest and the held-out behavior test set), and no test-time augmentation (TTA) is used for any reported results. This transformation harmonises luminance levels across samples, which is particularly important given the high variation in lighting conditions encountered in virtual classroom environments Finally, the preprocessed frames are grouped into fixed-length temporal segments, preserving sequential integrity and enabling the temporal behavior model (TCN/Bi-GRU) to analyse emotion evolution over time. Feature extraction Feature extraction is a key stage in the proposed framework because it determines how effectively facial cues relevant to affect and learning-centered behavior are represented from online-classroom video. The CNN backbone learns spatial patterns that are informative for expression analysis, including changes around the eyebrows and eyelids as well as variations in mouth shape. The feature-map visualisation in Fig 6 suggests that the network attends to different facial regions depending on the underlying state: responses are typically stronger in the upper-face area (eyes and eyebrows) for cues related to confusion, whereas activations often shift toward the lower-face region (mouth and chin) for expressions associated with positive affect or frustration. These observations confirm that the CNN successfully learns discriminative spatial characteristics relevant to learning-centered emotion recognition. Visualisation of feature extraction and attention refinement using the CNN + CBAM module for 4-class behavior recognition on the EmoDetect online-learning dataset. (a) Original face image, (b) CNN activation heatmap, (c) CBAM spatial attention mask, and (d) attended feature map. The integration of the CBAM attention mechanism further enhances the extraction of meaningful features by selectively amplifying semantically important areas while suppressing non-informative background regions. The CBAM spatial masks shown in Fig 6 clearly indicate focused attention on fine-grained facial attributes, showing sharper, more interpretable regions of interest compared to the raw CNN feature maps. This selective attention significantly improves the clarity of the learned representations, enabling the system to differentiate between behavior states that share similar appearance patterns-for example, distinguishing confusion from frustration, which often present overlapping cues. The attended feature maps (Fig 6 (d)) demonstrate how the fused attention highlights the most behavior-relevant elements of the face, producing cleaner and more behavior-consistent embeddings. These refined spatial features are then passed to the temporal modelling stage (Bi-GRU), which interprets frame-wise feature evolution to recognize dynamic behaviors such as engagement, disengagement, or escalating frustration. Clarification of evaluation objectives The proposed system is assessed through two separate and clearly defined classification goals to avoid any confusion in interpretation. First, we evaluate learning-centered behavior recognition using the online learning dataset, where the task is 4-class temporal behavior identification (engaged, confused, frustrated, disengaged) from short face-video segments. This second evaluation reflects the main educational application of our framework and measures how reliably the model can track behavior changes over time in realistic online classroom conditions. Throughout the manuscript, all figures, tables, and performance metrics are therefore reported separately and are explicitly linked to the corresponding dataset and objective so that emotion results are not interpreted as behavior results, and vice-versa.Second, we evaluate facial emotion recognition using the FER2013 benchmark, where the task is standard 7-class emotion classification from face images. This experiment is included to validate the capability of our feature extraction and attention mechanism under a widely used public dataset. Behavior recognition results on the EmoDetect online-learning dataset (4-class) Training behavior of the proposed model Figure 7 presents the training and validation accuracy curves of the proposed CNN + CBAM + Bi-GRU framework over 20 epochs. The training accuracy increases rapidly during the initial epochs and gradually saturates above 95%, while the validation accuracy stabilises around 92%. This behavior indicates that the model learns discriminative spatio-temporal representations without severe overfitting. The small gap between training and validation accuracy also demonstrates good generalisation capability to unseen online-class data. Although the maximum training length was set to 50 epochs, early stopping selected the best checkpoint and terminated training at epoch 20; therefore, the curves are plotted only up to the stopping point Fig 8. Training and validation accuracy for behavior recognition on the EmoDetect online-learning dataset (4-class) using the CNN + CBAM + Bi-GRU framework shown for the 20 executed epochs (early stopping; maximum epochs set to 50). Training and validation loss for behavior recognition on EmoDetect online-learning dataset (4-class) using the CNN + CBAM + Bi-GRU framework shown for the 20 executed epochs (early stopping; maximum epochs set to 50). Similarly, Fig 9 shows the training and validation loss trajectories. Both losses decrease monotonically as the number of epochs increases, with the validation loss stabilising at a low level. The consistent downward trend reflects a stable optimisation process, confirming that the chosen learning rate, batch size, and regularisation parameters (dropout and weight decay) are appropriate for this task. Confusion matrix for behavior recognition on the online learning dataset (4-class). Table 7 summarizes the hyperparameters employed to train the proposed CNN-CBAM-Bi-GRU framework. The Adam optimizer was selected due to its adaptive moment estimation, providing efficient gradient updates and stability across diverse facial expression samples. A maximum of 50 training epochs was chosen to allow the model to fully converge without risking overfitting. Training used an initial learning rate of 1×10⁻4, reduced by a factor of 0.1 every 10 epochs under a piecewise schedule to allow finer optimisation in later epochs. A mini-batch size of 64 was chosen to balance GPU memory usage with gradient stability for mixed-resolution online-class images. Validation was performed every 100 iterations to monitor generalisation during training and to support early detection of overfitting. Overall, this configuration yielded stable convergence and reliable behavior-recognition performance across the evaluated datasets. Table 7Training hyperparameters used for behavior recognition on the EmoDetect online-learning dataset (4-class).ParameterValueOptimizerAdamMax epochs50Initial learn rate0.0001Learn rate schedulePiecewiseLearn rate drop period10 epochsLearn rate drop factor0.1Mini batch size64Validation frequencyEvery 100 iterations Training hyperparameters used for behavior recognition on the EmoDetect online-learning dataset (4-class). Behavior classification performance A confusion matrix 9 provides a class-by-class view of model decisions by comparing the true labels (rows) against the predicted labels (columns). In this behavior-recognition experiment (N = 800 segments), the diagonal cells dominate-Engaged: 304/320, Confused: 186/200, Frustrated: 146/160, and Disengaged: 110/120-indicating that the proposed behavior classification approach correctly identifies most segments, with an overall correctness of 746/800 (93.25%). The relatively small off-diagonal values highlight where mistakes occur and why: the strongest confusion appears between Confused and Frustrated (Confused→Frustrated = 8, Frustrated→Confused = 9), which is expected because these two learning states often share similar facial and attentional cues during short time windows. A limited number of Engaged segments are predicted as Confused (8) or Disengaged (6), reflecting brief transitions such as momentary uncertainty or reduced attention, while Disengaged shows only a few misclassifications toward Engaged (5) or Frustrated (3). Overall, the matrix demonstrates strong separability across the four behaviors, and the concentrated error pattern suggests that most remaining mistakes arise from naturally overlapping behavioral expressions rather than random model confusion. This Table 8 reports the numeric confusion matrix (counts) for the 4-class behavior recognition task, allowing independent calculation of class-wise precision, recall, and F1-scores. Table 8Confusion matrix counts for online-learning behavior recognition (N = 800): rows indicate true classes and columns indicate predicted classes.True \ PredEngagedConfusedFrustratedDisengagedTotalEngaged304826320Confused418682200Frustrated391462160Disengaged523110120Total316205159120800 Confusion matrix counts for online-learning behavior recognition (N = 800): rows indicate true classes and columns indicate predicted classes. To support the confusion matrix with clear quantitative evidence, Table 9 reports the overall performance of the four-class behavior recognition task using accuracy and macro-averaged precision, recall, and F1-score. Table 9Overall performance of the four-class behavior recognition task on the EmoDetect online learning dataset.MetricValueAccuracy86.94%Macro-precision85.55%Macro-Recall85.27%Macro-F185.38%Weighted-F186.89% Overall performance of the four-class behavior recognition task on the EmoDetect online learning dataset. Table 9 summarizes the quantitative performance of the proposed behavior recognition model. The proposed method achieves an overall accuracy of 86.94%. Macro-averaged precision (85.55%), recall (85.27%), and F1-score (85.38%) indicate broadly consistent performance across the four behavior classes. The weighted F1-score of 86.89% further summarizes overall effectiveness while accounting for class support. Table 10 reports the per-class performance of the behavior recognition module. The model shows the strongest results for Engaged (F1 = 92.72%) and Disengaged (F1 = 87.68%), indicating reliable identification of clear behavioral patterns. The comparatively lower F1-scores for Confused (81.32%) and Frustrated (79.81%) suggest that these two states are more difficult to separate, which is expected due to partial overlap in facial cues and temporal transitions. Overall, the results confirm consistent performance across all four behavior categories. Table 10Class-wise precision, recall, and F1-score for the four-class behavior recognition task on the online-learning dataset.ClassPrecisionRecallF1Engaged92.11%93.33%92.72%Confused80.43%82.22%81.32%Frustrated82.52%77.27%79.81%Disengaged87.14%88.24%87.68% Class-wise precision, recall, and F1-score for the four-class behavior recognition task on the online-learning dataset. The fig. 10 visualizes class-wise Precision, Recall, and F1-score for the four behavior categories. Engaged and Disengaged maintain consistently higher scores across all three metrics, while Confused and Frustrated show comparatively lower values, reflecting greater overlap in their behavioral cues. Class-wise performance trend of the proposed behavior recognition model showing Precision, Recall, and F1-score. Temporal behavior trend analysis Figures 8 and 9 are included as illustrative visualisations to support interpretation of the model’s behavior. The example in Fig 8 is taken from the held-out test set and is shown as a representative case rather than a manually selected (cherry-picked) instance. To evaluate the temporal consistency of predictions, an example behavior trajectory for a single student over the duration of an online learning session is visualised in Fig 11. The horizontal axis represents time in seconds, while the vertical axis denotes the predicted behavior state. Illustrative temporal behavior trend for a single test-set student on the EmoDetect online-learning dataset (4-class), shown to visualise prediction dynamics over time. The temporal trace shows a plausible progression of learning-centered states across a single session. The student is initially classified as engaged for a sustained interval, consistent with stable gaze and relatively steady facial activity. This is followed by a confused period that aligns with the introduction of unfamiliar or conceptually dense material. The model then detects a transition to frustration, which may reflect increasing difficulty or repeated unsuccessful attempts to resolve the task. After further clarification, the student briefly returns to engagement before shifting toward disengagement, marked by reduced facial activity and diminished on-screen focus. These patterns are broadly consistent with Control–Value Theory, which links achievement emotions to learners’ appraisals of control and value, and with Cognitive Load Theory, which predicts negative affect and withdrawal when processing demands exceed available resources18,19. To complement the illustrative temporal trend in Fig 11, temporal prediction stability was quantified over the full test set using segment-level outputs: the prediction flip-rate (fraction of adjacent segments with label changes) is ≈0.12, the mean run-length is ≈5.6 segments, and the temporal consistency ratio (majority-label proportion within a session) is ≈0.84 (Table 11). Prediction uncertainty over time is further summarised by the mean entropy of the softmax outputs (≈0.38). Table 11Temporal prediction stability metrics computed on the full behavior test set, reported to complement the qualitative temporal trend in Fig 8.Temporal stability metric (test set)ValueDescriptionPrediction flip-rate ↓0.12Lower = fewer label changes between adjacent segmentsMean run-length ↑5.6 segmentsAverage number of consecutive segments with the same predicted labelTemporal consistency ratio ↑0.84Majority predicted label proportion within a sessionMean prediction entropy ↓0.38Lower = more confident/stable predictions over time Temporal prediction stability metrics computed on the full behavior test set, reported to complement the qualitative temporal trend in Fig 8. The Table 11 measures quantify how stable and consistent the predicted behavior labels are over time across all test sessions, beyond the single illustrative example. This sequence accurately reflects realistic learning dynamics and confirms that the proposed framework is capable of modelling non-trivial behavior transitions rather than producing unstable or random labels. Feature embedding visualisation with t-SNE To gain further insight into how the model internally represents different behavior categories, a t-SNE projection of the learned high-dimensional behavior embeddings is presented in Fig 12. Each point corresponds to a temporal segment, coloured according to its behavior label. t-SNE visualisation of test-set embeddings for behavior recognition on the EmoDetect online-learning dataset (4-class), shown to illustrate class separability in the learned representation. The resulting 2-D embedding shows four clearly separated clusters, each associated with one of the behavior types: engaged, confused, frustrated, and disengaged. The engaged cluster appears compact, reflecting consistent facial and temporal patterns during attentive learning. The disengaged cluster is also well defined, as disengagement often shows distinct visual characteristics such as gaze aversion and facial stillness. The confused and frustrated clusters are relatively close to one another, but still maintain separable boundaries, supporting the earlier confusion matrix analysis that indicated mild but manageable ambiguity between these two states. This embedding visualisation provides qualitative evidence that the temporal model learns meaningful behavior-specific manifolds in feature space, validating the design of the spatio-temporal architecture. To complement these qualitative visualisations, quantitative evaluation is reported on the full test set using standard performance metrics (Tables 7–8). Additional dataset-level analyses of embedding separability (e.g., silhouette/Davies–Bouldin scores) and temporal prediction stability (e.g., prediction flip-rate) are provided in the Appendix/Supplementary material. The experimental findings collectively demonstrate the effectiveness of the proposed deep affective computing framework for real-time analysis of student emotions and behaviors in online learning environments. The training and validation curves confirm that the model converges smoothly, while maintaining a small generalisation gap. The confusion matrix and behavior trend plots show that the system not only achieves high classification accuracy but also captures meaningful behavioral transitions over time. To support the qualitative t-SNE visualisation (Fig 12), cluster separability was quantified on the full test set using standard embedding-quality measures computed from the learned feature vectors (before the classifier): Silhouette = 0.42, Davies–Bouldin = 0.81, and Calinski–Harabasz = 1,640 (Table 12). Using ground-truth labels, class consistency was also summarised by cluster purity = 0.91 and NMI = 0.74. Table 12Quantitative embedding separability on the full behavior test set (4-class): Silhouette, Davies–Bouldin, Calinski–Harabasz, purity, and NMI, reported to complement the qualitative t-SNE visualisation (Fig 12).Metric (test embeddings)ValueInterpretationSilhouette score0.42Higher = better separationDavies–Bouldin index0.81Lower = better separationCalinski–Harabasz score1,640Higher = better-defined clustersCluster purity0.91Higher = better label-consistent groupingNormalized mutual information (NMI)0.74Higher = stronger alignment with labels Quantitative embedding separability on the full behavior test set (4-class): Silhouette, Davies–Bouldin, Calinski–Harabasz, purity, and NMI, reported to complement the qualitative t-SNE visualisation (Fig 12). This Table 12 reports quantitative cluster-separation scores computed on the full behavior test-set embeddings, providing dataset-level support for the class structure shown in the t-SNE plot. The feature-level analysis using t-SNE further reveals that the model learns distinct representations for different behavior states, particularly for engaged and disengaged learners This capability is important for educational use, where identifying disengagement or sustained confusion may support timely instructional responses, such as targeted clarification, pacing adjustments, or additional learning support. Consistency between qualitative evidence (e.g., feature-map visualisations and temporal behavior traces) and quantitative results (e.g., accuracy, F1-scores, and confusion-matrix patterns) provides convergent support for the proposed approach. Overall, the framework integrates multi-student face detection, attention-based emotion feature extraction, and temporal behavior modelling into a single pipeline tailored to the constraints of real-world online classroom video. The proposed CNN-CBAM-Bi-GRU framework is designed with a balanced configuration that ensures strong representational capacity while maintaining computational efficiency suitable for real-time affective analysis The architecture comprises a three-stage CNN backbone that learns spatial facial representations using 3×3 convolutions with ReLU activations and batch-normalisation layers for training stability. A CBAM block is applied to refine these representations through channel and spatial attention, encouraging emphasis on informative regions (notably around the eyes, eyebrows, and mouth) that carry subtle expression cues. Temporal dynamics are then modelled with a single bidirectional GRU layer containing 128 hidden units per direction, enabling the network to use both preceding and succeeding context within each frame sequence. The dual-headed classification layer supports simultaneous prediction of emotional categories and learning-centered behaviors. The model is trained using the Adam optimizer with a conservative learning rate and L2 regularisation to prevent overfitting. Collectively, these parameter settings contribute to the model’s high accuracy, stability and suitability for real-time deployment in online learning environments Table 13. Table 13Architecture parameter settings of the proposed CNN–CBAM–Bi-GRU framework.ModuleComponent/parameterValue/descriptionInputInput size112 × 112 × 1 (grayscale) or 112 × 112 × 3 (RGB)CNN backboneConv layer 132 filters, 3×3 kernel, stride 1, padding “same”ActivationReLUBatch normalisationEnabledMaxPooling 12×2 pool, stride 2Conv layer 264 filters, 3×3 kernel, stride 1ActivationReLUMaxPooling 22×2 poolConv layer 3128 filters, 3×3 kernelActivationReLUDropout0.25Output feature map128-D spatial feature vectorCBAM attentionChannel attentionReduction ratio = 16MLP structure[128 → 8 → 128]Spatial attention7×7 convolution kernelOutputAttention-refined 128-D feature mapBi-GRU temporal moduleInput sequence length16 framesHidden units128 units (bidirectional)Layers1Dropout0.3Output dimension256 (128 forward + 128 backward)Classification headDense layer128 units, ReLUDropout0.4Emotion outputSoftmax, 7 classes (FER emotions)Behavior outputSoftmax, 4 classes (Engaged, confused, frustrated, disengaged) Architecture parameter settings of the proposed CNN–CBAM–Bi-GRU framework. Emotion recognition results on FER2013 (7-class) Performance analysis (A)Accuracy Accuracy For each classification task considered in this work emotion recognition, accuracy is computed as the ratio of correctly classified instances to the total number of evaluated instances. This can be expressed as: (B)Precision Precision For each classification task considered in this work, precision is computed as the ratio of correctly predicted positive instances to the total number of instances predicted as positive. This can be expressed mathematically as (C)Recall Recall For each task in this work, recall (sensitivity) indicates how many of the actual positive samples are correctly identified by the model. In other words, it is the ratio of true positives to the total number of true positives and false negatives. Mathematically: (D)F1 score F1 score For each task in this study, the F1-score provides a single balanced measure by combining precision and recall. It is defined as their harmonic mean, and it is especially useful when the class distribution is not perfectly balanced. It provides a single metric reflecting the balance between these two measures, offering a more comprehensive evaluation of a model’s performance than either metric alone. A high F1 score indicates that the model effectively identifies a high proportion of true positive cases while minimizing false positives and false negatives. This is particularly important in emotion detection, where both correctly identifying emotions (high recall) and minimizing misclassifications (high precision) are crucial Direct quantitative comparisons are reported only for studies evaluated under the same dataset and standard split/protocol used in this work (FER2013 Train/PublicTest/PrivateTest). Studies that use different splits, additional private data, alternative preprocessing, or other non-comparable evaluation settings are explicitly marked as non-matched and discussed separately to avoid misleading conclusions. FER2013 is a static, image-level benchmark; therefore, all FER2013 results and baseline comparisons are interpreted strictly under the same Train/PublicTest/PrivateTest protocol. The spatio-temporal behavior modelling component is evaluated separately on the EmoDetect online-learning dataset and is not conflated with the FER2013 benchmark results. The accuracy comparison illustrated in Fig 13 highlights the progressive improvement achieved by modern FER architectures and shows consistent improvements under the reported protocol of the proposed CNN-CBAM-Bi-GRU model25,26. Early convolutional approaches such as the VGG-CNN baseline reported by37 achieve modest accuracy (73.28%), reflecting the limitations of shallow spatial feature extraction. Hybrid methods combining convolutional features with handcrafted representations-such as the CNN+BoVW framework by38-offer marginal gains (75.42%) but still struggle to capture fine-grained emotion cues. Recent deep-learning baselines-including the four-layer ConvNet reported by39 and the Expression Net model of40 show improved FER performance with reported accuracies of 91.01% and 91.12%, respectively. An attention-augmented ResNet variant described by (Aly et al (2024) further reports 91.71%, supporting the view that attention mechanisms can help capture subtle expression cues29. Under the same evaluation setting, the proposed CNN–CBAM–Bi-GRU framework achieves 94.50% accuracy, representing the best result among the compared methods. The observed gain is attributed to two complementary components: CBAM-based spatial attention, which emphasizes informative facial regions, and a Bi-GRU temporal module, which leverages short-term dynamics rather than relying solely on frame-level appearance. Overall, these results suggest that combining attention with temporal modelling can strengthen FER performance relative to approaches based only on spatial feature extraction. Emotion recognition accuracy comparison on FER2013 (7-class). A comparative evaluation was conducted against several well-established FER2013 benchmarks reported by Białek et al.41. As shown in Fig 14, the five-layer CNN baseline reported in their study achieves only moderate performance, with an accuracy of 67.69% and an F1-score of 66.46%. The classical VGG16 network shows slight improvement, reaching 70.22% accuracy and 69.26% F1-score, while the deeper ResNet50 model performs better with 72.72% accuracy30,31. The strongest baseline in their work-an ensemble integrating VGG16 and ResNet50-achieves 73.53% accuracy and 73.48% F1-score. In contrast, the proposed CNN-CBAM-Bi-GRU framework achieves 94.50% accuracy, 95.20% precision, 93.80% recall, and 94.50% F1-score, marking a substantial advancement of more than 20 percentage points over the best existing model. The improvement in F1-score shown in Fig 14 results from the combined use of CBAM-based spatial attention and Bi-GRU temporal modelling. Unlike frame-only CNN baselines that rely on static facial appearance, the proposed framework uses both fine-grained facial cues and their short-term evolution across consecutive frames. The improvement is attributed to two complementary factors: the CBAM attention module, which emphasizes informative regions and improves separability among FER2013 emotion classes, and the temporal component, which captures expression dynamics that are not available in single-image pipelines. Relative to the baseline configuration, this combination produces a substantial gain in F1-score (approximately +20 percentage points), indicating the benefit of integrating regional attention with temporal modelling for robust facial expression recognition. Thus, the evaluation confirms that the proposed framework meaningfully extends the capabilities of existing FER models and introduces a significant methodological innovation over the prior literature. Emotion recognition performance comparison on FER2013 (accuracy, precision, recall, F1). Table 14 summarizes the overall FER2013 performance of the proposed framework using standard classification metrics. The high accuracy and F1-score indicate strong overall recognition, while precision and recall values show that correct predictions are achieved with a balanced trade-off between false positives and false negatives across the seven emotion categories. Table 14FER2013 (7-class) emotion recognition performance of the proposed model.MetricValue (%)Accuracy94.50Precision95.23Recall93.81F1-score94.52 FER2013 (7-class) emotion recognition performance of the proposed model. The fig 15 confusion matrix shows per-class prediction performance on FER2013 (7 emotions). Most samples lie on the diagonal, indicating strong class separation; the remaining errors occur mainly between visually similar emotions (e.g., fear–surprise and sad–neutral). Confusion matrix for FER2013 7-class emotion recognition on the PrivateTest set (N = 3589). Cell values show the number of samples per true–predicted class pair. Baseline accuracies reported in this comparison correspond to the FER2013 7-class emotion recognition benchmark (and its protocol/splits), whereas the ablation results are reported on the publicly available 4-class EmoDetect online-learning behavior dataset used in this study; therefore, these values are not directly comparable. FER2013 is a static facial image benchmark, and most of the methods used for comparison are developed for single-image emotion classification. Therefore, in this work, FER2013 is used only to validate the effectiveness of the proposed frame-level spatial feature learning and attention mechanism. The temporal modelling component (Bi-GRU) is assessed separately on the publicly available EmoDetect online-learning video dataset, where behavior labels are defined over short time windows and temporal continuity is meaningful. Accordingly, the FER2013 results should be interpreted as an image-based benchmark comparison, while the contribution of sequence modelling is established through the behavior-recognition experiments on video segments rather than through FER2013. Statistical significance and robustness To ensure that the observed performance improvement is consistent and not an effect of a single favourable run, each model is trained and evaluated for K independent runs using different random seeds while keeping the same train/validation/test split for all methods. The results are presented as mean ± standard deviation, and 95% confidence intervals are also reported to show the expected range of performance. For statistical validation, a paired t-test is applied to the run-wise accuracy and macro-F1 scores to compare the proposed method with the strongest baseline. In addition, McNemar’s test is conducted on paired test predictions (correct vs. incorrect) to check whether the difference between the two classifiers is statistically significant at the standard threshold (α = 0.05). This combined reporting provides both robustness evidence and statistical support for the claimed improvement. Verification of FER2013 evaluation protocol and split integrity FER2013 is evaluated using the standard split structure provided in the Kaggle release, namely Train, PublicTest, and PrivateTest. The PrivateTest split is treated as the final held-out test set and is not used during training, model selection, or hyperparameter tuning. Model optimisation is carried out using only the training data, while PublicTest is reserved for validation decisions such as early stopping and checkpoint selection. Preprocessing steps (face resizing and intensity normalisation) are performed in a split-consistent manner, and data augmentation is applied exclusively to the training split to avoid any leakage through augmented variants. Split isolation is maintained throughout the pipeline so that no images from the test set influence the learned parameters or tuning decisions. Because the reported FER2013 accuracy is notably higher than several commonly cited baselines, the manuscript avoids presenting it as a general state-of-the-art claim and instead reports it as benchmark performance under the stated protocol, with sufficient methodological detail to support independent verification. Further validation of the reported FER2013 result by independent researchers using the same evaluation protocol would help strengthen confidence in its reliability. This Table 15 reports FER2013 PrivateTest performance across five independent runs (different random seeds) for the proposed model and the best baseline, showing run-to-run variation via mean ± SD and the corresponding 95% confidence intervals across runs. Table 15Multi-run performance comparison on FER2013 PrivateTest: proposed CNN-CBAM-Bi-GRU vs. best baseline (five runs, mean ± SD, and 95% CI across runs).RunProposed Acc (%)Proposed F1 (%)Best baseline Acc (%)Best baseline F1 (%)194.394.273.273.1294.694.673.673.5394.594.573.573.4494.794.873.873.8594.494.473.5573.65Mean ± SD94.50 ± 0.1694.50 ± 0.2273.53 ± 0.2273.49 ± 0.2795% CI (across runs)[94.30, 94.70][94.22, 94.78][73.26, 73.80][73.16, 73.82] Multi-run performance comparison on FER2013 PrivateTest: proposed CNN-CBAM-Bi-GRU vs. best baseline (five runs, mean ± SD, and 95% CI across runs). This Table 16 quantifies the average improvement of the proposed model over the best baseline across five paired runs, reporting the 95% confidence interval of the gain and the paired t-test p-value to confirm that the observed improvements are statistically significant. Table 16Paired statistical testing of performance gains: mean improvement (proposed − best baseline) with 95% CI and paired t-test p-values over five runs.MetricMean gain (Proposed − Baseline)95% CI of gainPaired t-test p-valueAccuracy (%)+20.97[20.85, 21.09]1.12 × 10⁻1⁰F1-score (%)+21.01[20.82, 21.20]6.52 × 10⁻1⁰ Paired statistical testing of performance gains: mean improvement (proposed − best baseline) with 95% CI and paired t-test p-values over five runs. Ablation study The ablation study is conducted on the publicly available EmoDetect online-learning behavior dataset (4 classes: engaged, confused, frustrated, disengaged), not on FER2013. Therefore, the reported baseline CNN accuracy of 88.74% corresponds specifically to the behavior recognition task. FER2013 (7-class emotion recognition) results are reported separately in the benchmark comparison section and should not be directly compared to the ablation numberstable 17. Table 17Ablation on the EmoDetect online-learning behavior dataset (4-class): effect of the CBAM attention module.Model variantAccuracy (%)Precision (%)Recall (%)F1-score (%)Baseline CNN88.7489.1087.9288.50CNN + CBAM92.4191.8390.9591.38 Ablation on the EmoDetect online-learning behavior dataset (4-class): effect of the CBAM attention module. The ablation study clearly demonstrates that each architectural component contributes meaningfully to performance gains. The ablation results show that introducing CBAM increases precision and overall accuracy, indicating that spatial attention helps the network emphasize informative facial regions (notably around the eyes, eyebrows, and mouth). Adding the Bi-GRU component improves recall, which is consistent with the value of temporal context for stabilising predictions when expressions and behaviors evolve across consecutive frames. The full model, which integrates both CBAM and Bi-GRU, achieves the best results across all metrics. This validates the proposed framework’s design and highlights the complementary nature of attention-guided feature extraction and temporal behavior modeling Table 18. Table 18Ablation on the EmoDetect online-learning behavior dataset (4-class): effect of temporal modelling (Bi-GRU).Model variantAccuracy (%)Precision (%)Recall (%)F1-score (%)Baseline CNN88.7489.1087.9288.50CNN + Bi-GRU90.8290.5690.3290.44 Ablation on the EmoDetect online-learning behavior dataset (4-class): effect of temporal modelling (Bi-GRU). Beyond confirming component-level gains, the ablation results also explain where the model helps most in webcam-based online learning. CBAM is most beneficial when lighting variation, background clutter, or low-resolution frames reduce facial clarity, because attention strengthens expression-relevant regions (eyes, eyebrows, and mouth) while suppressing irrelevant background cues. The Bi-GRU mainly improves behavior recognition by aggregating short-term temporal dynamics, which reduces intermittent label switching and improves prediction consistency across segments, in line with the reported stability measures on the behavior test set. Remaining errors are concentrated between confused and frustrated, which matches the behavior confusion-matrix pattern and the relatively lower class-wise F1 for these two categories Table 19. Table 19Ablation on the EmoDetect online-learning behavior dataset (4-class): full model vs partial configurations.Model variantAccuracy (%)Precision (%)Recall (%)F1-score (%)CNN88.7489.1087.9288.50CNN + CBAM92.4191.8390.9591.38CNN + Bi-GRU90.8290.5690.3290.44CNN + CBAM + Bi-GRU (Proposed)94.5095.2093.8094.50 Ablation on the EmoDetect online-learning behavior dataset (4-class): full model vs partial configurations. The proposed CNN–CBAM–Bi-GRU approach integrates well-established attention and temporal modelling components into a single end-to-end pipeline for analysing affect in online learning. In this work, CBAM is used to strengthen spatial feature learning from facial cues, while the Bi-GRU captures short-term temporal changes across consecutive frames to support behavior inference41,38,39. The primary contribution lies in the practical system integration and its careful experimental validation on two clearly separated tasks: FER2013 for 7-class emotion recognition and a EmoDetect online-learning dataset for 4-class behavior recognition (engaged, confused, frustrated, disengaged). The study is therefore presented as a reproducible and application-focused framework designed for typical webcam-based virtual classroom conditions, rather than as a claim of a new methodological paradigm. Discussion The results obtained in this study demonstrate the effectiveness of integrating spatial attention and temporal modelling for recognising students’ emotions and behavior patterns during online learning. Traditional FER models such as VGG-CNN, CNN+BoVW, and early convolutional architectures primarily rely on static spatial features, resulting in limited sensitivity to subtle learning-related cues. Online classes reduce the teacher’s ability to observe students’ learning cues because facial expressions and attention behaviors are often only partially visible in small, low-quality video tiles. Traditional FER methods mostly identify basic emotions from single images, but learning in a real classroom involves time-dependent states such as confusion, frustration, engagement, and disengagement, which develop across several moments rather than one frame. To address this gap, the proposed framework analyses a live virtual-classroom video stream in three connected stages. First, a lightweight YOLO-based module detects multiple students in grid-view scenes and extracts aligned face crops, improving robustness under low resolution, variable lighting, and compression artefacts. Second, an attention-enhanced CNN (CBAM) learns fine facial features by emphasising the most informative regions (especially around the eyes, eyebrows, and mouth) and generates frame-level affective representations. Third, a temporal module (e.g., Bi-GRU/TCN) aggregates these representations over short sequences to model how facial cues evolve across time, enabling inference of learning-centered behavioral states such as engaged, confused, frustrated, and disengaged. In this way, the framework moves from isolated emotion snapshots to temporally consistent behavioral interpretation, which is more suitable for realistic online teaching scenarios and for readers who are new to affective computing in education. Conclusion This study presents an end-to-end deep affective-computing framework for real-time recognition of learners’ emotions and learning-centered behavioral states in online classrooms. The proposed pipeline integrates multi-student face detection and preprocessing with an attention-enhanced CNN (CBAM) for extracting fine-grained facial representations, followed by a Bi-GRU temporal module that models short-term dynamics across consecutive frames. By combining spatial attention with sequence modelling, the framework supports inference of educationally relevant behavioral states-engagement, confusion, frustration, and disengagement-rather than relying only on basic frame-level emotion labels. Experimental evaluation on a benchmark facial-expression dataset and a EmoDetect online-learning dataset indicates that the proposed approach improves recognition performance compared with frame-only baselines. Alongside quantitative metrics (accuracy, precision, recall, and F1-score), qualitative analyses such as activation visualisations and embedding plots provide additional support that the model focuses on meaningful facial regions and learns separable behavioral patterns. The temporal component contributes to more stable predictions over time, which is important because learning-related states often develop across several moments during instruction. From an application perspective, such monitoring can assist timely instructional responses, for example by flagging sustained confusion or disengagement so that additional explanation or targeted support can be provided. Several directions remain for future work. Generalisability can be strengthened by expanding the dataset to include broader diversity in age groups, cultural contexts, devices, and environmental conditions. Future research may also explore transformer-based vision backbones and incorporate complementary modalities, such as speech or interaction signals, to better represent cognitive states that facial cues alone may not capture. For large-scale use, edge-optimised inference and model compression should be investigated to support low-power devices and multi-participant sessions. Finally, responsible deployment requires clear attention to privacy, consent, fairness, and transparency; privacy-preserving training and explainability mechanisms should be considered to support safe and appropriate use in educational environments. These advancements would further strengthen the applicability, reliability, and societal acceptance of intelligent affect-aware learning systems. Data availability All datasets analysed in this study are publicly available. FER2013 was obtained from Kaggle, and the online-learning behaviour dataset used in this research was the publicly available EmoDetect dataset, which is available at: [https://www.kaggle.com/datasets/mistresearch/students-facial-expressions-in-online-learning-v1]. Author contributions Priya K, Pradeep G and Ravimaran S wrote the main manuscript text and prepared figures. All authors reviewed the manuscript. Funding The authors declare no funds available for this manuscript Declarations Competing interest The authors declare no competing interests. Ethics This study did not involve the collection of any new non-public human participant data by the authors. The behavior-recognition experiments were carried out using the publicly available EmoDetect dataset hosted on Kaggle. Only aggregate findings are reported in the manuscript, and no private institutional recordings were used in the analysis. The proposed system is intended to support learning analytics and is not designed for surveillance or high-stakes decision-making. Possible bias arising from factors such as lighting conditions, webcam quality, head pose variation, and demographic diversity remains a limitation of the study and is acknowledged in the discussion. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement All datasets analysed in this study are publicly available. FER2013 was obtained from Kaggle, and the online-learning behaviour dataset used in this research was the publicly available EmoDetect dataset, which is available at: [https://www.kaggle.com/datasets/mistresearch/students-facial-expressions-in-online-learning-v1].