Impact of simulated glasses noise on facial emotion recognition with deep learning models Authors: Deniz Baharvand, Fatemeh Jodati, Kimia Shirini, Sina Samadi Gharehveran Published: 2026 ABSTRACT Facial emotion recognition (FER) is one of the main fields of research in image processing and artificial intelligence with applications in human–machine interaction, behavior analysis, and intelligent vision. The RAF DB dataset, a widely used benchmark dataset for emotion recognition, is used for the training and testing of deep learning models. In real application conditions, the facial image may be subjected to visual noise. Large glasses that occlude the eye region of the face are one example of environmental or artificial noise which distorts the visual information. If frames covering a large area of an expressive part of the face, like the eyes, are recognized as noise by the emotion recognition network this may lead to noticeable performance degradation. Artificial occlusion noise, similar to large glasses, was applied to the eye region of the facial images from the RAF DB dataset. The impact of this type of structural noise on the performance of deep-learning-based facial emotion recognition models is analyzed. Three state-of-the-art architectures (ResNet-50, Vision Transformer (ViT-Base/16) and Swin Transformer Tiny (Swin T)) are tested to determine their robustness to the noisy application conditions. The objective of the study is to determine and suggest possible methods to counter the decrease in recognition performance caused by the occlusion noise. Experimental results indicate that, within our evaluation setup, transformer-based architectures, especially Swin-T, tend to be more robust to the simulated visual noise, which may be useful for the design of facial emotion recognition models intended for more complex conditions. Introduction Human emotions play a crucial role in interpreting behavioral aspects and are pivotal for various applications in behavioral science and AI research1. In the driving context, a driver’s mental state directly impacts their ability to concentrate, make decisions, and react to road situations in a timely manner2. Negative emotions such as anger, fatigue, or stress can significantly affect an individual’s risk perception and response, increasing the potential for human error and traffic accidents3. As a result, reliable and accurate emotion recognition has been an important research direction for the development of intelligent transportation and safety solutions4–7. Emotion recognition in the real-world scenarios still poses significant challenges8. Partial occlusions of different types are common in driving contexts and include sunglasses, masks, hand gestures, or inconsistent lighting in facial images, concealing important facial regions such as eyes, nose, cheeks, and forehead9. When these key features are occluded, the inference of emotion would be more uncertain10–19. Therefore, the models with noisy inputs should still be applicable to work well, which raises the need for performance evaluation and model improvements in this aspect20–23. Recent years have seen great progress in deep neural network design24. Powerful deep learning architectures have been developed to extract both local and contextual information from images25. The Convolutional Neural Networks (CNNs) and Transformer-based models have shown impressive performance in facial emotion recognition tasks as well as in general visual recognition. Moreover, the integration of statistical feature reduction methods such as Principal Component Analysis (PCA) and Independent Component Analysis (ICA) has been shown to improve model stability and reduce redundancy in high-dimensional data representations, as demonstrated in our previous work on electricity consumption forecasting14. Specifically, ResNet-50, Vision Transformer (ViT-Base/16) ,and Swin Transformer Tiny (Swin T) represent three different paradigms of neural networks that jointly model local convolutional features and long-range global relationships8,26. To simulate the real-world occlusion and bridge the gap with lab-clean conditions27, this paper aims to evaluate and compare the robustness of the three models in an occlusion-resilient testing setting, with a new noisy version of RAF DB dataset, referred to as RAF DB noisy. The partial occlusion of the face is simulated by adding a rectangular area noise in the regions of eyes, nose, cheeks, and a portion of the forehead21,28,29. This specific type of structured noise in rectangular shape is different from the traditional random or pixel-level corruptions and better models the noisy conditions in the real world such as wearing big sunglasses or masks, shadows, or reflections on the face21,30,31. This way, we can simulate a more realistic noisy setting and help drive the research and model improvements towards real-world conditions and needs21. In this noisy setting, a comparative study is conducted to test the resistance to the rectangular occlusion noise among ResNet-50, Vision Transformer (ViT-Base/16), and Swin T. It was found that Transformer-based models, especially Swin T, were more robust and had better feature generalization capabilities under various levels of occlusion noise32,33. The hierarchical attention mechanism in Transformers allowed these models to be more adaptive and focus on the available regions to compensate for the missing facial information34,35. The method proposed in this paper can be extended to other real-world scenarios where visual occlusion and emotion recognition are both common and important. Such scenarios include public surveillance, remote communications, healthcare settings, human–robot interaction21,36, and others where visual information is often impaired. In these contexts, the accurate recognition of emotional information despite the partial occlusion of the face would be critical for safety, empathy, and user experience37. In conclusion, this paper provides a standardized variant for emotion recognition under structured occlusion noise and contributes to the development of robust, context-aware affective recognition systems for imperfect visual conditions in practical applications8,38. The main contributions of this study are summarized as follows: Development of the RAF-DB-Noisy Benchmark: We construct and release a standardized variant of the RAF-DB dataset featuring physiologically inspired structural occlusion (simulating eyewear/glare). Unlike random pixel noise used in prior works, this benchmark specifically targets the eye region to rigorously test model reliance on lower-facial cues.Systematic Architectural Behavior Analysis: Unlike prior works that primarily focus on CNNs or standard Vision Transformers, this study investigates the under-explored potential of Hierarchical Vision Transformers (Swin-T) in noisy FER. We provide a comparative analysis demonstrating how Swin-T’s hierarchical attention effectively bridges the gap between the local focus of CNNs and the global dependency of ViTs, making it uniquely suited for handling structural occlusions.Qualitative Validation via Explainable AI: We employ Grad-CAM visualization to provide visual evidence of robustness, confirming that resilient models successfully shift semantic focus from occluded ocular regions to informative mouth and cheek areas, validating the quantitative improvements. Development of the RAF-DB-Noisy Benchmark: We construct and release a standardized variant of the RAF-DB dataset featuring physiologically inspired structural occlusion (simulating eyewear/glare). Unlike random pixel noise used in prior works, this benchmark specifically targets the eye region to rigorously test model reliance on lower-facial cues. Development of the RAF-DB-Noisy Benchmark: We construct and release a standardized variant of the RAF-DB dataset featuring physiologically inspired structural occlusion (simulating eyewear/glare). Unlike random pixel noise used in prior works, this benchmark specifically targets the eye region to rigorously test model reliance on lower-facial cues. Systematic Architectural Behavior Analysis: Unlike prior works that primarily focus on CNNs or standard Vision Transformers, this study investigates the under-explored potential of Hierarchical Vision Transformers (Swin-T) in noisy FER. We provide a comparative analysis demonstrating how Swin-T’s hierarchical attention effectively bridges the gap between the local focus of CNNs and the global dependency of ViTs, making it uniquely suited for handling structural occlusions. Systematic Architectural Behavior Analysis: Unlike prior works that primarily focus on CNNs or standard Vision Transformers, this study investigates the under-explored potential of Hierarchical Vision Transformers (Swin-T) in noisy FER. We provide a comparative analysis demonstrating how Swin-T’s hierarchical attention effectively bridges the gap between the local focus of CNNs and the global dependency of ViTs, making it uniquely suited for handling structural occlusions. Qualitative Validation via Explainable AI: We employ Grad-CAM visualization to provide visual evidence of robustness, confirming that resilient models successfully shift semantic focus from occluded ocular regions to informative mouth and cheek areas, validating the quantitative improvements. Qualitative Validation via Explainable AI: We employ Grad-CAM visualization to provide visual evidence of robustness, confirming that resilient models successfully shift semantic focus from occluded ocular regions to informative mouth and cheek areas, validating the quantitative improvements. The remainder of this paper is organized as follows. Section 2 reviews relevant studies and benchmark datasets related to facial emotion recognition under noise and occlusion. Section 3 details the overall methodology, implementation procedures, and architectural configurations used in our experiments. Section 4 presents the quantitative results and corresponding performance analyses. Finally, Sect. 5 concludes the paper and outlines future research directions. Related work Deep learning and computer vision have brought significant progress to facial emotion recognition (FER) in recent years37,39. CNN architectures, in particular, show great capability in extracting hierarchical spatial features. Yet, real-world FER faces a major hurdle: visual noise and occlusion20,21,32,40,41. Elements like masks, hair, or glasses hide parts of the face, leading to sharp drops in accuracy. To tackle this, researchers have explored various techniques, including attention mechanisms, patch-based encoding, GNNs, and image reconstruction21,30,39,42–48. More recently, Transformer models have proven effective at capturing global semantic dependencies, helping to maintain performance even when faces are heavily covered49. Eye occlusion from glasses or headsets is especially problematic. Eyes provide critical emotional cues like gaze and muscle contractions, and covering them severely disrupts feature extraction21,30,50–52. While VR studies often compensate for Head-Mounted Displays (HMDs) using 3D modeling53, traditional FER studies show that CNNs struggle heavily against dark lenses and glare21,54–56. Furthermore, Although extensive recent research has focused on lower-face occlusion like medical masks57–59, systematic testing of FER models under controlled, glasses-like eye occlusion is still lacking21,60,61. Some researchers try to bypass these visual limits by building multimodal systems that incorporate physiological sensors (e.g., EEG, ECG, eye-tracking)62–70. These sensor-based methods improve classification stability but require expensive and intrusive hardware. Because of this limitation, our study focuses purely on non-intrusive, image-based models and investigates how they handle extreme simulated visual noise31,37. To mimic the effect of large eyewear, we applied a structured, non-dark rectangular mask over the eye region in the RAF-DB dataset21,71. This setup forces models to rely predominantly on the lower face, providing a rigorous test for their generalization skills72. Instead of evaluating just one model, we benchmark diverse architectures: ResNet-50 for local spatial modeling27, ViT-Base/16 for global self-attention25,33,39,73,74, and Swin-T for hierarchical processing75. Making FER models more resilient to occlusion is essential for practical applications across security, industry, healthcare, and robotics27,76–78. While standard ViTs are already used in FER, the specific capacity of Hierarchical Transformers (like Swin-T) to handle structural noise through shifted-window attention has not yet been systematically benchmarked against standard CNNs and ViTs. Methodology The proposed evaluation framework is designed to systematically assess the robustness of three distinct deep learning architectures—ResNet-50 (augmented with CBAM), Vision Transformer (ViT-Base/16), and Swin Transformer Tiny (Swin-T)—against structured upper-face occlusion25,30,77. As illustrated in the schematic overview in Fig. 1 (which utilizes symbolic royalty-free images for visual demonstration), the methodology follows a unified pipeline: inputs from both clean and noisy datasets undergo identical preprocessing, followed by parallel feature extraction through the selected backbones, and subsequent refinement via shared feature enhancement modules (MultiScaleConv and AttentionFusion) before final classification. This section details the dataset generation, model configurations, and the training strategy employed. Fig. 1The proposed FER evaluation pipeline. Both clean and noisy datasets undergo identical preprocessing before being processed by three parallel backbones (ResNet-50 + CBAM, ViT-Base/16, Swin-T) and shared feature enhancement modules for final classification. Due to copyright restrictions of the original RAF-DB dataset, royalty-free illustrative face images sourced from the internet have been used in this figure to comply with the CC BY license. These images serve solely as symbolic representations of the input data. The proposed FER evaluation pipeline. Both clean and noisy datasets undergo identical preprocessing before being processed by three parallel backbones (ResNet-50 + CBAM, ViT-Base/16, Swin-T) and shared feature enhancement modules for final classification. Due to copyright restrictions of the original RAF-DB dataset, royalty-free illustrative face images sourced from the internet have been used in this figure to comply with the CC BY license. These images serve solely as symbolic representations of the input data. Data We utilize the Real-world Affective Face Database (RAF-DB)27, containing thousands of in-the-wild facial images categorized into seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and neutral)79. Its high diversity in age, gender, ethnicity, and illumination makes it an ideal baseline for evaluating model generalization under realistic conditions80. To ensure compatibility with our baseline architectures (ResNet-50, ViT-B/16, and Swin-T), all images were resized to 224 × 224 pixels. Standard preprocessing steps, including pixel normalization and data augmentation (random rotation, horizontal flipping, brightness adjustment), were applied to mitigate overfitting33,81. Representative samples for each emotion class are shown in Fig. 2. Fig. 2Illustrative sample images representing the seven emotion classes evaluated in our study: (1) Surprise, (2) Fear, (3) Disgust, (4) Happiness, (5) Sadness, (6) Anger, and (7) Neutral. Due to the strict copyright policies of the original RAF-DB dataset, which prohibit publication under a CC BY license, these images are copyright-free, symbolic examples sourced from the internet. They are used solely to visualize the emotion categories and are not part of the original dataset. Illustrative sample images representing the seven emotion classes evaluated in our study: (1) Surprise, (2) Fear, (3) Disgust, (4) Happiness, (5) Sadness, (6) Anger, and (7) Neutral. Due to the strict copyright policies of the original RAF-DB dataset, which prohibit publication under a CC BY license, these images are copyright-free, symbolic examples sourced from the internet. They are used solely to visualize the emotion categories and are not part of the original dataset. Creation of RAF-DB-noisy To systematically evaluate model robustness under partial occlusion, we introduced a modified variant of the dataset, termed RAF-DB Noisy27,80. Clean and noisy sets share identical identity labels and train-test splits. For the noisy variant, an artificial rectangular occlusion is applied to the upper facial region (eyes, eyebrows, and nose bridge)82, combined with a white Gaussian noise component (σ = 50). This degradation mimics real-world visual obstructions like reflective eyewear, strong glare, or medical shields, forcing the models to rely heavily on uncovered expressive cues such as the mouth region81. This setup provides a controlled testbed for assessing feature-dependency without altering the affective semantics of the original images. The generated dataset will be publicly released for future occlusion-resistant emotion recognition research83. Sample images are illustrated in Fig. 3. Fig. 3Illustration of the synthetic occlusion and random noise interference applied during our robustness evaluation. To comply with copyright restrictions, the simulated glasses noise methodology which is identical to the data augmentation process applied to the RAF-DB dataset in our actual experiments is demonstrated here on the symbolic images from Fig. 2. Illustration of the synthetic occlusion and random noise interference applied during our robustness evaluation. To comply with copyright restrictions, the simulated glasses noise methodology which is identical to the data augmentation process applied to the RAF-DB dataset in our actual experiments is demonstrated here on the symbolic images from Fig. 2. Model architectures overview In order to compare robustness performance across different deep learning paradigms, three representative architectures were selected: ResNet-50, Vision Transformer (ViT Base/16), and Swin Transformer Tiny (Swin-T). ResNet-50 represents convolutional neural networks with strong local feature extraction enabled by residual connections25,84. Vision Transformer (ViT Base/16) pioneers global token-based self-attention over image patches, while Swin-T employs a hierarchical architecture with shifted window self-attention to capture local contextual information25,85. ResNet-50 We adopt ResNet-50 as our primary convolutional baseline due to its effective residual learning mechanism, which mitigates gradient vanishing and facilitates hierarchical feature reuse. To enhance its representational power and spatial focus, a Convolutional Block Attention Module (CBAM)25,86,87 is integrated after each residual stage. CBAM’s sequential channel and spatial attention allows the network to dynamically emphasize relevant facial regions while suppressing background distractions or occluded areas88. The architecture consists of four residual blocks with3,4,6 bottleneck units, formulated as1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{x}}_{\mathrm{o}\mathrm{u}\mathrm{t}\:\:}=\mathrm{F}\left({\mathrm{x}}_{\mathrm{i}\mathrm{n}}\right)+\:{\mathrm{x}}_{\mathrm{i}\mathrm{n}}$$\end{document} We fine-tuned an ImageNet-1 K pretrained model, replacing the original head with a two-layer classifier: a 512-unit hidden layer (ReLU, dropout 0.5) and a 7-unit output layer (dropout 0.3)82,89. A uniform partial fine-tuning strategy was applied across all evaluated architectures, freezing lower-level layers and fine-tuning higher-level components on the RAF-DB Noisy dataset. Training was optimized using AdamW (initial learning rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{5\times\:10}^{-5}$$\end{document}, weight decay 0.01) with a ReduceLROnPlateau scheduler. To ensure stability and robustness, we employed label smoothing (0.15), CutMix and MixUp augmentations (50% probability per batch), gradient clipping (max norm 0.3), and early stopping (patience of 8 epochs). The modified ResNet-50 + CBAM demonstrates strong resilience against noisy occlusions75, as visualized in Fig. 4. Fig. 4Illustration of the bottleneck residual block used in ResNet-50. Illustration of the bottleneck residual block used in ResNet-50. The shortcut connection employs a 1 × 1 convolution to match the feature map dimensions before element-wise addition. To evaluate occlusion robustness across different deep learning paradigms, we compare three representative architectures: ResNet-50 (convolutional with local residual connections)13,65, ViT-Base/16 (global token-based self-attention), and Swin-T (hierarchical shifted-window attention). To ensure a strictly fair comparison, all models share a unified training protocol and hyperparameters, using the weighted F1-score as the primary metric for evaluating robustness against upper-facial occlusions13,71. Vision Transformer (ViT-Base/16) The Vision Transformer (ViT-Base/16) replaces traditional convolutions with global self-attention mechanisms. The 224 × 224 input image is divided into non-overlapping 16 × 16 patches, flattened, and linearly projected into a D-dimensional embedding space. Positional encodings are added to retain spatial relationships before processing through 12 transformer encoder blocks (hidden dimension of 768, 12 parallel heads, ≈ 86)84. The core multi-head self-attention (MHSA) computes relationships across all tokens via:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{{\mathrm{Q}\mathrm{K}}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}\right)\mathrm{V}$$\end{document} where Q, K, and V are the query, key, and value matrices, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{k}$$\end{document} is the key dimension. For occlusion-robust FER, this global context is crucial; it allows the model to leverage spatially distant visible regions when key areas like the eyes are obscured90. Initialized with ImageNet weights, ViT-Base/16 was adapted using our progressive fine-tuning pipeline. The architectural workflow is illustrated in Fig. 5. Fig. 5Architecture of the Vision Transformer (ViT Base/16). The input image is divided into non-overlapping 16 × 16 patches, linearly projected into embeddings, and combined with positional encodings before being processed by a series of transformer encoder layers65. Due to copyright restrictions of the original RAF-DB dataset, royalty-free illustrative face images sourced from the internet have been used in this figure to comply with the CC BY license. These images serve solely as symbolic representations of the input data. Architecture of the Vision Transformer (ViT Base/16). The input image is divided into non-overlapping 16 × 16 patches, linearly projected into embeddings, and combined with positional encodings before being processed by a series of transformer encoder layers65. Due to copyright restrictions of the original RAF-DB dataset, royalty-free illustrative face images sourced from the internet have been used in this figure to comply with the CC BY license. These images serve solely as symbolic representations of the input data. Swin Transformer Tiny (Swin-T) Swin Transformer Tiny (Swin-T)3 introduces a shifted window-based self-attention mechanism, offering a localized and hierarchical alternative to the global attention used in conventional Vision Transformers. This design reduces computational complexity to O(n) and progressively merges patches across four stages (from 4 × 4 to 32 × 32 resolutions). This hierarchical structure is particularly advantageous for our task, as it enhances model robustness against localized facial occlusions33. We adopted the lightweight “Tiny” variant (approx. 28 million parameters, ImageNet-1 K pretrained) to balance accuracy with computational efficiency4. Following our unified adaptation strategy, the original classification head was replaced with a custom two-layer feedforward network: a 512-unit hidden layer (GELU activation, dropout 0.5) and a 7-unit output layer (dropout 0.3)91. Consistent with the overall methodology, partial fine-tuning and standardized augmentations5,92 were applied to adapt the network to the noise patterns of the RAF-DB dataset93. The overall methodology for Swin-T is illustrated in Fig. 6. Fig. 6Swin-T architecture for facial emotion recognition. Due to copyright restrictions of the original RAF-DB dataset, royalty-free illustrative face images sourced from the internet have been used in this figure to comply with the CC BY license. These images serve solely as symbolic representations of the input data. Swin-T architecture for facial emotion recognition. Due to copyright restrictions of the original RAF-DB dataset, royalty-free illustrative face images sourced from the internet have been used in this figure to comply with the CC BY license. These images serve solely as symbolic representations of the input data. The proposed network employs the Swin Transformer Tiny (Swin-T), a hierarchical architecture utilizing shifted-window self-attention to capture local and global dependencies efficiently. As illustrated in Fig. 6, the 224 × 224 × 3 input image is partitioned into non-overlapping patches and sequentially processed through four stages. Each stage progressively reduces spatial resolution while expanding the channel dimensions (from 96 to 768) and the number of attention heads (3, 6, 12, and 24). The core computational mechanisms of our implementation are mathematically formulated as follows:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{\mathrm{Q}{\mathrm{K}}^{\mathrm{T}}}{\sqrt{\mathrm{d}}}\right)\mathrm{V}$$\end{document}4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{X}}_{{\mathrm{w}}} = {\mathrm{Shift}}({\mathrm{X}}_{{\mathrm{w}}} ,{\mathrm{s}})$$\end{document}5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{x}}^{(\mathrm{l}+1)}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\mathrm{X}}_{2\mathrm{i},2\mathrm{j}},{\mathrm{X}}_{2\mathrm{i},2\mathrm{j}+1,}{\mathrm{X}}_{2\mathrm{i}+\mathrm{1,2}\mathrm{j},}{\mathrm{X}}_{2\mathrm{i}+\mathrm{1,2}\mathrm{j}+1,})\mathrm{W}$$\end{document}6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{y}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_{\mathrm{c}}\mathrm{h}\right)$$\end{document} Equations (3) and (4) represent the standard multi-head self-attention and the cyclic spatial shift operation, respectively. Here, Q, K, and V are the query, key, and value matrices, d is the embedding dimension, and Shift(⋅,s) enables cross-window information exchange with a shift size s = M/2. Equation (5) defines the hierarchical patch-merging mechanism, where l denotes the layer index, (i, j) are the spatial window coordinates, and W is the learnable linear projection weight matrix. Finally, Eq. (6) formulates the classification layer, which maps the global average-pooled feature vector h (extracted from the final stage) to the seven facial emotion categories using the learnable weight matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}_{c}$$\end{document}. This configuration achieves an optimal balance between recognition accuracy and computational efficiency, ensuring robust performance on noisy datasets such as RAF-DB. Emotion detection To enhance robustness against partial facial occlusions, our emotion detection pipeline incorporates two shared auxiliary modules: MultiScaleConv and AttentionFusion. After extracting features from the respective backbones (ResNet-50 + CBAM, ViT-B/16, and Swin-T), the MultiScaleConv module employs parallel convolutional branches to capture both fine-grained local details and broader contextual cues. Subsequently, the AttentionFusion module adaptively reweights these multi-scale features, emphasizing salient facial regions while suppressing occlusion artifacts. These modules are introduced before global average pooling, yielding a unified tensor of size [32,510,14,14] across all models84. This shared implementation avoids architecture-specific customizations and ensures a fair comparison of robustness under occlusion noise (noise_std = 50)19,78. Global average pooling then compresses the spatial dimensions into compact feature vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{f}\in\:{R}^{D}$$\end{document}, where D = 2048 for ResNet-50 and D = 768 for both ViT-B/16 and Swin-T. These representations are fed into a classification head—comprising a fully connected layer, batch normalization, dropout (p = 0.2), and a Softmax activation—to map features into the seven emotion categories of the RAF-DB dataset. The final emotion prediction \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{y}_{i}$$\end{document} is computed as:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{y}}_{\mathrm{i}}=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\right({\mathrm{z}}_{\mathrm{i}}\left)\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Z}_{i}$$\end{document} denotes the logit score for the i-th emotion class. Fig. 7Overview of the three backbone architectures (ViT-B/16, Swin-T, and ResNet-50 + CBAM) and their evaluation process in the proposed facial emotion recognition framework. Overview of the three backbone architectures (ViT-B/16, Swin-T, and ResNet-50 + CBAM) and their evaluation process in the proposed facial emotion recognition framework. As illustrated in Fig. 7, the three backbones capture complementary visual features: ViT-B/16 leverages global self-attention, Swin-T utilizes hierarchical shifted window attention, and ResNet-50 + CBAM extracts fine-grained textural cues. Performance is evaluated using accuracy and the weighted F1-score to assess overall and class-balanced robustness. All models are optimized end-to-end using cross-entropy loss with a uniform label smoothing factor of 0.15 and the AdamW optimizer. To achieve stable representation learning and robustness, we adopt a three-stage fine-tuning strategy: Stage 1: Initialization using ImageNet pre-trained weights.Stage 2: Fine-tuning on the clean RAF-DB dataset to learn discriminative facial representations without occlusion bias.Stage 3: Robustness-oriented fine-tuning on the noisy RAF-DB dataset (containing structured, glasses-like occlusions), allowing the models to adapt to occluded regions while preserving previously learned features (Table 1). Stage 1: Initialization using ImageNet pre-trained weights. Stage 1: Initialization using ImageNet pre-trained weights. Stage 2: Fine-tuning on the clean RAF-DB dataset to learn discriminative facial representations without occlusion bias. Stage 2: Fine-tuning on the clean RAF-DB dataset to learn discriminative facial representations without occlusion bias. Stage 3: Robustness-oriented fine-tuning on the noisy RAF-DB dataset (containing structured, glasses-like occlusions), allowing the models to adapt to occluded regions while preserving previously learned features (Table 1). Stage 3: Robustness-oriented fine-tuning on the noisy RAF-DB dataset (containing structured, glasses-like occlusions), allowing the models to adapt to occluded regions while preserving previously learned features (Table 1). Table 1Architectural specification of the MultiScaleConv and AttentionFusion modules.ModuleDescriptionOutput SizeShared Across BackbonesMultiScaleConvParallel convolutional branches applied to backbone feature maps to extract multi-scale spatial features[32, 510, 14, 14]Yes (identical for all backbones)AttentionFusionAttention-based reweighting of multi-scale feature maps to emphasize salient, non-occluded regions[32, 510, 14, 14]Yes (identical for all backbones)Integration StageApplied after backbone feature extraction and before global average pooling–Yes Table 1Architectural specification of the MultiScaleConv and AttentionFusion modules.ModuleDescriptionOutput SizeShared Across BackbonesMultiScaleConvParallel convolutional branches applied to backbone feature maps to extract multi-scale spatial features[32, 510, 14, 14]Yes (identical for all backbones)AttentionFusionAttention-based reweighting of multi-scale feature maps to emphasize salient, non-occluded regions[32, 510, 14, 14]Yes (identical for all backbones)Integration StageApplied after backbone feature extraction and before global average pooling–Yes Architectural specification of the MultiScaleConv and AttentionFusion modules. Feature enhancement modules To enhance robustness against structured occlusion noise, two auxiliary feature enhancement modules, namely MultiScaleConv and AttentionFusion, are incorporated after the backbone feature extractors. MultiScaleConv Module The MultiScaleConv module extracts complementary facial representations at multiple spatial resolutions. It consists of three parallel convolutional branches operating on the same input feature map: A 1 × 1 convolution (pointwise features).A 3 × 3 convolution (local features) with padding of 1 .A 5 × 5 convolution (broader context) with padding of 2 . A 1 × 1 convolution (pointwise features). A 1 × 1 convolution (pointwise features). A 3 × 3 convolution (local features) with padding of 1 . A 3 × 3 convolution (local features) with padding of 1 . A 5 × 5 convolution (broader context) with padding of 2 . A 5 × 5 convolution (broader context) with padding of 2 . The padding ensures that spatial dimensions remain consistent. The outputs of all branches are concatenated along the channel dimension and fused via a final 1 × 1 convolution to produce a unified multi-scale representation. AttentionFusion Module The AttentionFusion module adaptively reweights the features to suppress occlusion noise. Given an input tensor, it computes: Channel Attention: Using Global Average Pooling followed by a shared MLP to capture inter-channel dependencies.Spatial Attention: Using a 7 × 7 convolution on channel-wise pooled features to localize salient regions. Channel Attention: Using Global Average Pooling followed by a shared MLP to capture inter-channel dependencies. Channel Attention: Using Global Average Pooling followed by a shared MLP to capture inter-channel dependencies. Spatial Attention: Using a 7 × 7 convolution on channel-wise pooled features to localize salient regions. Spatial Attention: Using a 7 × 7 convolution on channel-wise pooled features to localize salient regions. The resulting attention maps are applied via element-wise multiplication, yielding a refined representation robust to partial occlusion. Experimental analysis This section presents an evaluation of the proposed FER models—ResNet-50, ViT-Base/16, and Swin-T—under both clean and structured-occlusion scenarios generated on the RAF-DB-Noisy dataset. All experiments were conducted following the standardized three-stage fine-tuning strategy described in Sect. 3, employing AdamW optimizer, label smoothing (0.15), and hybrid augmentation (CutMix + MixUp). During training, a stage-wise mixed data strategy was adopted. Models were first fine-tuned on the clean RAF-DB dataset and subsequently adapted to the noisy RAF-DB dataset to improve robustness against occlusion. Experimental setup Experiments were conducted on the Google Colab platform (NVIDIA Tesla T4 16GB VRAM, 12GB RAM) using PyTorch 2.3.0. All input images were resized to 224 × 224 pixels and augmented with random cropping, horizontal flipping, and color jittering. To ensure a rigorous and fair comparison, a completely unified training protocol was applied across all architectures94,95. Models were trained for 30 epochs with a batch size of 32 using the AdamW optimizer (weight decay 0.01, uniform initial learning rate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{5\times\:10}^{-5}$$\end{document}) and a ReduceLROnPlateau scheduler. Regularization techniques, including label smoothing (0.15) and CutMix/MixUp augmentations (50% probability per batch), were applied identically. No architecture-specific hyperparameter tuning was performed; thus, all performance variations are strictly attributable to the intrinsic structural differences of the models. Evaluation was based on Accuracy and Weighted F1-Score, with best checkpoints selected via the highest validation F1-Score. To guarantee statistical reliability under occlusion noise, all experiments were repeated five times using different random seeds. The reported metrics represent the mean performance, accompanied by standard deviations and 95% confidence intervals. Quantitative results To evaluate the quantitative performance of the proposed models under both clean and occluded facial conditions, we conducted a comprehensive comparison across three baselines — ResNet-50, Vision Transformer (ViT-B/16), and Swin-T Tiny.Each model was trained and tested on the RAF-DB and RAF-DB-noisy datasets using identical hyper-parameters and regularization settings. The metrics reported include overall accuracy and weighted F1-score for clean and noisy subsets, together with the robustness drop (ΔF1) that quantifies the degradation in performance when noise is introduced. Table 2Performance comparison of ResNet-50, ViT-B/16, and Swin-T Tiny under clean and occluded-noisy facial image conditions.ModelAccuracy (Clean)F1-Score (Clean)Accuracy (Noisy)F1-Score (Noisy)Robustness Drop (ΔF1)Relative RobustnessResNet-5087.0000000.83000072.8000000.720000− 0.1100000.87ViT-B/1682.5000000.84000070.0000000.720000− 0.1200000.86Swin-T Tiny88.6200000.86000075.0300000.740000− 0.1200000.86 Table 2Performance comparison of ResNet-50, ViT-B/16, and Swin-T Tiny under clean and occluded-noisy facial image conditions.ModelAccuracy (Clean)F1-Score (Clean)Accuracy (Noisy)F1-Score (Noisy)Robustness Drop (ΔF1)Relative RobustnessResNet-5087.0000000.83000072.8000000.720000− 0.1100000.87ViT-B/1682.5000000.84000070.0000000.720000− 0.1200000.86Swin-T Tiny88.6200000.86000075.0300000.740000− 0.1200000.86 Performance comparison of ResNet-50, ViT-B/16, and Swin-T Tiny under clean and occluded-noisy facial image conditions. Table 2 summarizes the quantitative performance of the baseline models under clean and noisy (occluded) conditions. Swin-T Tiny achieved the highest clean accuracy and F1-score (0.86), outperforming both ResNet-50 (0.83) and ViT-B/16 (0.84). Under occlusion, all models experienced an expected performance degradation. However, the absolute drop (ΔF1) remained consistent across architectures (ranging from − 0.11 to − 0.12), leading to comparable Relative Robustness scores (∼0.86 to 0.87). This indicates that all three backbones preserve a similar proportion of their recognition capability when facing local perturbations. Overall, Swin-T Tiny provides the most effective balance between clean accuracy and robustness. This advantage stems from its hierarchical transformer design, which effectively balances local feature sensitivity with global contextual modeling. Further qualitative interpretations of these response patterns and characteristic errors are discussed in Sect. 4.3. Table 3Performance stability on RAFDB-Noisy (five runs).ModelF1-score (Mean ± Std)95% Confidence IntervalResNet50 + CBAM0.720 ± 0.014[0.692, 0.748]ViT-B/160.720 ± 0.016[0.689, 0.751]Swin-T Tiny0.740 ± 0.012[0.717, 0.763] Table 3Performance stability on RAFDB-Noisy (five runs).ModelF1-score (Mean ± Std)95% Confidence IntervalResNet50 + CBAM0.720 ± 0.014[0.692, 0.748]ViT-B/160.720 ± 0.016[0.689, 0.751]Swin-T Tiny0.740 ± 0.012[0.717, 0.763] Performance stability on RAFDB-Noisy (five runs). As shown in Table 3, Swin-T exhibits not only the highest mean performance under occlusion but also the lowest variance across repeated runs, indicating superior robustness and result stability compared to CNN- and ViT-based baselines. Table 4P-value comparison for the algorithms.ComparisonΔF1p-valueSwin-T vs. ResNet50 + CBAM+ 0.02p = 0.031Swin-T vs. ViT-B/16+ 0.02p = 0.038 Table 4P-value comparison for the algorithms.ComparisonΔF1p-valueSwin-T vs. ResNet50 + CBAM+ 0.02p = 0.031Swin-T vs. ViT-B/16+ 0.02p = 0.038 P-value comparison for the algorithms. As shown in Table 4 to further validate the significance of the observed improvements, paired t-tests were conducted on the weighted F1-scores across repeated runs. The results show that Swin-T significantly outperforms ResNet-50 + CBAM (p = 0.031) and ViT-B/16 (p = 0.038) on the RAFDB-Noisy dataset, confirming that the performance gains are statistically significant rather than due to random variation.To analyze the contribution of the CBAM module, an ablation study was performed by removing CBAM from the ResNet-50 backbone. As shown in Table 5, the absence of attention leads to a noticeable performance degradation under occlusion, with a weighted F1-score drop of 0.036. This demonstrates that channel–spatial attention effectively enhances feature discrimination in occluded facial images. Table 5Ablation study of CBAM on ResNet-50 performance under occlusion noise.ModelAccuracy (%)Weighted F1-scoreResNet-50 (without CBAM)69.50.685ResNet-50 + CBAM72.80.720Improvement+ 3.3+ 0.035 Table 5Ablation study of CBAM on ResNet-50 performance under occlusion noise.ModelAccuracy (%)Weighted F1-scoreResNet-50 (without CBAM)69.50.685ResNet-50 + CBAM72.80.720Improvement+ 3.3+ 0.035 Ablation study of CBAM on ResNet-50 performance under occlusion noise. Table 5 reports the ablation results for the CBAM module. The baseline ResNet-50 without attention shows a noticeable performance drop compared to its CBAM-enhanced counterpart, demonstrating the effectiveness of channel–spatial attention in improving robustness under occlusion noise. Table 6Ablation study of MultiScaleConv and AttentionFusion on RAFDB-Noisy (Swin-T backbone).ConfigurationMultiScaleConvAttentionFusionAccuracy (%)Weighted F1Swin-T (Full)✓✓75.030.740w/o MultiScaleConv✗✓72.900.715w/o AttentionFusion✓✗72.400.708w/o Both✗✗71.800.698 Table 6Ablation study of MultiScaleConv and AttentionFusion on RAFDB-Noisy (Swin-T backbone).ConfigurationMultiScaleConvAttentionFusionAccuracy (%)Weighted F1Swin-T (Full)✓✓75.030.740w/o MultiScaleConv✗✓72.900.715w/o AttentionFusion✓✗72.400.708w/o Both✗✗71.800.698 Ablation study of MultiScaleConv and AttentionFusion on RAFDB-Noisy (Swin-T backbone). To evaluate the individual contribution of the proposed auxiliary modules, we conduct an ablation study using Swin-T as the representative backbone on the RAF DB-Noisy dataset. As reported in Table 6, removing either MultiScaleConv or AttentionFusion leads to a clear degradation in both accuracy and weighted F1-score, confirming their effectiveness in improving robustness under partial facial occlusion. Notably, excluding the AttentionFusion module results in a larger performance drop, highlighting the importance of adaptive feature reweighting in suppressing occlusion-corrupted responses. The most significant decline is observed when both modules are removed simultaneously, indicating that MultiScaleConv and AttentionFusion provide complementary benefits when jointly employed. Qualitative analysis In addition to the quantitative analysis presented in Sect. 4.2, this section offers a comprehensive qualitative investigation into the behavior of the models, both in the presence and absence of noise (ocular occlusion). The objective is to visually and analytically interpret how each architecture reacts to the class-specific perturbations introduced in Sect. 3.1 and how these reactions are reflected in the observed F1 Scores96. The evaluation of each model (ResNet-50 + CBAM, ViT B/16, Swin-T Tiny) is segmented into four visual elements: (a) Confusion Matrix, (b) Per Class Metrics, (c) TP/FP/FN Analysis, and (d) ROC AUC Curves. Collectively, these illustrations not only underscore the strengths and weaknesses but also reveal unique error characteristics that are not immediately apparent from the aggregate scores (Fig. 8). Fig. 8Attention visualization (Grad-CAM) under clean and occluded conditions. Due to copyright restrictions of the original RAF-DB dataset, a royalty-free illustrative face image has been used as the base. The displayed heatmaps, however, represent the authentic attention patterns extracted from our models (ResNet-50, ViT, and Swin-T). These real attention patterns are mapped onto this symbolic image solely to visually demonstrate the models’ focus areas under different conditions while complying with copyright policies. Attention visualization (Grad-CAM) under clean and occluded conditions. Due to copyright restrictions of the original RAF-DB dataset, a royalty-free illustrative face image has been used as the base. The displayed heatmaps, however, represent the authentic attention patterns extracted from our models (ResNet-50, ViT, and Swin-T). These real attention patterns are mapped onto this symbolic image solely to visually demonstrate the models’ focus areas under different conditions while complying with copyright policies. Qualitative Analysis for ResNet-50 + CBAM To further understand the spatial behavior of ResNet-50 enhanced with the Convolutional Block Attention Module (CBAM), qualitative visualizations were examined under both clean and occluded conditions. This analysis highlights how the integrated attention mechanism redirects feature focus from the occluded eye region to the visible lower-face areas, such as the mouth and cheeks, improving robustness to simulated glasses noise. The following figures illustrate activation responses and attention heatmaps across different emotion categories. Fig. 9Qualitative analysis results for ResNet-50 + CBAM on RAF-DB-Noisy. (a) confusion matrix, (b) per-class metrics, (c)TP/FP/FN analysis, (d) ROC–AUC Analysis. Qualitative analysis results for ResNet-50 + CBAM on RAF-DB-Noisy. (a) confusion matrix, (b) per-class metrics, (c)TP/FP/FN analysis, (d) ROC–AUC Analysis. Figure 9(a–b) illustrate the qualitative performance of ResNet-50 + CBAM on RAF-DB under occlusion noise. As shown in the confusion matrix (Fig. 9a), the model achieves its highest accuracy on neutral expressions, while visually similar and subtle emotions such as fear and sad exhibit higher confusion, particularly under eye-region occlusion. This indicates the sensitivity of CNN-based models to fine-grained affective cues disrupted by noise. The per-class precision, recall, and F1 scores in Fig. 9b further confirm this trend, where neutral and happy maintain relatively stable performance, whereas fear and sad show noticeable degradation. Despite this, the incorporation of CBAM improves feature localization and preserves a reasonable overall performance (weighted F1 = 0.72), indicating partial robustness against occlusion. The TP–FP–FN distribution and ROC curves (Fig. 9c–d) consistently reflect this behavior: dominant expressions yield higher separability, while subtle emotions remain more vulnerable. Overall, ResNet-50 + CBAM demonstrates moderate robustness under occlusion, benefiting from attention-based reweighting, yet remaining less resilient than transformer-based architectures. Qualitative Analysis for ViT-B/16 In order to further interpret the behavior of the Vision Transformer under structured occlusion, a qualitative analysis was conducted using attention heatmaps and token-based visualizations from ViT-B/16. These visual patterns illustrate how the model reallocates its attention across non-occluded facial regions when the upper eye area is covered by simulated glasses noise. Compared to CNN-based architectures, ViT demonstrates a more distributed and adaptive attention flow, dynamically integrating visual cues from the mouth, nose, and cheek regions to maintain semantic understanding of the emotional state. The dense activation across multiple facial zones indicates the model’s ability to capture long-range dependencies even when high-salience regions are absent. The following figures display representative attention maps for different emotion classes under clean and noisy conditions, highlighting ViT-B/16’s superior resilience to occlusion artifacts through global context aggregation. Fig. 10The confusion matrix visualization of the ViT B/16 is depicted on the RAF DB Noisy condition. (a) Confusion matrix, (b) per-class metrics, (c)TP/FP/FN analysis. (d) ROC–AUC Analysis. The confusion matrix visualization of the ViT B/16 is depicted on the RAF DB Noisy condition. (a) Confusion matrix, (b) per-class metrics, (c)TP/FP/FN analysis. (d) ROC–AUC Analysis. As shown in Fig. 10(a), the confusion matrix indicates that ViT B/16 preserves a moderate level of robustness under occlusion, with strong diagonal dominance for globally salient emotions such as Happy (class 4) and Angry (class 1), while experiencing noticeable confusion for detail-dependent categories, particularly Fear (class 2) and Sadness (class 3). This behavior reflects the transformer’s reliance on global token interactions, which remain effective for expressions involving large-scale facial activations but are less resilient to localized perturbations in the eye and mouth regions. The per-class Precision, Recall, and F1-score distributions in Fig. 10(b) further confirm this trend, where Happy achieves the most balanced performance, whereas Fear and Sadness suffer from reduced recall, indicating under-recognition under occlusion. The intermediate categories (Disgust, Neutral, and Surprise) exhibit moderate and more balanced performance, suggesting partial generalization to mid-intensity affective patterns. Consistent observations are reflected in the TP/FP/FN distributions shown in Fig. 10(c), where ViT B/16 demonstrates high true positive rates for globally distinctive emotions but elevated false negatives for subtle expressions, highlighting the absence of strong localized inductive bias. Finally, the ROC curves in Fig. 10(d) reveal consistently higher AUC values for Happy, followed by Angry and Surprise, while Fear and Sadness exhibit flatter curves, indicating reduced sensitivity to fine-grained cues under noisy conditions. these results characterize ViT B/16 as a globally attentive yet locally fragile architecture, capable of robust overall emotion recognition but limited in preserving fine-grained facial details under occlusion. To further analyze model behavior under occlusion, attention-based visualizations were conducted. Channel-spatial attention maps, as shown in Fig. 8, confirm that while occlusion forces standard models to inappropriately focus on the frame edges, the attention-fusion mechanism dynamically redirects focus towards the lower face and unoccluded periocular regions, thereby preserving feature integrity. Qualitative Analysis for Swin-T Tiny In this section, a qualitative examination is conducted to interpret the attention distribution and feature adaptability of the Swin Transformer Tiny when exposed to occluded facial conditions. The hierarchical shifted-window mechanism of Swin-T enables dynamic focus realignment across multiple spatial scales, facilitating the capture of meaningful cues even when a large portion of the eye region is concealed by simulated glasses noise. Unlike the globally uniform attention observed in ViT-B/16, Swin-T exhibits progressive localization — concentrating first on unoccluded mid-face areas and then aggregating contextual signals from peripheral regions such as the mouth and cheeks. This adaptive multi-scale response explains its superior robustness and balanced activation density, as visualized in the following heatmaps. The depicted qualitative results demonstrate that Swin-T efficiently compensates for lost upper-face information by reorganizing feature dependencies within shifted attention windows, maintaining discriminative awareness of emotional classes under severe occlusion. Fig. 11Qualitative analysis results for Swin T Tiny. (a) Confusion matrix, (b) per-class metrics, (c)TP/FP/FN analysis. (d) ROC–AUC Analysis. Qualitative analysis results for Swin T Tiny. (a) Confusion matrix, (b) per-class metrics, (c)TP/FP/FN analysis. (d) ROC–AUC Analysis. Figure 11(a–d) present the qualitative performance of Swin-T Tiny on RAF-DB under eye-region occlusion. As shown in the confusion matrix (Fig. 11a), the majority of predictions lie on the main diagonal, indicating strong robustness to localized noise, particularly for globally defined expressions such as neutral, happy, and angry. In contrast, subtle emotions including fear and sadness exhibit higher confusion, primarily with neutral and angry, reflecting their dependence on fine-grained ocular cues. The per-class precision, recall, and F1 scores (Fig. 11b) further confirm this behavior: neutral and happiness maintain stable performance under occlusion, while fear and sadness experience noticeable degradation. Intermediate classes such as disgust and surprise remain moderately separable, benefiting from Swin-T’s hierarchical window-based attention. The TP–FP–FN distribution (Fig. 11c) and ROC curves (Fig. 11d) consistently support these observations, showing higher separability and smoother class transitions compared to CNN-based backbones, with most AUC values exceeding 0.85. Overall, Swin-T Tiny demonstrates superior robustness to eye-region occlusion by effectively integrating local and global facial cues, outperforming ResNet-based models under noisy conditions. Summary of qualitative findings This subsection summarizes the overall qualitative behaviors observed across all architectures and presents the class-wise quantitative metrics of Swin-T Tiny, the best-performing model on the RAF-DB-Noisy dataset. Among the evaluated networks, the convolutional ResNet-50 with CBAM relied primarily on localized spatial patterns, leading to strong responses in easily separable categories but confusions within occluded regions. The ViT-B/16 model, based on patch-level global attention, exhibited visible sensitivity to ocular occlusion due to its rigid token dependency, which degraded its per-class consistency under noise. In contrast, the hierarchical transformer Swin-T Tiny demonstrated stable activations and smoother region attention under occlusion, maintaining coherent intra- and inter-class boundaries. The detailed per-class metrics summarized in Table 7 confirm this qualitative trend. Swin-T Tiny achieved notably balanced Precision and Recall for most emotions, with high AUC values validating its discriminative robustness. Classes such as Happy and Neutral remain highly reliable, while Surprise shows the most noticeable degradation under occlusion. Table 7Class-wise Precision, Recall, F1-score, and AUC of Swin-T Tiny on RAF-DB-Noisy.EmotionPrecisionRecallF1-ScoreAUCSurprise0.8240.7110.7630.96Fear0.5160.3480.4160.78Disgust0.5930.1500.2390.75Happy0.8730.9180.8950.98Sad0.6140.6720.6420.89Angry0.7560.5040.6050.91Neutral0.6490.8160.7230.93Average (Macro)0.6890.5880.6120.886 Table 7Class-wise Precision, Recall, F1-score, and AUC of Swin-T Tiny on RAF-DB-Noisy.EmotionPrecisionRecallF1-ScoreAUCSurprise0.8240.7110.7630.96Fear0.5160.3480.4160.78Disgust0.5930.1500.2390.75Happy0.8730.9180.8950.98Sad0.6140.6720.6420.89Angry0.7560.5040.6050.91Neutral0.6490.8160.7230.93Average (Macro)0.6890.5880.6120.886 Class-wise Precision, Recall, F1-score, and AUC of Swin-T Tiny on RAF-DB-Noisy. As presented in Table 7, the Swin-T Tiny demonstrates resilient performance metrics across key emotion categories on the RAF-DB-Noisy dataset. The model achieves robust class separability, particularly for Happy (F1 = 0.90, AUC = 0.98) and Surprise (F1 = 0.76, AUC = 0.96), indicating that the hierarchical attention mechanism effectively preserves distinctive facial features associated with these expressions despite occlusion. Neutral also maintains solid recognition (F1 = 0.72), serving as a reliable baseline. Conversely, categories relying on subtle micro-expressions, specifically Disgust (F1 = 0.24) and Fear (F1 = 0.42), exhibit noticeable degradation. This performance drop is attributed to the heavy occlusion of the eye and nose regions, which are critical for distinguishing these specific emotions. the results confirm that while structured noise impacts fine-grained recognition, Swin-T retains high structural capability for globally distinct expressions. Limitations While the proposed evaluation framework enables a controlled comparison of model robustness under structured occlusion, and statistical significance was validated across five independent runs, evaluating performance across a broader range of noise intensities remains computationally constrained. Furthermore, we simulated glasses-like noise because large-scale, well-annotated datasets with real facial occlusions are scarce. While this synthetic approach guarantees precise control and reproducibility, real-world interferences such as hands, dynamic lighting, or irregular medical masks are naturally more complex. The base RAF-DB dataset, despite its diversity, may not fully capture the extreme distribution shifts present in uncontrolled, real-world environments. Conclusion In this study, we evaluated CNN- and Transformer-based architectures on the RAF-DB-Noisy dataset to analyze their behavior against regional occlusion and corrupted labels. Our results showed that the Swin-T Tiny architecture achieved the highest Accuracy and F1-Score under both clean and noisy conditions. Interestingly, all tested models experienced a similar performance drop when facing extreme noise, showing a stability decrease of ΔF1 ≈ − 0.11 to − 0.12. Qualitative analysis revealed that Swin-T Tiny maintains a robust balance between Precision and Recall, yielding per-class AUC values over 0.85. This highlights its strong generalization capability under partial occlusion. In contrast, while ViT-B/16 was effective at global feature extraction, it struggled with local facial details. ResNet-50, on the other hand, remained reliable for recognizing coarser facial patterns. Overall, Hierarchical Transformers like Swin-T provide the best trade-off between feature fidelity and noise robustness for emotion recognition. To ensure a fair comparison across models with fundamentally different learning dynamics, we applied architecture-aware hyperparameter tuning within a unified training protocol. A key limitation of our work, however, is the use of synthetic occlusion. We simulated glasses-like noise because large-scale, well-annotated datasets with real facial occlusions are scarce. While this approach guarantees precise control and reproducibility, real-world interferences—such as hands, medical masks, or dynamic lighting—are naturally more complex. Additionally, the base RAF-DB dataset may not fully capture the extreme diversity of uncontrolled environments. Moving forward, we plan to address these limitations through two strategic directions. First, since severe visual occlusion inevitably destroys crucial facial cues, integrating non-visual data (such as audio or physiological signals via Multi-modal Data Fusion) is a logical step to compensate for the missing information. Second, to improve training stability on datasets like RAF-DB-noisy, we propose exploring an Occlusion Curriculum Learning strategy. Training models on progressively increasing noise levels, rather than static occlusion, could significantly boost the adaptability and resilience of real-world emotion recognition systems.