Leaving Some Facial Features Behind
Cheng Qiu · 2024
Read Leaving Some Facial Features Behind by Cheng Qiu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00824.txt.
Description
A 2024 arXiv research paper by Cheng Qiu. Facial expressions are crucial to human communication, offering insights into emotional states. This study examines how specific facial features influence emotion classification, using facial perturbations on the Fer2013 dataset. As expected, models trained on data with the removal of some important facial feature experienced up to an 85% accuracy drop when compared to baseline for emotions like happy and surprise. Surprisingly, for the emotion disgust, there seem to be slight improvement in accuracy for classifier after mask have been applied.
Catalog record
| CREATOR | Cheng Qiu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leaving Some Facial Features Behind Cheng Qiu Abstract Facial expressions are crucial to human communication, offering insights into emotional states. This study examines how specific facial features influence emotion classification, using facial perturbations on the Fer2013 dataset. As expected, models trained on data with the removal of some important facial feature experienced up to an 85% accuracy drop when compared to baseline for emotions like happy and surprise. Surprisingly, for the emotion disgust, there seem to be slight improvement in accuracy for classifier after mask have been applied. Building on top of this observation, we applied a training scheme to mask out facial features during training, motivating our proposed Perturb Scheme. This scheme, with three phases—attention-based classification, pixel clustering, and feature-focused training, demonstrates improvements in classification accuracy. The experimental results obtained suggests there are some benefits to removing individual facial features in emotion recognition tasks. 1 Introduction Understanding emotions plays a pivotal role in how we perceive and interact with others. When interpreting facial emotions, key landmarks such as the eyes and mouth provide significant insights into a person’s emotional state [ 2 ] . Additionally, faces can be divided into two halves, focusing on the eyes and eyebrows on one side and the mouth on the other [ 8 ] . To enhance our comprehension of facial emotions, it is essential to understand how these facial landmarks contribute to emotion prediction. To investigate the impact of facial landmarks on emotion classification, we introduced perturbations such as a physical mask to Fer2013, a facial emotion dataset, through similar procedure to that of [ 3 ] . In this paper, the masked facial emotion dataset will be referred to as MaskFer 1 1 1 Dataset can be accessed at https://github.com/chengq220/MaskFerDataset . Models were trained on both Fer2013 and MaskFer. The performance results for accuracy are presented in Table 1 . The accuracy ratio for each emotion class generally decreased for model trained on MaskFer when compared to model trained on Fer2013. This is to be expected because by applying a mask, important features are removed from the context, resulting in less information models to determine the emotion. However, the percent change in accuracy ratio varied across different classes. For example, the accuracy for emotions like happiness dropped by around 60% while for fear, there was only a slight drop of around 10%. Surprisingly, for the emotion sad, there’s even a slight increase in accuracy ratio which means that the model trained on MaskFer performed better than its counterpart trained on Fer2013. For this specific emotion, we hypothesized that masking the mouth allowed the model to extract better information relevant to the emotion of sad such as the eyebrow. Emotion [ 5 ] [ 1 ] [ 9 ] [ 4 ] [ 12 ] Ensemble Angry -39% -35% -32% -20% -23% -32% Happy -49% -63% -42% -55% -35% -42% Sad -12% -24% -4% -16% -8% -4% Neutral -51% -60% -53% -60% -43% -53% Surprise -19% -26% -35% -19% -20% -35% Disgust -49% -59% -39% -49% -49% -39% Fear -20% -23% -13% -18% -17% -13% Table 1: Percent accuracy gain/loss for each emotion class with models trained on MaskFer compared to models trained on FER2013 By examining the saliency map shown in Figure 1 , it becomes evident that for emotions such as disgust, anger, and sadness, the model trained on Fer2013 may not have learned the optimal salient features. In contrast, the model trained on MaskFer was able to capture the features in the eyebrows when identifying an angry face. It might be case that models trained on Fer2013 do not effectively leverage all the information present in the vicinity of major facial landmarks like the eyebrows. While classifying emotions based on overall facial landmarks may provide a global context that’s beneficial to emotion classification, considering them as individual features provide better localized context that’s better suited for classification of specific emotions. Figure 1: Saliency map for trained models on Fer2013 and MaskFer 2 Related Works Neural networks have become a natural choice for facial emotion recognition tasks due to their ability to learn complex representations from visual data. With the rise of deep learning, networks like ResNet [ 4 ] have paved the way for deeper architectures by introducing skip connections, which allow information to flow across layers without degradation. This innovation enables deeper networks that can learn richer features, albeit with increased computational complexity. DenseNet [ 5 ] later addressed this issue by establishing direct connections between layers, effectively reducing the number of parameters while maintaining performance, which is crucial for real-time applications. Recent advances have also seen the development of the Dual Path Network (DPN) [ 1 ] , which combines the advantages of both ResNet’s and DenseNet’s connectivity patterns. By incorporating a dual-path structure, DPN allows features to be reused efficiently across layers, further enhancing classification accuracy while managing computational load. The FER2013 benchmark dataset has been widely used to train and evaluate facial emotion recognition models. Many studies have built upon convolutional neural networks (CNNs) to improve baseline models. For example, the addition of attention mechanisms has enabled models to selectively focus on significant facial areas, such as the eyes and mouth, enhancing the interpretability and accuracy of emotion recognition [ 7 ] . Attention mechanisms, inspired by human visual perception, help networks prioritize crucial regions that are most indicative of emotions, particularly under challenging conditions where parts of the face might be occluded or not visible. Additionally, researchers have experimented with hyper-parameter tuning using genetic algorithms to optimize hyper-paramaters of model such as VGG architecture. This approach improved the FER2013 classification accuracy by refining aspects like kernel size and learning rate, ultimately achieving a single-network classification accuracy of 76% [ 11 ] . Spatial transformations have also been applied to increase robustness against rotations, translations, and other variations, which are common in real-world scenarios [ 2 ] . Despite these improvements, emotion recognition models still face substantial challenges, particularly when deployed in noisy, ambiguous, or complex environments. Issues such as inconsistent annotations and the diverse real-world backgrounds can impact performance significantly. Furthermore, in applications where parts of the face are occluded, like mask-wearing scenarios, critical facial landmarks might be hidden, presenting additional challenges. Emotion classification under partial face occlusion requires models to make predictions based on incomplete information, often resulting in lower accuracy. To address these challenges, researchers have turned to techniques like transfer learning, where models pre-trained on large, general-purpose datasets are fine-tuned on smaller, task-specific datasets. This approach has shown promising results, as pre-trained models can leverage learned representations from a broad data distribution to perform well on specific tasks, even with limited data [ 6 ] . Moreover, the introduction of new datasets designed to include occlusions, such as MaskFer, enables models to better generalize to masked or partially visible faces. MaskFer, for instance, includes images with the lower part of the face masked, simulating scenarios in which only the upper facial features are visible. Research utilizing CNNs in these scenarios aims to identify and adapt to visible facial landmarks, acknowledging that masking impacts emotion recognition by obscuring expressive regions like the mouth and nose [ 8 ] . Innovative approaches specifically