Weaknesses of Facial Emotion Recognition Systems
Aleksandra Jamróz ∗, Patrycja Wysocka ∗, Piotr Garbat · 2026
Read Weaknesses of Facial Emotion Recognition Systems by Aleksandra Jamróz ∗, Patrycja Wysocka ∗, Piotr Garbat (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12402.txt.
Introduction
Aleksandra Jamroz, Patrycja Wysocka, and Piotr Garbat review systems that read emotion from faces. Such systems play a role in how humans and computers interact. The range of methods in the field is vast. That spread alone makes a close review of the work worthwhile.
The team pick out three of the most notable and best solutions for close study. They also pick three datasets that stand out for range and size. With those in hand, they probe where face based emotion systems fall short. Weak spots can range from odd lighting to faces that do not fit the norm. A clear map of flaws helps builders know what to fix first.
Catalog record
| CREATOR | Aleksandra Jamróz ∗, Patrycja Wysocka ∗, Piotr Garbat |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12402
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weaknesses of Facial Emotion Recognition Systems Aleksandra Jamróz ∗, Patrycja Wysocka ∗, Piotr Garbat Abstract Emotion detection from faces is one of the machine learning problems needed for human-computer interaction. The variety of methods used is enormous, which motivated an in-depth review of articles and scientific studies. Three of the most interesting and best solutions are selected, followed by the selection of three datasets that stood out for the diversity and number of images in them. The selected neural networks are trained, and then a series of experiments are performed to compare their performance, including testing on different datasets than a model was trained on. This reveals weaknesses in existing solutions, including differences between datasets, unequal levels of difficulty in recognizing certain emotions and the challenges in differentiating between closely related emotions. * * footnotetext: These authors contributed equally to this work 1 Introduction This study presents a comprehensive analysis of the most advanced facial emotion recognition solutions currently available. The approach evaluates network performance when trained on one dataset and applied to others. This methodology enables to ascertain the model’s generalisation capabilities, which is a crucial aspect in determining its state-of-the-art status. The study showed that a network’s performance on different datasets is significantly worse than when it is tested on the same dataset on which it was trained. The motivation for this study lies in the crucial role emotions play in decision-making and interpersonal relationships. As technology advances, human-computer systems are expected to evolve, and incorporating emotion recognition is a way for enhancing their responses. Computer vision methods play a crucial role here as humans perceive messages only in 7% from the verbal part - words, while as much as 55% from non-verbal messages - body language and facial expressions [ 1 ] . By understanding users’ emotional states, human - computer interaction systems will be able to engage more naturally, improve user interactions, and foster stronger connections. A significant gap in current FER research lies in the thorough verification and evaluation of proposed solutions. While many papers claim state-of-the-art performance based on accuracy metrics within their training domain, a comprehensive cross-dataset testing reveals concerning limitations in real-world generalization capabilities. This work addresses this critical need, acknowledging that true progress in FER requires not just innovation, but thorough verification and replication of existing approaches. 2 Related work Universal recognition of facial expressions supports the development of computer vision-based facial emotion recognition (FER) systems, making them effective across different populations. In 1972, Paul Ekman and Wallace V. Friesen [ 2 ] conducted a pioneering study identifying six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. By photographing individuals worldwide and asking people from different cultures to match expressions to emotion words in their language, they found strong cross-cultural agreement, suggesting that some facial expressions are universally recognized, even if cultures differ in emotional expressiveness. Cross-database evaluation is a practice that allows the generalisation capabilities of a model to be tested outside the laboratory environment. It is particularly useful in cases where multiple datasets are available for the same task, such as object recognition [ 3 ] [ 4 ] and medical cases [ 5 ] [ 6 ] . Robustness, which is natural for humans, is not so trivial for neural networks, which are vulnerable to noise in the data [ 7 ] and can also learn not obvious features that interfere with correct predictions [ 8 ] . There have been similar reviews [ 9 ] [ 10 ] of FER methods on different datasets, but they do not include the latest deep learning solutions or the now most popular datasets on which the benchmarks are calculated. Current state-of-the-art models, such as POSTER++ [ 13 ] , DMUE [ 14 ] and DAN [ 15 ] were only evaluated on the same test set as they were trained on. These are at the same time models that were chosen for experimentation. 3 General concept Although well-trained models can directly produce high-confidence inferences, the inference results may actually be erroneous. Therefore, evaluating and selecting a model with respect to its robustness to data of different origins is essential. This study examined the basic performance parameters of the model by multi-validation on different datasets. This work is focused on a discrete representation of emotion, along with a static image-based approach. This choice was driven by the broader availability and greater variety of image-based datasets compared to video-based ones and multiplicity of state-of-the-art models available for discrete problems. We specifically selected "in-the-wild" datasets to better simulate real-world conditions. These large-scale facial expression recognition datasets often face issues of annotation ambiguity. The challenges of creating reliable labels in static image-based FER datasets are influenced by many factors, such as: • Complexity of emotions - real-world facial expressions often represent a mixture of emotions, rather than the exaggerated, single emotions typically observed in controlled laboratory settings. Annotators are forced to select a primary emotion (out of 6 basic emotions), potentially misinterpreting the true emotional state. • Lack of context - one picture may not provide enough information to distinguish between certain emotions. For example, an expression may be surprise or fear depending on context. Video-based approaches are a solution to this issue, but they require much more resources. • Annotators biases - the ambiguity of facial expressions and the lower quality of images result in different biases in the interpretation of emotions, with a greater inclination towards negativity (anger) rather than positivity (happiness) [ 19 ] . These challenges have an impact on the final appearance of the datasets, and the occurrence of internal errors and discrepancies between different datasets. Creating reliable and robust FER solutions is crucial because errors in emotional recognition can have severe consequences in critical domains like healthcare. Misinterpreting a patient’s emotional state could lead to improper mental health diagnoses and potentially harmful treatment decisions. Similarly, false positives identifying emotions that aren’t present could trigger unnecessary security responses or interventions. Furthermore, the cumulative impact of these failures can severely damage organizational credibility and public trust in FER technology, making it essential to prioritize reliability and robustness in development to ensure these systems genuinely benefit society rather than cause unintended harm. 4 Experiments 4.1 FER Models The three most effective solutions were selected based on their actual benchmark ranking. These are: POSTER++ [ 13 ] , DMUE [ 14 ] and DAN [ 15 ] . All of them were declared state of the art in their publications at the time the work was produced. The objective was to train each model using three various datasets, validate them on their respective training datasets, and then multi-validate them on the other datasets. The initial implementations of the models were obtained from the official repositories provided by the authors. The code was adapted to the hardware capabilities, keeping the original training parameters as close as possible or adjusting them based on the authors’ suggestions. Batch size was the parameter we modified most often. 4.2 Datasets As the primary goal of developing a FER system is to use it in real-world application, so datasets collected in uncontrolled environments were selected for analysis. They all employ a single classification method for identifying emotional