Machine and deep learning in facial expression recognition: a survey based on facial action units Authors: Mauricio Priego, Itzel Aranguren, Diego Oliva, Arturo Valdivia-G, Mario A Navarro, Marco Pérez-Cisneros Published: 2026 ABSTRACT AbstractFacial Expression Recognition (FER) has become increasingly relevant in engineering applications spanning security, healthcare, human-robot interaction, and intelligent systems. Accurate recognition of emotional states through facial cues is particularly important for enabling autonomous systems to interpret human intent in dynamic environments. This study presents a survey of recent advances in FER, with a focus on machine learning (ML) and deep learning (DL) methodologies developed between 2019 and 2024. Special attention is given to Facial Action Unit Detection (FAUD) as an intermediate and physiologically grounded representation for expression recognition. The analyzed literature is organized according to the final scope of the proposed models, distinguishing FAUD-oriented, FER-oriented, AU-assisted FER, and FAUD-to-FER approaches. Through this structured comparison, the survey identifies methodological trends, reported performance patterns, and the conditions under which FAUD-based intermediate modeling appears particularly advantageous for reliable FER.Keywords: Facial expression recognition, Action units, Facial action unit detection, Computer vision, Deep learningSubject terms: Computational biology and bioinformatics, Engineering, Mathematics and computing Introduction Facial Expression Recognition (FER), a specialized area within computer vision, has experienced substantial advancement in recent years due to its broad impact across multiple domains. In driver monitoring systems and automotive safety, FER is utilized to estimate drivers’ internal moods, assess their influence on accident risk, and enhance driver assistance systems1. In the context of human-machine interaction, FER enables real-time detection of workers’ emotions, allowing collaborative robots to adjust their behavior and speed according to the operator’s mood or stress level2. FER has also been implemented to evaluate students’ emotional engagement during virtual classes, which assists teachers in identifying signs of disinterest or frustration and adapting instructional strategies accordingly3,4. In healthcare, FER provides an objective and quantifiable measure of pain in patients who have difficulty communicating verbally, including newborns, preverbal children, individuals with autism spectrum disorder, and adults with dementia5,6. Furthermore, FER is employed in social research to analyze interpersonal synchrony and manipulate social cues in a controlled manner, serving as a precise indicator for evaluating group interaction quality and the causal mechanisms underlying human attraction7,8. Although the scientific community has published various research in recent decades aimed at improving the detection of emotions9,10, the FER area still presents ample challenges, such as the natural distortion that facial expressions suffer with gesticulation resulting from speech, the temperament of individuals, or the innate capacity of the human being to feign emotions11,12. There are also technical challenges, such as obtaining large datasets with a diversity of people, genders, ages, and ethnic and cultural origins without violating the right to privacy and protection of personal data of those involved (which includes their faces), the cost of long hours of certified frame-by-frame labeling, or the consequent imbalance and bias of doing it through online calls due to the subjectivity of well-intentioned people but not trained for the task13,14. Real-time and edge computing15 solutions are an additional latent requirement. Whatever the intended use of FER models, most will eventually need to be deployed in the field, such as for patient treatment, the prevention of dishonest activities, or the interaction of robots with people. Therefore, these application types are not yet seen in the software market. However, it is only a matter of time before the same market demands them, and scientific and technological development must be prepared accordingly. The interest in FER is such that the Affective Behavior Analysis in-the-wild (ABAW) workshop and competition is periodically organized in collaboration with the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) to motivate and accelerate research into models that interpret human emotions16,17. To address the challenges described above, novel algorithms and methodologies have been proposed, some of which have been included in previously published surveys. A review of recent surveys shows that they primarily focus on proposals based on Deep Learning (DL) technologies and their comparison with Machine Learning (ML) techniques, training strategies with single or multiple labeling, with static images or dynamic sequences, and the identification of emotional intensity18–22. Nevertheless, new tools have been developed technologically that reinforce or even compete with Convolutional Neural Networks (CNNs) for computer vision issues, such as Vision-Transformers23, attention tools24 or graph neural networks25, and even tools outside the neural area such as Bayesian Networks have been implemented with high results26, which can improve existing FER models. Based on the above, this survey seeks to integrate into a single document the main advances that FER has experienced in the last five years, focusing on the line of research based on Facial Action Unit Detection (FAUD) for the recognition of facial expressions, comparing the Artificial Intelligence (AI) techniques applied, the principal datasets utilized, and the results obtained in the principal proposals throughout these years. The primary contributions of this article are as follows: A comparative analysis of algorithms and methodologies for Facial Action Unit Detection (FAUD), emphasizing reported performance patterns across the analyzed studies. A comparative analysis of algorithms and methodologies for Facial Action Unit Detection (FAUD), emphasizing reported performance patterns across the analyzed studies. A comparative analysis of diverse FAUD-based algorithms and methodologies for Facial Expression Recognition (FER), emphasizing reported performance patterns and methodological trends. A comparative analysis of diverse FAUD-based algorithms and methodologies for Facial Expression Recognition (FER), emphasizing reported performance patterns and methodological trends. A comparative evaluation of prevalent datasets used in FAUD and FER, assessing their influence on training and predictive modeling. A comparative evaluation of prevalent datasets used in FAUD and FER, assessing their influence on training and predictive modeling. An interpretative synthesis of the main findings observed during the study and evaluation of AI models, together with the identification of promising directions for future research. An interpretative synthesis of the main findings observed during the study and evaluation of AI models, together with the identification of promising directions for future research. The remainder of this document is organized as follows. Section “Background” provides essential information to facilitate the understanding of terms related to emotion detection, such as the primary methodologies for identifying facial expressions, the Emotion Facial Action Coding System (EMFACS), and the Arousal-Valence-2D-Model (AV2D). It also includes information related to the PRISMA methodology, which was followed to analyze the state of the art on which this work is based. For ease of reference, the acronyms defined in the main body of the manuscript are listed in Appendix B. Section "The facial expression recognition through facial action unit detection" focuses on FER based on FAUD, reviewing the principal datasets used for training and testing the proposed models, examining the challenges posed by both FER and FAUD, and presenting the different proposals included in this work, divided into those that focus on expression recognition, those that focus on Action Unit (AU) detection, multitasking solutions, and real-time detection proposals. In Section “Comparisons”, the different models presented in the literature and their reported results are compared. The results of these comparisons are analyzed in Section "Discussion and conclusion", from which the corresponding conclusions are drawn. Finally, Section “Future directions” outlines possible future directions. Background This section presents previous research and studies in the field of Facial Expression Recognition (FER), as well as noteworthy efforts for its dissemination, based on which the main advances of FER within the field of Artificial Intelligence have been achieved. Related literature reviews Humans are not taught to interpret facial expressions to identify emotions. We learn this in the first years of life as a fundamental part of our communication and empathetic abilities. What we do, fundamentally, is identify patterns in facial movement, and based on this principle, efforts have been directed to teach machines to recognize them as well. Facial expression recognition techniques Efforts aimed at FER focus on detecting facial landmarks (e.g., eyebrows, eyes, corners of the lips) whose shape or position characterize the emotions expressed through the face. The techniques usually used can be divided mainly into three branches: Semantic detection is based on algorithms that identify the differences in density and depth of the pixels that characterize the facial landmarks and their shape27. It is the simplest way to implement FER, as its principle is similar to face recognition. However, it tends to present significant biases based on the ethnicity or age of the individual, such as confusing a serious person of Asian origin with a smiling face just because they have slanted eyes. Semantic detection is based on algorithms that identify the differences in density and depth of the pixels that characterize the facial landmarks and their shape27. It is the simplest way to implement FER, as its principle is similar to face recognition. However, it tends to present significant biases based on the ethnicity or age of the individual, such as confusing a serious person of Asian origin with a smiling face just because they have slanted eyes. Orthogonal distances, another branch of FER, are designed to calculate the shift of facial landmarks from their position in a neutral expression to the position that signifies the facial expression28. This method shows promise in mitigating biases based on ethnic origin, offering a more equitable approach. However, it remains sensitive to other factors, such as the individual’s temperament or cultural background. Orthogonal distances, another branch of FER, are designed to calculate the shift of facial landmarks from their position in a neutral expression to the position that signifies the facial expression28. This method shows promise in mitigating biases based on ethnic origin, offering a more equitable approach. However, it remains sensitive to other factors, such as the individual’s temperament or cultural background. Facial Action Unit Detection (FAUD). The third branch of FER is rooted in the behavior of the facial muscles that orchestrate any gesture or expression. In the Facial Action Coding System (FACS), each Action Unit (AU) corresponds to a specific, observable facial movement produced by the contraction or relaxation of one or more muscles29. The Emotion FACS (EMFACS) establishes the mapping between combinations of AUs and discrete emotions, a relationship that has demonstrated universality across humans and instilled confidence in the reliability of this method30 (Fig. 1). Facial Action Unit Detection (FAUD). The third branch of FER is rooted in the behavior of the facial muscles that orchestrate any gesture or expression. In the Facial Action Coding System (FACS), each Action Unit (AU) corresponds to a specific, observable facial movement produced by the contraction or relaxation of one or more muscles29. The Emotion FACS (EMFACS) establishes the mapping between combinations of AUs and discrete emotions, a relationship that has demonstrated universality across humans and instilled confidence in the reliability of this method30 (Fig. 1). Example of an Action Unit30: The blue-shaded region indicates the lateral portion of the frontalis muscle extending to the supraorbital ridge. When relaxed, the forehead skin remains smooth and the eyebrow follows the ridge (left). Upon contraction, oblique forehead wrinkles and eyebrow arching appear, defining this facial movement as AU2 (“Outer Brow Raiser”), which is associated with surprise and fear (right). Sketch created by Mariana Priego for this study. Beyond this three-way categorization, the distinction among these approaches is not only methodological but also conceptual. Semantic detection relies primarily on appearance-level cues extracted directly from facial images, making it operationally simple but also more vulnerable to superficial confounders such as age-related traits, ethnicity-linked facial morphology, illumination, or pose. Orthogonal-distance approaches shift the emphasis from appearance to geometry by measuring landmark displacement relative to a neutral facial configuration, which may reduce some appearance-related bias but depends heavily on accurate landmark localization and the availability of a meaningful neutral reference. By contrast, FAUD-based approaches aim to represent the muscular events underlying visible facial changes using Action Units (AUs), thereby providing an intermediate, physiologically grounded representation between the face image and the final emotion label. From a computational perspective, these paradigms also involve different trade-offs. Appearance-based FER methods can often be implemented as more direct end-to-end pipelines. In contrast, FAUD-based systems generally require either explicit AU detection or the learning of AU-aware intermediate representations, which may increase annotation demands, architectural complexity, and computational cost. However, this additional structure provides an important benefit: interpretability. Because FAUD-based methods operate on facial components explicitly linked to muscular activation and EMFACS-based emotion formation, they offer a more transparent and structurally meaningful account of why a given facial expression is recognized. In addition, AU-based representations are less dependent on superficial appearance cues alone, which makes them particularly suitable for examining how physiologically grounded intermediate modeling can contribute to FER. For this reason, FAUD-based FER warrants dedicated attention within the broader FER literature, especially in applications where traceability and human interpretability matter, such as healthcare, behavioral analysis, or human-centered intelligent systems. The comparative results analyzed later in this study (3.2) suggest that FAUD-based FER is especially advantageous when AU information is not treated merely as an auxiliary signal, but as a structurally meaningful intermediate stage for expression recognition. In particular, the proposals grouped under the FAUD-to-FER (F2F) scope achieve the highest average FER accuracy among the considered scopes, exceeding both conventional FER and IFA-oriented approaches. At the same time, this advantage should not be interpreted as universal. In real-time or resource-constrained scenarios, more compact appearance-driven solutions may remain preferable due to lower inference cost and simpler deployment requirements. Accordingly, the main trade-off is not accuracy alone, but the balance between structural interpretability and operational simplicity. Among these alternatives, the use of action units in FER has been the most extensively studied, largely because Ekman’s Facial Action Coding System (FACS) provides a standardized framework for representing facial muscular activity29. Given its relative universality and its support in comparative evaluations31, AU-based analysis constitutes the main methodological axis around which the present study is organized. The Emotion Facial Action Coding System (EMFACS) Facial muscles play a fundamental role in the expression of emotions. Their activation, controlled by electrical impulses originating both from the individual’s will and from subconscious impulses (such as those associated with emotions), generates observable gestures and expressions. Each specific muscle movement, whose contraction or elongation is reflected in the face through the region that overlies that muscle –such as raising an eyebrow–, is known as an Action Unit (AU). In the scientific literature on FAUD and FER, the Facial Action Coding System (FACS), developed by Paul Ekman and Wallace Friesen29, is often mentioned. However, it is important to note that FACS is essentially a descriptive system, designed to exhaustively code facial muscle movements into AUs without directly establishing their relationship with specific emotions. Its applications include, for example, the assessment of pain intensity or the detection of deception indicators. The relationship between AU combinations and emotions corresponds to the Emotion Facial Action Coding System (EMFACS)30, an adaptation of FACS that maps patterns of muscular activation to emotional categories. Originally, EMFACS considered six basic emotions: happiness, sadness, anger, fear, disgust, and surprise. Later, a seventh emotion, contempt, was incorporated, recognizing its relevance as a distinctive and universal expression, though with less consensus than the original six. This framework has been refined over time and validated across diverse contexts, confirming its universal applicability to the human species regardless of gender, ethnicity, culture, or age. For instance, surprise is universally associated with the raising of the eyebrows and the opening of the eyes and mouth (Fig. 2). Example of surprised face. Taken from the CK+ dataset32 in accordance with the CK and CK+ database user agreement. EMFACS, based on FACS, has found broad and successful applications in fields such as psychology, marketing, advertising, and even digital animation, where it is used to enhance the realism of characters. Furthermore, FACS certification is employed for legal purposes to determine the authenticity of expressed emotions. Given its practical utility, it is not surprising that a considerable amount of research in computer vision and FER projects is based on AU detection to recognize human emotions through facial expressions. It should be noted that there is no independent certification for EMFACS. In practice, FACS-certified coders, trained to objectively identify AUs, apply the EMFACS framework to recognize the basic emotions derived from specific AU combinations. Distinction between action units, facial landmarks, and regions of interest An essential aspect to consider when working with FAUD is the distinction between Action Units (AU), Facial Landmarks (FL), and Regions of Interest (ROI), since differentiating among them is crucial to avoid conceptual and methodological ambiguities (Fig. 3). In the left panel, the AUs activated when expressing surprise are shown: AU1 (Inner Brow Raiser, in blue), AU2 (Outer Brow Raiser, in green), AU5 (Upper Lid Raiser, in dark red), and AU26 (Jaw Drop, in yellow). The central panel displays the FLs used to track the displacement of facial features during the expression of surprise. The right panel highlights the ROIs that the observer –or the algorithm– focuses on to detect the AUs. The original image was taken from the CK+ dataset32, in accordance with the CK and CK+ database user agreement. AUs correspond to the activation of specific facial muscle groups, as defined in Paul Ekman’s Facial Action Coding System (FACS). Each AU is associated with a particular muscular contraction or elongation which, in combination with others, gives rise to facial expressions that reflect the individual’s emotions. In contrast, Facial Landmarks (FL) do not focus on muscular activity but on geometric reference points distributed across the facial surface, such as the corners of the lips, the tips of the eyebrows, or the chin. These landmarks describe the shape and position of facial features, and their relative displacement is useful for detecting changes associated with different expressions. Finally, Regions of Interest (ROI) refer to specific areas of the face where Action Units are located, usually delimited on the basis of landmarks. Their main purpose is to guide the algorithm to concentrate attention on physiologically relevant areas during AU detection. This strategy not only improves recognition accuracy by reducing noise from irrelevant facial regions but also optimizes computational resources by focusing analysis on key points of muscular activation. The valence-arousal two-dimensional model One drawback of EMFACS is that identification is limited to seven basic emotions, whereas a wide range of emotionality characterizes human nature. On the other hand, the correlation between AUs and emotions becomes increasingly confusing as the number of facial expressions to be recognized increases. Several researchers propose using the Valence-Arousal Two-Dimensional model (VA2D) to address the issue of nondiscrete emotions. It is a tool derived from the field of psychology, which identifies emotions based on two axes: the Valence, which represents the degree to which the emotion is positive or negative for the person, and the Arousal, an axis that represents the intensity of the emotion experienced. In Fig. 4, we can see how a negative valence with a high arousal could be indicated as frustration, and that the same negative valence but with a low arousal would be depression. Example of the Valence-Arousal Two-Dimensional model. Although VA2D was not used as a metric parameter for classifying the reviewed articles, studies employing this tool for FER frequently adopt unsupervised learning approaches. This model often leads to the identification of emotions that characterize broader emotional groups rather than specific, deterministic emotions33. Nevertheless, VA2D has demonstrated utility as a complementary tool to EMFACS for assessing the intensity of expressed emotion. Adherence to preferred reporting items for a systematic review and meta-analysis guidelines The systematic literature analysis required for the development of the comparative synthesis presented in this article is based on the Preferred Reporting Items for a Systematic Review and Meta-analysis (PRISMA) statement, which is a methodology developed by the PRISMA Group to provide standards and guidelines that ensure transparency and complete research reporting of scientific documentation for the development of systematic reviews and related evidence syntheses, seeking to strengthen its efficiency, precision and reliability34. Initially developed for use in the medical field, its efficiency has led to its widespread use in scientific research35. In adherence to these guidelines, the literature search was conducted exclusively in Scopus, selected for its comprehensive coverage, international recognition, and standardized structure as a scientific database. Its multidisciplinary scope includes leading journals in artificial intelligence, computer vision, and behavioral sciences, ensuring that the retrieved works are representative of the research relevant to this study. Furthermore, Scopus provides consolidated filtering and export tools that facilitate the traceability and reproducibility of the search process. By consensus among the authors, the use of Scopus alone was considered to provide an adequate balance between breadth of coverage, quality of sources, and methodological rigor. Future research may extend the scope of the search to additional databases to further reinforce the external validity of the findings. First phase: identification The Scopus academic database was chosen for article searching due to its high-quality research papers, which are recognized by the scientific and academic community, and its powerful, customizable search engine. The keywords used to identify the potential research were: “Facial Expression Recognition”. “Facial Expression Recognition”. “Action Units”. “Action Units”. “Review” was excluded. “Review” was excluded. The search returned 433 articles. Several filters were defined to sift through the returned articles. Articles published between 2019 and 2024. Articles published between 2019 and 2024. Written in English. Written in English. Conference paper and article as document type. Conference paper and article as document type. Computer science and engineering as research areas. Computer science and engineering as research areas. The articles identified as potential candidates for consideration in this article were 199. Second phase: screening Selection criteria were set to consider only those studies within the scope of this survey. The main ones are: Experiments conducted based on datasets created from videos and images of neurotypical people. Experiments conducted based on datasets created from videos and images of neurotypical people. The emotion prediction of the proposed models should be discrete, based on EMFACS. The emotion prediction of the proposed models should be discrete, based on EMFACS. Research focused on identifying facial expressions induced by physical sensations, such as pain, was excluded. Research focused on identifying facial expressions induced by physical sensations, such as pain, was excluded. Research focused on determining the intensity of emotions was excluded. Research focused on determining the intensity of emotions was excluded. For comparison purposes, only investigations using F1-Score as a measure of precision for FAUD and accuracy for FER were considered. This decision stems from the fact that state-of-the-art FER models typically identify the seven basic emotions discretely, making accuracy the most commonly used metric for comparative purposes. On the other hand, due to the inherent complexity of detecting and labeling AUs, datasets are often highly unbalanced, thus rendering the F1-Score the most frequently employed comparative metric. For comparison purposes, only investigations using F1-Score as a measure of precision for FAUD and accuracy for FER were considered. This decision stems from the fact that state-of-the-art FER models typically identify the seven basic emotions discretely, making accuracy the most commonly used metric for comparative purposes. On the other hand, due to the inherent complexity of detecting and labeling AUs, datasets are often highly unbalanced, thus rendering the F1-Score the most frequently employed comparative metric. To ensure consistency, the screening criteria were operationalized as follows: articles were first filtered by title and abstract to determine their compliance with the inclusion criteria, followed by an examination of the introduction to confirm alignment with the research scope. Disagreements regarding inclusion were resolved through consensus among the authors. This process allowed for the systematic application of the criteria and minimized subjective bias, resulting in 145 works being considered and 54 being excluded. Third phase: eligibility In addition to applying the inclusion and exclusion criteria, a systematic quality assessment of the eligible studies was performed. Each article was evaluated based on the clarity of the architectural diagram provided, as well as the presence of at least a schematic diagram of the implementation, together with the robustness of the reported metrics. Studies lacking a clear architectural representation, an implementation schematic, or presenting insufficiently supported results were excluded. Through this quality assessment, 95 studies were removed, ensuring that the final set of 50 included in this survey met a minimum standard of rigor and comparability. Fourth phase: included In conclusion, 50 research papers, listed in Table 12 (Appendix A), were selected for analysis in this study. A summary of the selection methodology is shown in Fig. 5. PRISMA searching process diagram. In the “Identification” phase, filters and keywords are established to identify potential research papers for consideration. In the “Screening” phase, criteria are established to exclude those papers that, based on the subject matter of their title, abstract, or introduction, are deemed out of scope for this survey. The “Eligibility” phase involves a full-text assessment of the papers that passed the screening, excluding those whose content falls outside the scope. Finally, the “Included” phase determines the research papers that will be systematically reviewed in this research. Taken together, the four phases of identification, screening, eligibility, and inclusion–supported by the operationalization of selection criteria and a systematic quality assessment–ensure that the survey process is transparent, reproducible, and methodologically rigorous, thereby providing a reliable foundation for the subsequent comparative analysis. The facial expression recognition through facial action unit detection Since FAUD has shown promising results as an intermediate strategy for identifying emotions through facial expressions31, multiple research efforts have been carried out in recent years to address the challenges involved in detecting AUs. First, there’s the intrinsic complexity in computer vision for accurately detecting both AUs and facial expressions. Second, there’s a need for readily available datasets with a sufficient number of labeled images for both FAUD and FER to train and test new models. The challenges of FER and FAUD Like all computer vision-based research, FER and FAUD models must be able to identify AUs and expressions despite dynamic lighting changes, varied image collection conditions, partial facial occlusion, and variations in face position relative to the observer. Furthermore, these models must avoid dependence on image resolution and be able to manage multiple planes or the presence of more than one face. Focusing on FER, the challenges are equally numerous and complex. These include a person’s inherent attributes such as age, gender, ethnic origin, temperament, or general appearance, as well as expressions that may be ambiguous or confusing36,37. Emotions themselves pose an intrinsic challenge, as some are easily confused by the average person, such as surprise and fear or anger and disgust38. Additionally, the dynamism of the human face and expressions must be considered. Expressions can last mere fractions of a second (known as microexpressions). Moreover, AUs are not only activated when we express emotion; they also occur in everyday situations like yawning or coughing, as an automatic response to external and fortuitous events such as a mosquito bite, or during gesticulation while speaking or singing39. This broader functional role of AUs has also supported FAUD applications beyond emotion recognition, including the inference of conversational context in natural social interaction3 and real-time driver drowsiness detection40. In such cases, AU activations represent noise that must be accounted for when using FAUD for FER. It is also crucial to recognize that the same AUs are utilized in different expressions; some always operate in correlation, while others are mutually exclusive41. Besides, to distinguish between FER and FAUD, it is necessary to start from a person’s neutral face to calculate AU changes. However, algorithms become more complex when a person is already expressing an emotion upon observation, without prior knowledge of their neutral face. While the FACS has demonstrated independence from the aforementioned personal attributes, its reliance on certified emotion experts for dataset labeling–particularly for frame-by-frame video annotation–incurs considerable person-hours, thereby increasing database development costs. Although engaging psychologists or non-verbal language experts is an option, and some datasets are even labeled through crowdsourcing, such methods raise concerns about labeling reliability due to potential biases from labeler fatigue and individual interpretive subjectivity42. An inherent effect of the previously described challenge is the existence of datasets that are not entirely labeled at both the AUs and facial expression levels, and this lack of labeling is not balanced across frame, AU, or emotion levels. Studies show that the accuracy of the same model can vary significantly depending on the dataset with which it is trained37. An important methodological implication of this imbalance is the potential propagation of error from FAUD to downstream FER when AU predictions are used as intermediate inputs. If some AUs are underrepresented during training, their detection may become less reliable, which can in turn distort the AU combinations used to infer emotions, particularly in cascaded or FAUD-to-FER (F2F) pipelines. This issue is especially relevant for low-frequency or highly transient AUs, whose omission or misclassification may alter the recognition of compound or fine-grained expressions. Although the reviewed studies do not usually quantify this propagation explicitly, several proposals address the imbalance problem through mitigation strategies such as Multi-label Random Over-Sampling (ML-ROS)43, Multi-Label Multi-Level learning (MLML)24, and weighted asymmetric loss44. In addition, prior work has shown that annotation bias and label uncertainty can significantly affect FER behavior, reinforcing the need to treat imbalance as a structural issue rather than only as a dataset characteristic45. Finally, the methods used for both FER and FAUD typically require hundreds of millions of parameters to achieve an acceptable level of accuracy. The above consumes a large amount of memory and processing46, often posing a significant limitation for researchers due to their limited available computing power. The challenge of datasets As in other areas of artificial intelligence, models that aim to recognize facial expressions require the use of training datasets. Specific datasets for FER are based on human faces expressing emotions, either in images or videos, which are labeled. Algorithms that use FAUD follow the same principle, being labeled at the Action Unit (AU) level. Currently, four methods are primarily used to label FER and FAUD datasets (Table 1): Certified FACS coders, those endorsed by the Paul Ekman Group for their expertise in the use and anatomically based interpretation of the Facial Action Coding System. Certified FACS coders, those endorsed by the Paul Ekman Group for their expertise in the use and anatomically based interpretation of the Facial Action Coding System. Experts in psychology and Affective Sciences, including researchers and graduate students with field experience in these areas. Experts in psychology and Affective Sciences, including researchers and graduate students with field experience in these areas. Crowdsourcing, requesting the support of participants via the internet, who can be trained for labeling or allowed to label based on their subjective perception and experience. Crowdsourcing, requesting the support of participants via the internet, who can be trained for labeling or allowed to label based on their subjective perception and experience. Algorithmic labeling, in which images or videos are commonly acquired from public internet platforms using emotion-based keywords and directly labeled upon download; additionally, emotion and AU classification can be performed using algorithms trained on labeled datasets. Algorithmic labeling, in which images or videos are commonly acquired from public internet platforms using emotion-based keywords and directly labeled upon download; additionally, emotion and AU classification can be performed using algorithms trained on labeled datasets. Table 1Labeling methodologies characteristics for FER and FAUD datasets.LabelersLabelingBiasCostCertified FACS CodersObjectiveLowVery HighExperts in psychologyObjectiveMediumHighCrowdsourcingSubjectiveHighLowAlgorithmAutomatedDependentLowLabeling quality is primarily assessed based on the annotators’ intrinsic qualifications. Bias stems from the degree to which an emotion or AU assignment relies on a proven methodology rather than subjective, even if professional, judgment. Labeling algorithms present a special case, as their outcomes are influenced by model bias, training data composition, and thresholding, beyond the programmer’s direct control. Finally, cost refers to the estimated fees charged by each annotator group. Labeling methodologies characteristics for FER and FAUD datasets. Labeling quality is primarily assessed based on the annotators’ intrinsic qualifications. Bias stems from the degree to which an emotion or AU assignment relies on a proven methodology rather than subjective, even if professional, judgment. Labeling algorithms present a special case, as their outcomes are influenced by model bias, training data composition, and thresholding, beyond the programmer’s direct control. Finally, cost refers to the estimated fees charged by each annotator group. Beyond labeling, the method by which dataset images and videos are acquired significantly influences their reliability and effectiveness in training robust algorithms. For instance, datasets constructed in controlled environments with experienced actors may achieve high accuracy in FER, while datasets using “in-the-wild” material downloaded from the Internet may offer better generalizability. Given the importance of these acquisition methods, datasets were grouped as described in the following subsections; see Fig. 6. Proposed classification system for datasets used in FER and FAUD. The classification is based on how the images and videos of faces expressing emotions were acquired. Posed datasets Posed datasets are those in which the participating subjects are asked to pose emotions, allowing FER and FAUD studies to be conducted under fully controlled conditions. In these cases, the participants lack an external stimulus enabling them to genuinely experience the emotion. The advantage of this type of dataset is that the researcher maintains total control of the environment in which the photos or videos are taken, making them especially useful when it is necessary to assess the authenticity of the expressed emotion (Table 2). Table 2Posed datasets summary with their main characteristics.DatasetUsedEmotionsLabelersAgeAudioAFEW4737ExpertsChildren–elderlyYesCK+32297Certified Coders18–50NoMMI48117Certified Coders19–62NoOulu-CASIA4967Experts23–58NoUsed = number of research papers that used the dataset; Emotions = number of labeled emotions in the dataset; Labelers = type of annotators responsible for labeling; Age = age range of the participants; Audio = whether the dataset includes audio. CK+ contains sequences starting with a neutral expression and progressing to peak intensity; neutral is not included as a separate labeled category in the emotion set.In all cases, efforts are made to ensure the inclusion of diverse ethnic groups and balanced gender representation. Posed datasets summary with their main characteristics. Used = number of research papers that used the dataset; Emotions = number of labeled emotions in the dataset; Labelers = type of annotators responsible for labeling; Age = age range of the participants; Audio = whether the dataset includes audio. CK+ contains sequences starting with a neutral expression and progressing to peak intensity; neutral is not included as a separate labeled category in the emotion set. In all cases, efforts are made to ensure the inclusion of diverse ethnic groups and balanced gender representation. The Acted Facial Expressions in-the-Wild (AFEW) dataset consists of frame sequences extracted from movies and TV series and has been updated annually since 2013 as part of the Emotion Recognition in the Wild Challenge (EmotiW). Version 7.0 includes 773 training sequences, 383 validation sequences, and 653 testing sequences, covering subjects of diverse genders, ethnicities, and ages. Facial expressions were annotated by certified experts into six emotions plus neutral. Methodological details are provided in the original publication47, while access requirements are available on the webpage of the College of Engineering and Computer Science at the Australian National University: https://users.cecs.anu.edu.au/~few_group/AFEW.html. The Extended Cohn-Kanade Dataset (CK+) expands the original Cohn-Kanade dataset and contains 593 image sequences from 123 individuals of different genders, aged 18 to 50, displaying the six basic emotions as well as contempt. Each sequence evolves from a neutral expression to peak intensity, although only 327 sequences include AU annotations. Methodological details are described in the original publication32, and access requirements are available on the University of Pittsburgh webpage: https://sites.pitt.edu/~emotion/ck-spread.htm. The MMI Facial Expression Database (MMI) includes 326 image sequences from 32 individuals aged 19 to 62, of diverse genders and ethnicities, portraying six basic emotions plus neutral. Each sequence starts from a neutral face, reaches peak expression, and returns to neutral, with labeling performed by certified FACS experts. Methodological details are provided in the original publication48, while access requirements are available on the webpage of the Intelligent Behaviour Understanding Group at Imperial College London: https://ibug.doc.ic.ac.uk/resources/mmi-facial-expression-database/. The Oulu-CASIA NIRVIS Facial Expression Database (Oulu-CASIA) contains 2,880 sequences from 80 individuals of different genders and ethnicities, recorded in both visible (VIS) and near-infrared (NIR) conditions. Each sequence begins with a neutral face and ends at the peak expression, and frames are labeled with six basic emotions plus neutral. Methodological details are reported in the original publication49, while access requirements are available on the webpage of the Center for Machine Vision and Signal Analysis at the University of Oulu: https://www.oulu.fi/cmvs/node/41316. Spontaneous datasets Spontaneous datasets refer to cases where the subject is genuinely experiencing the emotion and expresses it through body language. These datasets can be divided into two subgroups: contextual and in-the-wild. Contextual datasets Contextual datasets are collected in controlled environments where participants are exposed to emotion-inducing videos or tasks (Table 3). Table 3Contextual datasets summary with their main characteristics.DatasetUsedEmotionsLabelersAgeAudioBP4D50158Certified Coders18–29NoDISFA51150Certified Coders18–50NoFG-Emotions52133ExpertsTeen-elderlyNoGFT5310Experts21–28YesUsed = number of research papers that used the dataset; Emotions = number of labeled emotions in the dataset; Labelers = type of annotators responsible for labeling; Age = age range of the participants; Audio = whether the dataset includes audio. BP4D participants were exposed to eight target emotions, but the dataset is annotated primarily at the AU level; neutral states are present as baselines but not as a labeled class. DISFA is annotated exclusively at the AU level with intensity values (0–5), not with discrete emotions. GFT focuses on genuine vs. fake behavior, not on discrete emotion categories. Contextual datasets summary with their main characteristics. Used = number of research papers that used the dataset; Emotions = number of labeled emotions in the dataset; Labelers = type of annotators responsible for labeling; Age = age range of the participants; Audio = whether the dataset includes audio. BP4D participants were exposed to eight target emotions, but the dataset is annotated primarily at the AU level; neutral states are present as baselines but not as a labeled class. DISFA is annotated exclusively at the AU level with intensity values (0–5), not with discrete emotions. GFT focuses on genuine vs. fake behavior, not on discrete emotion categories. The Binghamton-Pittsburgh 4D Spontaneous Facial Expression Database (BP4D) comprises more than 140,000 2D/3D frames from 328 dynamic sequences collected from 41 participants aged 18 to 29. Participants performed tasks designed to elicit eight target emotions; however, the annotations are provided mainly at the Action Unit (AU) level by FACS-certified coders rather than as discrete emotion labels. Methodological details are described in the original publication50, while access requirements are available on the webpage of Jeffrey F. Cohn’s research group at the University of Pittsburgh: https://www.jeffcohn.net/resources/. The Denver Intensity of Spontaneous Facial Action (DISFA) dataset contains 130,815 frames extracted from videos of 27 individuals aged 18 to 50, recorded while they watched videos intended to elicit spontaneous emotional reactions. FACS-certified coders annotated 12 AUs with intensity values from 0 to 5, but no discrete emotion labels are provided. Methodological details are reported in the original publication51; access can be requested directly from the original authors at the University of Denver. The Fine-Grained Facial Expression Database (FG-Emotions) extends the conventional six-emotion framework to 33 categories based on a hierarchical taxonomy. It contains 1,491 video sequences extracted from television series and movies, from which 10,371 frames were obtained for static-image analysis. Each sequence progresses from a neutral expression to peak intensity and then returns to neutral, and annotations were performed by trained experts. Methodological details are provided in the original publication52; access can be requested directly from the original authors. The Sayette Group Formation Task (GFT) dataset was developed to study the distinction between genuine and simulated behavior in facial and vocal expressions. It includes subjects of diverse ages, genders, and ethnicities recorded under controlled conditions. The data were annotated by experts in terms of authenticity (genuine vs. fake), rather than discrete emotions, together with relevant AUs. Methodological details are available in the original publication53, while the dataset can be downloaded directly from Jeffrey F. Cohn’s research group webpage at the University of Pittsburgh: https://www.jeffcohn.net/resources/. In-the-wild datasets In-the-wild datasets consist of recordings captured in natural, uncontrolled environments, where individuals express emotions spontaneously. These datasets typically include images and videos sourced from publicly available online content, such as personal photos, home videos, film scenes, television broadcasts, music videos, and news footage (Table 4). Table 4In-the-wild datasets summary with their main characteristics.DatasetUsedEmotionsLabelersAgeAudioAffectNet5488CrowdsourcingYoung-elderlyNoAff-Wild25557CrowdsourcingChildren–elderlyYesEmotioNet5627AlgorithmsNot specifiedNoFER20135767CrowdsourcingChildren–elderlyNoRAF-DB58157CrowdsourcingYoung–elderlyNoSEMAINE59335Experts22–60YesSFEW6077ExpertsChildren–elderlyNoUsed = number of research papers that used the dataset; Emotions = number of labeled emotions in the dataset; Labelers = type of annotators responsible for labeling; Age = age range of the participants; Audio = whether the dataset includes audio. EmotioNet and RAF-DB also include compound emotion categories in addition to the basic set with neutral. The original dataset description does not provide a precise age interval. In-the-wild datasets summary with their main characteristics. Used = number of research papers that used the dataset; Emotions = number of labeled emotions in the dataset; Labelers = type of annotators responsible for labeling; Age = age range of the participants; Audio = whether the dataset includes audio. EmotioNet and RAF-DB also include compound emotion categories in addition to the basic set with neutral. The original dataset description does not provide a precise age interval. AffectNet is a large-scale database containing approximately one million internet images of individuals across a broad age range. The images were retrieved using emotion-related keywords and manually annotated through crowdsourcing into eight categories: the six basic emotions, neutral, and contempt. In addition to categorical labels, the dataset provides dimensional annotations for valence and arousal. Methodological details are described in the original publication54; access can be requested directly from the original authors. Aff-Wild2 consists of millions of audiovisual frames collected from internet sources such as YouTube, interviews, movies, and television. Its annotations include seven categorical emotions (the six basic emotions plus neutral), AUs, and valence-arousal dimensions, labeled through trained crowdsourcing. The dataset has served as a central benchmark in the ABAW challenge series. Methodological details are provided in the original publication55, while access requirements are available on the webpage of the Intelligent Behaviour Understanding Group at Imperial College London: https://ibug.doc.ic.ac.uk/resources/aff-wild2/. EmotioNet contains approximately one million static facial images collected from the internet. Its labels were generated automatically using algorithms trained on AU-emotion relationships. The dataset includes the six basic emotions, neutral, and some compound categories, with a strong emphasis on both AU-level and discrete emotion annotations. Methodological details are reported in the original publication56; access can be requested directly from the original authors. The Facial Expression Recognition 2013 (FER2013) dataset was collected through the Google Image Search API and contains 35,887 grayscale facial images annotated via crowdsourcing into the six basic emotions plus neutral. The dataset is partitioned into training, validation, and test sets. Methodological details are available in the original publication57, and the dataset can be downloaded from its official Kaggle repository: https://www.kaggle.com/datasets/msambare/fer2013. The Real-world Affective Face Database (RAF-DB) includes 29,672 internet images of individuals from diverse demographic groups. The images were annotated through crowdsourcing into the six basic emotions plus neutral, as well as 11 compound emotion categories. Each image is independent and does not belong to a sequence. Methodological details are described in the original publication58, while access requirements are available on the official RAF-DB webpage: http://www.whdeng.cn/RAF/model1.html. The SEMAINE dataset was developed to study emotional interaction in spontaneous and semi-structured conversations. It comprises 959 audiovisual recordings of people interacting with either another person or a virtual agent. Expert annotators labeled both basic and complex emotions, including neutral, as well as AUs. Methodological details are provided in the original publication59; access can be requested directly from the original authors. The Static Facial Expressions in-the-Wild (SFEW) dataset consists of keyframes selected by clustering facial landmarks. Its most widely used version, SFEW 2.0, includes 958 training frames, 436 validation frames, and 372 testing frames, annotated into the six basic emotions plus neutral. Methodological details are reported in the original publication60, while access requirements are available on the FEW project webpage at the Australian National University: https://cs.anu.edu.au/few/AFEW.html. This wide variety of datasets –whether posed, contextual, or in-the-wild– highlights the diversity of acquisition conditions and annotation strategies that must be considered when designing and benchmarking FER and FAUD models. Most relevant research on FER and AUs from 2019 to 2024 In the full-text evaluation, it was observed that, although the selection criteria targeted studies jointly addressing Action Units (AU) and Facial Expression Recognition (FER), the resulting literature could be grouped into three main categories: studies primarily focused on Facial Action Unit Detection (FAUD), studies focused on FER, and studies proposing multi-task solutions. Additionally, some real-time proposals were identified, unlike most research, which typically restricts experimentation to benchmark datasets. Due to their comparatively low number, these studies were analyzed as an additional group regardless of whether their primary focus was FER or FAUD. The 2019-2024 period was selected to focus the analysis on a recent and methodologically consistent stage of the literature, capturing the consolidation of contemporary AU-informed FER approaches -including attention-based, graph-based, Transformer-based, and hybrid architectures- within a sufficiently coherent technological context for structured synthesis. Summary of FAUD research FAUD is distinctive in that it targets subcutaneous muscle activity, which cannot be directly observed or measured. Consequently, detection must rely on the observable effects of such activity on facial appearance29, as illustrated in Fig. 7. Several of the researchers included in this survey (see section "Adherence to preferred reporting items for a systematic review and meta-analysis guidelines") have grounded their approach to FER in FAUD; their corresponding proposals are outlined in this section (Table 5). Examples showing the correlation between AUs and their FACS-assigned numbers. Taken from the CK+ dataset32 in accordance with the CK and CK+ database user agreement. Table 5Architecture components utilized in FAUD-based approaches, grouped by algorithmic category.CategoryProposed ModelArchitecture ComponentsBayesian/probabilisticAUD-EA(gBN)61BN, ANNwang201826LRBNFRBM+41FRBM+he201938BN, AAM, GdGraph-basedSRERL25GGNNFE-AURDGCN62cGAN, GCNCNN/attention/region-awareEfficientNet-B015CNN, MLPSJMT63CNN, lossAWS-Net64CNN, lossMTAA65DLN, DLNATF66CNNSituTech43FPN, SSH, CAM, BCE loss, ML loss, ML-ROSAFFDEX67R-CNN, CNN, BCE lossakay202268GoogLeNet, MHI, CNN, MLP, SoftmaxGA69GAM, MSE loss, CE losszhao202370Deep-Block, Square lossMTLRT71CNN, LSTM, ML-CEAUCaps++72DenseNet, CapsNetGTLE-Net12Triplet loss, CNNAEL-NN73CNN, Attention ModTransformer/encoder-basedPMVT11ViT, SGD, MLPPOSTER-AU44WA loss, CTE, FPN, MLPLibreFace23MediaPipe, MAE, ResNet-18, MSE loss, CE loss, KDzhang202374MAE, CE loss, TMF, GFyang202475RAE, TC&FR, Encoder, ALGenerative/weakly supervisedDSGAN42GANDGAN13GAN, AdamFG-Net76MLP, Pyramid CNN, MSE lossClassical ML/optimization-orientedli201924AAM, PCA, MLML, Lovász, ADMM, MMIB-SF, LDACategory refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. Architecture components utilized in FAUD-based approaches, grouped by algorithmic category. Category refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. A wide range of FAUD studies have explored different strategies to improve AU detection, with most recent proposals converging on three recurring directions: stronger local feature extraction, explicit modeling of AU relationships, and improved robustness under weak supervision or in-the-wild conditions. Several works enhance spatial representation through deep backbone architectures. For example, AUCaps++72 combines DenseNet with CapsNet to better capture both facial features and their spatial relationships, whereas the approach in70 divides the face into regions and introduces a Deep-Block module on top of MTCNN to emphasize local appearance changes and inter-region influence. Along a similar line, MTLRT71 avoids landmark-based AU localization and instead preserves convolutional feature detail while using LSTM layers to model relationships among local regions. FRBM+41 also exploits AU dependencies, but from a probabilistic perspective, by modeling correlations and exclusions between AUs through a four-layer RBM. Other proposals focus on attention and contextual modeling. Guided Attention69 restricts the search space to AU-relevant facial areas by coupling DenseNet-121 with AU-specific attention masks, thereby reducing unnecessary computation. PMVT11 incorporates a Vision Transformer after ResNet-34-based feature extraction to improve multi-scale AU analysis. Likewise, the method in75 integrates spatial and temporal information through ROI extraction, regional attention, temporal feature combination, and a Transformer encoder, highlighting the growing interest in spatiotemporal AU modeling rather than frame-by-frame analysis alone. Another important line of work addresses data imbalance, annotation scarcity, and weak supervision. The ABAW 2022 second-place model in43improves AU, expression, and face detection through an enhanced ResNet architecture complemented by FPN, SSH, CAM, and multi-label balancing strategies. In contrast24, proposes a lighter machine-learning-oriented alternative based on AAM, PCA, structured multi-label learning, and LDA, explicitly aiming to reduce the computational burden of deep models. GTLE-Net12 combines global expression encoding with local AU extraction to improve robustness under appearance variations, while DGAN13, AUE-CRL14, and FG-Net76 address limited or missing AU labels through generative modeling, pseudo-labeling, or semantic feature learning. Finally, some studies emphasize deployment-oriented or in-the-wild FAUD. AFFDEX67 integrates face detection, landmark localization, and AU recognition into an SDK-oriented pipeline designed for practical use, whereas68 proposes a multi-level stacking strategy that combines frame-based, segment-based, and transition-based classifiers to capture AU dynamics more comprehensively. Taken together, the reviewed FAUD literature suggests that performance improvements are rarely due to a single architectural choice alone. Instead, stronger results tend to emerge when local AU-sensitive representations, inter-AU dependency modeling, temporal context, and supervision-aware learning strategies are combined in a coherent manner. Summary of FER research The semantic relationships among AUs -including co-activation, inhibition, and mutual exclusivity- can improve facial representation for FER. For example, AU1 (raised eyebrows) is commonly associated with surprise, whereas AU4 (frowning) is more consistent with anger, making both units semantically opposed (Fig. 8). The proposals discussed in this section exploit such AU-level structure to refine FER and are summarized in Table 6. Action units exclusion example. On the left, the surprise causes the frontalis muscle to contract (AU1), resulting in forehead wrinkles. On the right, the anger causes the corrugator supercilii muscle to contract (AU4), resulting in the person frowning. These muscles cannot be contracted at the same time. Taken from the CK+ dataset32 in accordance with the CK and CK+ database user agreement. Table 6Architecture components utilized in FER approaches, grouped by algorithmic category.CategoryProposed ModelArchitecture ComponentsBayesian/probabilisticFER-IK(gBN)61BN, ANNwang201977DBN, MLP, MSE losshe201938BN, AAM, Gdwang2019B78BN, Ensemble, CVCNN/attention/region-awareDAUGN79CNNMSAU-Net80CNNSAANet81CNN, RNNSJMT63CNN, lossAWS-Net64CNN, lossMiniExpNet+46HLBP, CNNMTAA65DLN, DLNhong202082LSTM, CAM, CNNR-FENet83R-CNNAEL-NN73CNN, Attention ModGraph-basedFE-AURDGCN62cGAN, GCNkim202384LAM, GCN, MLPULC-AG36DNN, GCNTransformer/encoder-basedPOSTER85CTE, FPN, MLPTESGCN86DAE, ChebConv, EncoderLibreFace23MediaPipe, MAE, ResNet-18, MSE loss, CE loss, KDClassical ML/lightweightyao202187AL, SVMFER(rt)88HOG, CECLMHOG-XGBoost89HOG, XGBoostperveen202090DRMFbelhouchette201991Viola-Jones, AAM, BMatching, SOMWeakly supervised/domain adaptationAUE-CRL14MTCNN, LCM, MSE lossAdaFER37RetinaFace, CE Loss, KL Loss, Triplet lossMTAC92CCC, CE lossCategory refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. Architecture components utilized in FER approaches, grouped by algorithmic category. Category refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. A first line of work models AU relationships explicitly through probabilistic or relational structures. For instance26, uses a hierarchical Bayesian framework to capture dependencies among AUs and between AUs and facial expressions, while SRERL25 organizes AUs as a graph processed by a Gated Graph Neural Network (GGNN) within a multi-scale CNN. Similarly, DAUGN79 generates facial AU maps to encode both appearance and geometry, allowing a CNN to learn hybrid FER features. Other methods combine AU evidence with geometric cues or region-based analysis. In77, facial landmarks and AUs are jointly exploited through a CNN and a Dynamic Bayesian Network, whereas R-FENet83 divides the face into three ROIs and fuses regional and global features to improve FER. A second group of studies aims at fine-grained FER by strengthening attention, temporal modeling, or graph reasoning. MSAU-Net and its temporal extension TMSAU-Net80 use AU-aware multi-scale representations to recognize a larger variety of expressions. SAANet81combines spatial AU attention with temporal modeling through parallel attention modules and recurrent analysis. More recently, graph-based FER has gained relevance84: represents AU correlations through facial graphs processed with Graph Convolutional Networks (GCNs), while TESGCN86 combines ROI extraction, graph construction, graph convolution, and Transformer encoding to improve contextual expression modeling. Another important direction addresses weak supervision, uncertain labels, or domain shift. In87, active learning is used to identify AUs and then classify expressions with an SVM. ULC-AG36 employs auxiliary AU graphs to detect and correct uncertain expression labels during training, whereas AdaFER37 uses AU-guided pseudo-labeling and triplet-based adaptation to improve FER under domain mismatch and limited annotation quality. Some studies approach FER from more specialized or lightweight perspectives. The method in82argues that landmarks alone are insufficient and therefore complements AU-level reasoning with a semantic facial description generated from encoded visual features. In contrast90, reduces the number of relevant AUs by relying on L-FACS rules rather than the full FACS framework, aiming to simplify the recognition process and reduce computational burden. Likewise, DSGAN42 combines partially labeled AUs with fully labeled expressions in a semi-supervised generative setting, while91 shows that simpler machine-learning pipelines based on AAM, motion estimation, and self-organizing maps can also support AU-informed FER. From a comparative perspective, the reviewed FER literature indicates that AU-based performance gains are not tied to a single architecture. Instead, stronger results tend to arise when AU semantics are incorporated through structured dependency modeling, region-aware analysis, temporal reasoning, or supervision-aware learning. This suggests that the main advantage of AU-informed FER lies in the quality of the intermediate facial representation rather than in any single backbone alone. Summary of multi-task solutions research Although FER research is often divided between image-based recognition and AU-based analysis, several multi-task approaches seek to improve performance by combining both within a shared framework (Table 7). In general, these methods exploit complementarity between global facial appearance, AU information, and sometimes additional affective cues such as intensity, valence-arousal, or even audio. Table 7Architecture components utilized in multi-task approaches, grouped by algorithmic category.CategoryProposed ModelArchitecture ComponentsBayesian/probabilistic fusionFER-IK(gBN)61BN, ANNhe201938BN, AAM, Gdwang2019B78BN, Ensemble, CVShared CNN-based multi-task learningSJMT63CNN, lossAWS-Net64CNN, lossMTAA65DLN, DLNATF66CNNGraph/generative integrationFE-AURDGCN62cGAN, GCNTransformer/cross-fusionPOSTER85CTE, FPN, MLPPOSTER-AU44WA loss, CTE, FPN, MLPMultimodal/temporal fusionzhang202374MAE, CE loss, TMF, GFCategory refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. Architecture components utilized in multi-task approaches, grouped by algorithmic category. Category refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. A first group of studies combines convolutional and probabilistic models. For example78, integrates CNN-derived features with facial landmarks and a Bayesian Network (BN) for AU inference, and then fuses both outputs through an ensemble strategy to improve emotion prediction. Similarly, FER-IK(gBN)61 estimates FER directly from the image and independently from detected AUs, and then refines both through a BN. Other works focus on shared deep representations, such as SJMT63, which jointly learns facial expressions and partially labeled AUs through a selective multi-task loss, and AWS-Net64, which uses two VGG19 branches connected by an adaptive weight-sharing mechanism. A second line of work extends unified architectures to multiple related affective tasks. POSTER85 combines geometric facial landmarks and semantic image features through a cross-fusion Transformer encoder for FER, while its later adaptation, POSTER-AU44, modifies the output design and loss function to support AU detection under class imbalance. Along the same direction, MTAA65 jointly models FAUD, FER, and emotion intensity through hierarchical correlations among AUs, expressions, and valence-arousal cues. Other proposals address specific practical limitations. ATF66 introduces adversarial multi-task training to compensate for the limited number of AU-labeled samples, while FE-AURDGCN62generates neutral faces through a conditional GAN and models AU dependencies with a GCN to reduce ambiguity when neutral references are unavailable. In a more cognitively inspired direction38, proposes a hierarchical Bayesian architecture that links facial features, muscles, AUs, and expressions in an attempt to emulate the human interpretation process. Finally, some recent work expands the multi-task paradigm beyond visual cues alone. In74, audio is incorporated to refine AU, FER, and valence-arousal prediction through temporal and multimodal fusion, illustrating a broader trend toward integrated affective analysis rather than isolated facial classification. Collectively, the reviewed multi-task literature suggests that the main advantage of these approaches lies in their ability to exploit complementary supervision sources. Rather than treating AU detection and FER as separate problems, multi-task models tend to improve performance when they jointly leverage facial appearance, AU structure, probabilistic dependencies, and auxiliary affective information in a coherent learning scheme. Summary of real-time research A relevant practical objective of FER research is real-time deployment (Table 8). In this direction, several studies seek to reduce inference time while preserving acceptable recognition accuracy, often by simplifying feature extraction, restricting the analysis to AU-relevant regions, or using lightweight architectures. Table 8Architecture components utilized in real-time approaches, grouped by algorithmic category.CategoryProposed ModelArchitecture ComponentsClassical ML/lightweightFER(rt)88HOG, CECLMHOG-XGBoost89HOG, XGBoostCNN-basedEfficientNet-B015CNN, MLPMiniExpNet+46HLBP, CNNTransformer/distillation-basedLibreFace23MediaPipe, MAE, ResNet-18, MSE loss, CE loss, KDCategory refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. Architecture components utilized in real-time approaches, grouped by algorithmic category. Category refers to the dominant algorithmic or learning paradigm used to group each proposal. Architecture Components summarizes the main computational modules, models, losses, or techniques reported for each proposal. See Table 13 in Appendix A for detailed descriptions of the listed components. One of the simplest proposals is HOG-XGBoost89, which combines CECLM-based AU and landmark extraction with XGBoost classification, achieving an average processing time of 30 ms on modest hardware. However, it was evaluated on static images, which limits its direct applicability to fully dynamic real-time scenarios. FER(rt)88 adopts a more complete deployment-oriented architecture, connecting a Raspberry Pi camera system to a cloud-based FER engine. In this pipeline, HOG, CECLM, PCA, and a pre-trained CNN are combined to estimate expressions in approximately 38 ms per sample. Other studies target lightweight embedded or mobile deployment. EfficientNet-B015, proposed in the context of the ABAW challenge, fine-tunes a pre-trained CNN to obtain emotional embeddings that are later combined with facial landmarks and processed by an MLP. The model was successfully deployed on an Android device for front-camera emotion recognition. Similarly, MiniExpNet+46 defines ROIs from facial landmarks and extracts compact HLBP descriptors before feeding a reduced CNN enhanced with attention and self-distillation, reaching 5.42 ms per frame on CPU. In a multi-task setting, LibreFace23 combines MAE-based AU learning, ResNet-18-based FER, and knowledge distillation to reduce computational cost, achieving an average inference time of 37.2 ms on CPU-only hardware. In practical terms, the reviewed real-time literature suggests that practical FER depends less on architectural complexity alone than on efficient region selection, compact feature representation, and deployment-aware optimization. Lightweight CNN-based or hybrid pipelines remain especially attractive for edge and mobile scenarios, whereas more sophisticated multi-task or Transformer-based models require additional compression strategies to remain feasible in real time. Comparisons Since the present survey aims to compare and evaluate research focused on facial expression recognition through action unit (AU) detection, and to define the relevant studies accordingly (Table 9) based on PRISMA guidelines (see section "Adherence to preferred reporting items for a systematic review and meta-analysis guidelines"), we observed that some authors prioritized AU detection over expression recognition. This decision was justified by two premises: first, that more precise and reliable AU detection leads to greater accuracy in facial expression recognition; and second, the inherent complexity of detecting tiny AU areas within the highly dynamic human face. Therefore, while all included works share the common objective of improving and innovating methodologies for facial expression recognition, they can be grouped in different ways: Categorizing the reviewed research papers into three groups based on the primary purpose of their implemented algorithms: Facial Action Unit Detection (FAUD): Focused on AU detection.Facial Expression Recognition (FER): Focused on FER.Hybrid Approach: Addressing both FAUD and FER. Categorizing the reviewed research papers into three groups based on the primary purpose of their implemented algorithms: Facial Action Unit Detection (FAUD): Focused on AU detection.Facial Expression Recognition (FER): Focused on FER.Hybrid Approach: Addressing both FAUD and FER. Facial Action Unit Detection (FAUD): Focused on AU detection. Facial Action Unit Detection (FAUD): Focused on AU detection. Facial Expression Recognition (FER): Focused on FER. Facial Expression Recognition (FER): Focused on FER. Hybrid Approach: Addressing both FAUD and FER. Hybrid Approach: Addressing both FAUD and FER. By the ultimate scope of the proposals, as categorized in our survey, falling into four distinct groups: Facial Expression Recognition (FER): Proposals that focus directly on FER without explicitly using Action Units as an intermediate stage.FAUD (Enhanced Detection): Proposals that specifically improve Action Unit detection by utilizing FER.Improve FER Accuracy (IFA): Proposals aiming to enhance FER accuracy based on detected AUs.FAUD-to-FER (F2F): Proposals that leverage FAUD as a foundational step to achieve FER. By the ultimate scope of the proposals, as categorized in our survey, falling into four distinct groups: Facial Expression Recognition (FER): Proposals that focus directly on FER without explicitly using Action Units as an intermediate stage.FAUD (Enhanced Detection): Proposals that specifically improve Action Unit detection by utilizing FER.Improve FER Accuracy (IFA): Proposals aiming to enhance FER accuracy based on detected AUs.FAUD-to-FER (F2F): Proposals that leverage FAUD as a foundational step to achieve FER. Facial Expression Recognition (FER): Proposals that focus directly on FER without explicitly using Action Units as an intermediate stage. Facial Expression Recognition (FER): Proposals that focus directly on FER without explicitly using Action Units as an intermediate stage. FAUD (Enhanced Detection): Proposals that specifically improve Action Unit detection by utilizing FER. FAUD (Enhanced Detection): Proposals that specifically improve Action Unit detection by utilizing FER. Improve FER Accuracy (IFA): Proposals aiming to enhance FER accuracy based on detected AUs. Improve FER Accuracy (IFA): Proposals aiming to enhance FER accuracy based on detected AUs. FAUD-to-FER (F2F): Proposals that leverage FAUD as a foundational step to achieve FER. FAUD-to-FER (F2F): Proposals that leverage FAUD as a foundational step to achieve FER. Table 9Proposals’ summary with the principal elements of their comparative analysis.Proposed ModelFAUDFAUD-dFAUD-F1FERFER-dFER-AccFinal ScopeSRERL25Y20.60NFAUDDAUGN79NY377.7F2FMSAU-Net80NY975.5F2FSAANet81NY578.6F2FFER-IK(gBN)61NY490.5IFAAUD-EA(gBN)61Y30.63NFAUDyao202187NY193.0F2FFER(rt)88NY287.7F2Fwang201826Y30.72NFAUDDSGAN42Y20.67NFAUDEfficientNet-B015Y10.54NFAUDHOG-XGBoost89NY286.6FERSJMT63Y10.73Y270.0FERAWS-Net64Y10.58Y185.2FERMiniExpNet+46NY391.3IFAMTAA65Y10.15Y180.6IFAFRBM+41Y30.70NFAUDFE-AURDGCN62Y20.93Y294.7F2FATF66Y20.52NFAUDli201924Y20.67NFAUDSituTech43Y10.50NFAUDPMVT11Y20.62NFAUDkim202384NY482.7F2FAFFDEX67Y10.63NFAUDAUE-CRL14NY266.9F2FULC-AG36NY275.5IFAAdaFER37NY269.4F2Fhong202082NY369.2F2Fwang201977NY196.8IFAR-FENet83NY259.0IFAakay202268Y30.72NFAUDGA69Y20.57NFAUDperveen202090NY377.1F2Fbelhouchette201991NY288.5F2Fhe201938Y10.89Y191.7F2FPOSTER85NY383.7FERPOSTER-AU44Y20.67NFAUDzhao202370Y20.70NFAUDLibreFace23Y20.63Y266.3IFAzhang202374Y10.56NFAUDDGAN13Y20.67NFAUDMTLRT71Y20.63NFAUDwang2019B78NY282.8IFAAUCaps++72Y10.53NFAUDFG-Net76Y20.65NFAUDGTLE-Net12Y20.67NFAUDTESGCN86NY375.8F2Fyang202475Y10.66NFAUDMTAC92NY475.5IFAAEL-NN73Y30.86Y293.6F2FColumn meanings: FAUD, whether an FAUD model is proposed; FAUD-d, number of datasets used for testing; FAUD-F1, average F1-score obtained; FER, whether a FER model is proposed; FER-d, number of datasets used for testing; FER-Acc, average accuracy obtained; Final Scope, proposal category within the comparative taxonomy adopted in this survey (see Section "Comparison based on the proposal final scope"), where FAUD refers to studies focused on Facial Action Unit Detection, FER to studies focused directly on Facial Expression Recognition, IFA to studies that use Facial Action Units to improve FER accuracy, and F2F to studies that explicitly implement FAUD as an intermediate stage for FER. Proposals’ summary with the principal elements of their comparative analysis. Column meanings: FAUD, whether an FAUD model is proposed; FAUD-d, number of datasets used for testing; FAUD-F1, average F1-score obtained; FER, whether a FER model is proposed; FER-d, number of datasets used for testing; FER-Acc, average accuracy obtained; Final Scope, proposal category within the comparative taxonomy adopted in this survey (see Section "Comparison based on the proposal final scope"), where FAUD refers to studies focused on Facial Action Unit Detection, FER to studies focused directly on Facial Expression Recognition, IFA to studies that use Facial Action Units to improve FER accuracy, and F2F to studies that explicitly implement FAUD as an intermediate stage for FER. Another important element that should be considered beyond the groupings we propose is the number of datasets with which the authors test their models. This is because, no matter how many people a dataset includes and how varied it is in terms of ethnic origins, ages, and cultures, it will never be large enough to encompass all existing human beings. Therefore, the greater the number of datasets used in the same research, the greater the reliability of the model’s efficiency when this is generalized. Finally, for comparison purposes, F1-Score for FAUD and the accuracy for FER are used because state-of-the-art FER models typically identify the seven basic emotions discretely, making accuracy the most commonly used metric for comparative purposes. On the other hand, due to the inherent complexity of detecting and labeling AUs, datasets are often highly unbalanced, thus rendering the F1-Score the most frequently employed comparative metric. In cases where the same proposal yields more than one result per dataset (for example, when static images and video are evaluated separately), the average of these results was taken. Although the statistical and artificial intelligence tools used in the different proposals were initially extracted to evaluate their possible impact on the reported results, it was observed that hybrid technologies were employed in the vast majority of the studies, making it extremely difficult to isolate the individual contribution of each tool. It was also noted that the authors used different criteria when describing the implemented tools in their papers: some provided detailed architectural descriptions, while others reported only the tools associated with the innovative component of the model. For these reasons, no direct comparative analysis based on the implemented tools was carried out. Instead, they are summarized in Table 13 (Appendix A) so that future researchers may use this information as a reference. Finally, pre-trained models are reported separately as backbones when they were used to facilitate image preprocessing or the identification of relevant facial regions or landmarks (Table 14, Appendix A). Comparison based on the algorithm’s purpose One of the first things detected during the analysis of the proposals was that the purposes of these differed in terms of the approach to improve facial expression identification. On the one hand, some authors consider that the best way to achieve this is to focus on enhancing FAUD under the premise that improved FAUD may lead to better FER accuracy. Others, on the other hand, although they use facial Action Units (AU) for the extraction of the Facial Landmarks (FL) with which facial expressions are detected, their effort is focused on improving the algorithms for FER, so they do not elaborate on how they identified the AUs. Finally, a last group of authors contemplates in their proposals both methods to improve the FAUD and methods to improve the FER, thus providing a complete solution. Facial action unit detection research comparison Figure 9 shows the results of the proposals aimed at improving FAUD. The F1-score was adopted because it is the metric most consistently reported by the original authors to assess model effectiveness, reflecting the balance between precision and recall. The same figure also shows the number of datasets used to test each proposal, which is considered here as a complementary indicator of evaluation breadth. For studies tested on more than one dataset, the reported F1-scores were averaged across datasets. Mean F1-score achieved by the evaluated FAUD models (bars), and the number of test datasets used for each model (yellow circles). No dispersion bars are shown because the included studies do not provide directly comparable variance estimates under a unified evaluation setting. The overall mean F1-score across all proposals is 0.66, which also coincides with the median. Rather than inferring a specific statistical distribution from this observation, it is more informative to examine how high performance relates to evaluation breadth. The three proposals with the highest F1-scores are FE-AURDGCN62 (0.93), he201938 (0.89), and AEL-NN73 (0.86). However, these studies differ in the number of datasets used for testing (two, one, and three, respectively), which suggests that performance should not be interpreted independently of testing scope. In particular, results obtained on a single dataset provide more limited evidence of robustness than those observed across multiple datasets. For this reason, the comparison below prioritizes proposals that combine relatively high F1-scores with broader evaluation coverage. Under this criterion, the most informative cases are AEL-NN73 (0.86), akay202268 (0.72), wang201826 (0.72), and FRBM+41 (0.70), all of them tested on three datasets. AUD-EA(gBN)61 was not included in this subset because, although it was also evaluated on three datasets, its F1-score falls below the overall average. These four proposals do not share a single dominant design pattern. However, none of them relies on a pre-trained backbone, which may suggest that FAUD-specific architectures can remain competitive when they are explicitly designed around AU representation or dependency modeling. AEL-NN73: Its core architecture combines two convolutional neural networks (CNNs) with two interleaved attention blocks, followed by a dense layer with sigmoid activation for classification. Each attention block integrates both channel and spatial attention. AEL-NN73: Its core architecture combines two convolutional neural networks (CNNs) with two interleaved attention blocks, followed by a dense layer with sigmoid activation for classification. Each attention block integrates both channel and spatial attention. Reference68: It implements multiple stacked CNNs as classifiers and uses Motion History Images (MHI) to capture and model muscle motion. A multilayer perceptron is used as a meta-classifier. Reference68: It implements multiple stacked CNNs as classifiers and uses Motion History Images (MHI) to capture and model muscle motion. A multilayer perceptron is used as a meta-classifier. Reference26: This proposal departs from CNN-based design by combining two Latent Regression Bayesian Network (LRBN) layers with two Bayesian Network (BN) layers, jointly performing detection and classification. Reference26: This proposal departs from CNN-based design by combining two Latent Regression Bayesian Network (LRBN) layers with two Bayesian Network (BN) layers, jointly performing detection and classification. FRBM+41: Its central architecture is a four-layer Restricted Boltzmann Machine (FRBM), extended with facial-expression-related nodes to improve AU detection through joint modeling. FRBM+41: Its central architecture is a four-layer Restricted Boltzmann Machine (FRBM), extended with facial-expression-related nodes to improve AU detection through joint modeling. The F1-scores of26,68, and FRBM+ differ by at most two percentage points, suggesting that competitive FAUD performance can be achieved through substantially different modeling strategies. By contrast, AEL-NN reaches a noticeably higher F1-score, exceeding the other three proposals by approximately 14 percentage points. Although AEL-NN and68 both incorporate CNNs, their comparative difference suggests that the attention design of AEL-NN may play an important role in its stronger performance. Nevertheless, attention should not be interpreted as a universally sufficient factor. Other proposals that incorporate attention-related mechanisms, such as75 (0.66, 1 dataset), SituTech43 (0.50, 1), GA69 (0.57, 2), and PMVT11 (0.62, 2), remain at or below the overall average. This suggests that the contribution of attention depends on how it is integrated into the broader FAUD pipeline rather than on its presence alone. Taken together, this comparison suggests that strong FAUD performance is not restricted to a single architectural family. CNN-based, Bayesian, and RBM-based solutions all appear viable when they are coupled with effective AU-oriented design choices. At the same time, the result of AEL-NN indicates that custom attention mechanisms may provide an additional advantage when they are specifically tailored to the FAUD task. Facial expression recognition research comparison Figure 10 shows the results of the proposals focused on improving FER. Accuracy was adopted because it is the metric most consistently reported by the original authors to evaluate model effectiveness. The same figure also shows the number of datasets used to test each proposal, which is considered here as a complementary indicator of evaluation breadth. For studies tested on more than one dataset, the reported accuracies were averaged across datasets. Mean accuracy achieved by the evaluated FER models (bars), and the number of test datasets used for each model (yellow circles). No dispersion bars are shown because the included studies do not provide directly comparable variance estimates under a unified evaluation setting. Following the logic used in the FAUD comparison, priority was given to proposals that combine relatively high accuracy with broader evaluation coverage. The overall average accuracy across all proposals is 79.7%. Two studies, MSAU-Net80 (75.5%) and SAANet81 (78.6%), fall slightly below this overall average, but they were retained in the comparison because they were evaluated on the largest number of datasets (nine and five, respectively), thus providing broader evidence of robustness. To make the comparison more balanced, the following discussion considers their average performance on the three most frequently used datasets. When two or more datasets appeared with the same frequency within a proposal, the dataset associated with the highest reported average was retained for this restricted comparison. FER-IK(gBN)61 (92.7%): This proposal relies on a Bayesian Network (BN) supported by Artificial Neural Networks (ANNs), based on the idea that facial expression recognition has a strong probabilistic component. A softmax layer is used for classification. FER-IK(gBN)61 (92.7%): This proposal relies on a Bayesian Network (BN) supported by Artificial Neural Networks (ANNs), based on the idea that facial expression recognition has a strong probabilistic component. A softmax layer is used for classification. MiniExpNet+46 (91.3%): This model is built on a Convolutional Neural Network (CNN) enriched with an attention block to improve AU detection within facial regions of interest. A self-distillation mechanism (SDM) is further employed to enhance generalization during training, followed by classification through a softmax layer. MiniExpNet+46 (91.3%): This model is built on a Convolutional Neural Network (CNN) enriched with an attention block to improve AU detection within facial regions of interest. A self-distillation mechanism (SDM) is further employed to enhance generalization during training, followed by classification through a softmax layer. Reference84 (84.5%): This proposal uses ResNet-18 as backbone, followed by a Linear Attention Module (LAM) to extract landmark-related information and convert it into graphs, which are then processed with a Graph Convolutional Network (GCN). A Multilayer Perceptron (MLP) is used as classifier. Reference84 (84.5%): This proposal uses ResNet-18 as backbone, followed by a Linear Attention Module (LAM) to extract landmark-related information and convert it into graphs, which are then processed with a Graph Convolutional Network (GCN). A Multilayer Perceptron (MLP) is used as classifier. POSTER85 (83.7%): The model uses MobileFaceNet and IR50 as backbones to extract facial-landmark features and global image features, respectively. A Feature Pyramid Network (FPN) is employed to strengthen landmark-related representation, while the authors’ proposed Multi-Head Self-Attention (MSA) module extracts and refines attention points from both streams. The resulting representations are then processed by a Transformer encoder (Encoder), and an MLP is used for final classification. POSTER85 (83.7%): The model uses MobileFaceNet and IR50 as backbones to extract facial-landmark features and global image features, respectively. A Feature Pyramid Network (FPN) is employed to strengthen landmark-related representation, while the authors’ proposed Multi-Head Self-Attention (MSA) module extracts and refines attention points from both streams. The resulting representations are then processed by a Transformer encoder (Encoder), and an MLP is used for final classification. SAANet81 (83.4%): This proposal uses a VGG16-based cascade structure composed of CNN and Recurrent Neural Network (RNN) components. Between them, the authors introduced a Global-AU Attention Module (GAUAM) to focus the model on each detected AU. The architecture is further enriched with an Attentive Pooling Module (APM) to capture AU changes over time. While the Distance Loss Function is reported as the training objective, the original study does not explicitly identify the final classification layer. SAANet81 (83.4%): This proposal uses a VGG16-based cascade structure composed of CNN and Recurrent Neural Network (RNN) components. Between them, the authors introduced a Global-AU Attention Module (GAUAM) to focus the model on each detected AU. The architecture is further enriched with an Attentive Pooling Module (APM) to capture AU changes over time. While the Distance Loss Function is reported as the training objective, the original study does not explicitly identify the final classification layer. MSAU-Net80 (75.5%): This architecture combines two CNN streams, one focused on the whole face and another on AU-relevant regions, and extends temporal analysis through an additional CNN branch. Attention blocks are used to strengthen the relationship between detected AUs and facial expressions, and final classification is performed through a softmax layer. MSAU-Net80 (75.5%): This architecture combines two CNN streams, one focused on the whole face and another on AU-relevant regions, and extends temporal analysis through an additional CNN branch. Attention blocks are used to strengthen the relationship between detected AUs and facial expressions, and final classification is performed through a softmax layer. With only one exception, all proposals in this comparison incorporate some form of attention or region-aware refinement, which suggests that such mechanisms are frequently associated with competitive FER performance. However, the reviewed evidence does not support identifying a universally superior architectural family; instead, it indicates that performance depends on how effectively a model captures AU locality, inter-region dependencies, temporal or cross-scale context, and the computational constraints of the target application. FER-IK(gBN) (92.7%), based on a Bayesian Network, is the highest-performing proposal within this comparison and the only one that does not incorporate explicit attention modules. Additional BN-based studies, such as77(96.8%, 1 dataset)38, (91.7%, 1), and78 (82.8%, 2), also report results above the overall average, although they were excluded from the core comparison because of their narrower evaluation coverage. Taken together, these results suggest that probabilistic modeling remains a competitive alternative for FER, particularly when AU dependencies are explicitly represented. Regarding the use of pre-trained backbones, the reviewed studies generally employ them as facilitators for feature extraction, landmark processing, or attention-guided refinement, rather than as sufficient explanations of improved FER performance by themselves. Their contribution therefore appears to depend more on how they are integrated with AU-aware, relational, or attention-based components than on backbone choice alone. From a comparative perspective, the reviewed FER literature indicates that competitive performance can be achieved through substantially different modeling strategies, including Bayesian, CNN-based, graph-based, and Transformer-enhanced approaches. Rather than pointing to a single dominant family, the evidence suggests that FER performance benefits most when AU-related structure is effectively incorporated into the representation and decision process. Comparison of FER research that include FAUD As noted at the beginning of this section, some studies propose integrated solutions aimed at improving both FER and FAUD, and report comparative results for both tasks. However, this type of research remains relatively limited, representing only 23% of the FER-oriented proposals and 25% of the FAUD-oriented proposals considered in this survey, as shown in Fig. 11. Comparison of evaluated research papers that address both FAUD and FER with those that focus exclusively on either FAUD or FER. Table 10 summarizes the studies that report results for both FER and FAUD, including their corresponding performance metrics, the datasets used for each task, and the degree of overlap between FER and FAUD evaluation datasets. A first relevant observation is that this subgroup remains methodologically heterogeneous. Only AEL-NN73 was evaluated on three datasets for FAUD, whereas the remaining proposals were tested on one or two datasets only. Consequently, the evidence provided by these studies should be interpreted with caution, particularly when considering robustness across datasets. Table 10Studies that encompass both FER and FAUD, along with the datasets used in each study.ProposalFERFAUDFER DatasetsFAUD DatasetsSimilaritiesFE-AURDGCN6294.7%0.93CK+; RAF-DBCK+; RAF-DB100%AEL-NN7393.6%0.86RAF-DB; FER2013+ (FERPlus)BP4D; DISFA; CK+0%he20193891.7%0.89CK+CK+100%AWS-Net6485.2%0.58RAF-DBGFT0%MTAA6580.6%0.15Aff-Wild2Aff-Wild2100%Global average79.7%0.66SJMT6370.0%0.73SFEW; Oulu-CASIAEmotioNet0%LibreFace2366.3%0.63AffectNet; RAF-DBBP4D; DISFA0%The “FER” and “FAUD” columns report the average accuracy and F1-score, respectively. The “FER Datasets” and “FAUD Datasets” columns list the datasets used for each task. The “Similarities” column indicates the percentage of dataset overlap between FER and FAUD. Studies that encompass both FER and FAUD, along with the datasets used in each study. The “FER” and “FAUD” columns report the average accuracy and F1-score, respectively. The “FER Datasets” and “FAUD Datasets” columns list the datasets used for each task. The “Similarities” column indicates the percentage of dataset overlap between FER and FAUD. A second relevant observation is that most studies did not use the same datasets for FER and FAUD evaluation, including AEL-NN. Among the proposals with complete dataset overlap, two were evaluated on only one shared dataset, which limits the breadth of the evidence. In contrast, FE-AURDGCN62 stands out as the clearest case in which the same two datasets (CK+ and RAF-DB) were used to assess both FER and FAUD, providing a more directly aligned basis for cross-task interpretation. Overall, the reviewed studies do not follow a consistent evaluation strategy regarding whether FER and FAUD should be tested on matched or distinct datasets, and in most cases this choice is not explicitly justified by the authors. Taken together, these results suggest that integrated FER-FAUD research remains promising but still methodologically inconsistent, particularly in terms of evaluation breadth and dataset alignment across both tasks. Comparison based on the proposal final scope Defining the scope is a foundational step in any research project, particularly in a field as broad as Facial Expression Recognition (FER). The proposals considered in this survey (Table 9) can be grouped according to the final scope defined by their authors. Rather than reducing the literature to a simple FAUD-versus-FER dichotomy, the reviewed studies were organized into a four-scope framework that distinguishes FAUD, FER, IFA, and F2F: Studies that propose innovative models for FAUD under the assumption that improving AU detection may contribute to better FER, but without experimentally validating this transfer; Studies that propose innovative models for FAUD under the assumption that improving AU detection may contribute to better FER, but without experimentally validating this transfer; Studies that seek new FER methodologies without relying on action units; Studies that seek new FER methodologies without relying on action units; Studies that aim to improve FER accuracy (IFA) by incorporating AU-related information during recognition; Studies that aim to improve FER accuracy (IFA) by incorporating AU-related information during recognition; Studies that explicitly implement FAUD as an intermediate stage for FER (F2F). Studies that explicitly implement FAUD as an intermediate stage for FER (F2F). Figure 12 shows the distribution of the reviewed proposals across these four scopes. Although FAUD-oriented studies represent 46% of the proposals selected under the PRISMA-based process (see Section "Adherence to preferred reporting items for a systematic review and meta-analysis guidelines"), they were not included in the final comparative FER analysis because they do not experimentally validate FER as an output. This distinction is important because the present survey is not concerned only with whether AU detection is improved in isolation, but with whether such improvement is actually translated into facial expression recognition performance. Comparison of evaluated research articles categorized as follows: (1) those focused directly on Facial Expression Recognition (FER); (2) those that use Facial Action Unit Detection to improve FER accuracy (IFA); (3) those that begin with FAUD to achieve FER (F2F); and (4) those focused exclusively on FAUD without proposing FER as an outcome. As shown in Fig. 13, the average accuracy of proposals belonging to the FER scope is slightly below the overall average, whereas the IFA scope is only marginally above it. By contrast, the F2F scope reaches the highest average FER accuracy, approximately 1.8 percentage points above the overall mean. Within the limits of the reviewed evidence, this suggests that approaches that treat FAUD as an explicit intermediate stage tend to provide a more effective basis for FER than either direct FER-only methods or approaches that use AU information only as an auxiliary refinement signal. Comparison of the average Facial Expression Recognition (FER) accuracy obtained by the F2F, IFA, and FER models, against the overall average achieved by all evaluated models with a FER scope. This interpretation should be understood as a survey-level trend rather than as a controlled head-to-head conclusion, since the studies differ in datasets, labeling protocols, architectural choices, and evaluation settings. Even so, the comparative pattern remains relevant: the strongest average performance is observed in the scope where AU detection is not merely associated with FER, but structurally integrated into the recognition pipeline. Figure 14 shows the F2F proposals in greater detail. Among them, the most competitive results correspond to the studies highlighted in Section "Facial expression recognition research comparison": kim202384, SAANet81, and MSAU-Net80. Although these proposals rely on different architectures -including graph-based modeling, recurrent temporal processing, and multi-branch CNN designs- they all share one relevant characteristic: AU-related information is not treated as a peripheral cue, but as a structurally meaningful component of the FER process. This recurring pattern is more informative than the use of any single backbone or module in isolation. Comparison of the average accuracy of models with a FAUD-to-FER scope, including the overall average of all evaluated models, along with the number of test datasets used. No dispersion bars are shown because the included studies do not provide directly comparable variance estimates under a unified evaluation setting. Taken together, the scope-based comparison suggests that the most favorable FER outcomes tend to emerge when FAUD is incorporated as an explicit and experimentally validated intermediate stage. Thus, the main comparative advantage does not appear to lie in using AU information in any form, but in structuring the FER pipeline so that AU detection meaningfully mediates the recognition process. Other comparisons During the documentary research, certain elements were considered valuable for evaluation because they provide additional information on state-of-the-art trends or are essential in the FER research process. We will examine these elements in this section. Real-time research comparison Although facial expression recognition from static images remains relevant, many of its practical applications -including security, human-computer interaction, and patient care- require real-time performance. This requirement becomes even more relevant when deployment on edge or mobile devices is considered. Among the evaluated studies, five explicitly address real-time recognition, and two report deployment-oriented solutions involving mobile or edge settings. Comparative information is summarized in Table 11. Table 11Comparison of real-time proposals.ProposalFAUDFERInfrastructureAvg F1-SAvg AccDatasetsTime (ms)HOG-XGBoost89NYCPU Computer86.6%230FER(rt)88NYDistributed87.7%238EfficientNet-B015YNMobile0.541No dataMiniExpNet+46NYCPU Computer91.3%35.42LibreFace23YYCPU Computer0.6366.3%2-237.2The “FAUD” and “FER” columns indicate whether the proposal addresses FAUD, FER, or both. The “Infrastructure” column specifies the type of hardware on which the model was executed. The “Avg F1-S” and “Avg Acc” columns report the average F1-score and accuracy values, respectively. The “Datasets” column indicates the number of datasets used to evaluate each model; in the case of LibreFace, separate counts are reported for FER and FAUD. Finally, the “Time” column presents the average inference time, in milliseconds, required to recognize a facial expression or facial action unit after the model has been trained. Comparison of real-time proposals. The “FAUD” and “FER” columns indicate whether the proposal addresses FAUD, FER, or both. The “Infrastructure” column specifies the type of hardware on which the model was executed. The “Avg F1-S” and “Avg Acc” columns report the average F1-score and accuracy values, respectively. The “Datasets” column indicates the number of datasets used to evaluate each model; in the case of LibreFace, separate counts are reported for FER and FAUD. Finally, the “Time” column presents the average inference time, in milliseconds, required to recognize a facial expression or facial action unit after the model has been trained. Two main patterns emerge from Table 11. First, most proposals remain centered on FER, whereas only two include FAUD, and only one of them -EfficientNet-B015- focuses exclusively on FAUD. However, this study does not report inference time, which limits its direct comparability with the remaining real-time proposals. Second, most reported response times fall within a relatively narrow range of 30 to 38 milliseconds, suggesting that real-time FER is already feasible under several implementation strategies. Within this subgroup, MiniExpNet+46 stands out for combining the best reported FER accuracy (91.3%) with the shortest response time (5.42 ms), while also being evaluated on three datasets. Its architecture is based on a CNN, complemented by an attention block to improve AU detection within facial regions of interest. A self-distillation mechanism (SDM) is incorporated to enhance generalization during training, and final classification is performed through a softmax layer. FER(rt)88 is also noteworthy because it targets edge-oriented deployment through a Raspberry Pi camera connected to a distributed microservice-based architecture. Although its response time (38 ms) is slower than that of MiniExpNet+, it still remains within a real-time range and achieves 87.7% accuracy across two test datasets. This suggests that deployment-oriented design choices, such as task distribution across computing nodes, may support practical FER implementation even under constrained local hardware. The evidence analyzed in this survey suggests that current real-time FER proposals should be interpreted as promising deployment-oriented prototypes rather than fully mature real-world solutions. Although several studies already report latency values in the range of approximately 5–38 ms, these results are still tied to limited benchmark settings, specific hardware configurations, and a small number of datasets. Therefore, the present survey does not assign a formal Technology Readiness Level (TRL), but the reviewed literature more plausibly reflects early-stage technical feasibility than large-scale operational readiness. In this context, future work should prioritize edge-oriented optimization, temporal modeling for continuous video streams, and privacy-aware architectures capable of supporting practical deployment under realistic constraints. Datasets comparison To the best of our knowledge, most survey-oriented studies in this area emphasize state-of-the-art model performance while paying limited attention to the characteristics of the datasets used for training and evaluation. In the present study, dataset comparison was incorporated because the reported results are closely related to the evaluation conditions under which the models were tested, particularly in FER. Accordingly, this section provides contextual information intended to support a more careful interpretation of the comparative trends observed across the reviewed studies. FER and FAUD datasets differ substantially in difficulty depending on how they were constructed. As discussed in the section on dataset-related challenges (see Section "The challenge of datasets"), some datasets are collected in controlled environments, whereas others aim to represent in-the-wild conditions. Likewise, some are posed, while others attempt to capture spontaneous affective behavior. Labeling procedures also vary considerably, ranging from expert coding to crowdsourced annotation. In addition, some datasets are specialized for particular ethnic groups93, pain-related expressions94, or specific clinical conditions such as Down syndrome95. For these reasons, the average results observed across studies should not be interpreted as evidence that one dataset is intrinsically easier, better, or more suitable than another. Dataset popularity may also reflect factors such as accessibility, historical adoption, or community familiarity rather than performance alone. Even so, some comparative observations remain informative. Figure 15 shows that the Cohn-Kanade extended dataset (CK+) is by far the most frequently used dataset in FER-oriented studies, appearing in approximately two-thirds of the proposals and yielding the highest average accuracy (94.8%). This result is consistent with its highly controlled acquisition conditions, its EMFACS-based design, and its expert annotation protocol. The Real-world Affective Face Database (RAF-DB) is the second most frequently used dataset, appearing in roughly half of the proposals, with an average accuracy of 84.6%. This is notable because RAF-DB belongs to the in-the-wild category, where greater visual variability and less controlled labeling conditions would typically make recognition more challenging. AffectNet and Static Facial Expressions in the Wild (SFEW) appear in approximately one-quarter of the reviewed proposals; their associated average accuracies are 62.8% and 57.5%, respectively. Datasets statistics in FER proposals. The bars show the proportional usage of each dataset across the reviewed proposals, while the line graph presents the overall average accuracy associated with each dataset. The colors indicate the dataset type: Blue for Posed, Green for Contextual, and Purple for in-the-Wild. At the aggregate level, posed datasets reach an average accuracy of 85.1%, whereas in-the-wild datasets reach 81.0%. Contextual datasets remain substantially lower at 62.3%. Rather than indicating that contextual datasets are inherently less appropriate for FER, this pattern may reflect the fact that dataset categories capture different sources of variability, annotation strategies, and expression elicitation conditions. Therefore, the observed averages are more appropriately interpreted as descriptive tendencies within the reviewed literature than as direct evidence of dataset superiority. Turning to FAUD, Fig. 16 shows that CK+ is again among the most frequently used datasets and yields the second highest average result (0.82 F1-score). The other most commonly used datasets are BP4D, a posed dataset with an average F1-score of 0.63, and DISFA, a contextual dataset with an average F1-score of 0.64. Compared with the FER analysis, these lower values suggest that AU detection remains a more demanding task than facial expression recognition, even when partially overlapping datasets are used. In this context, the relative stability of CK+ across both FER and FAUD is noteworthy and may deserve further investigation. Datasets statistics in FAUD proposals. The bars show the proportional usage of each dataset across the reviewed proposals, while the line graph presents the overall average F1-score associated with each dataset. The colors indicate the dataset type: Blue for Posed, Green for Contextual, and Purple for in-the-Wild. Finally, RAF-DB, which is only rarely used in FAUD-oriented studies, shows an unusually high average F1-score (0.96). Given its limited presence in this subset and its contrast with the broader pattern observed across other datasets, this result should be interpreted cautiously, as it may reflect study-specific conditions rather than a stable dataset-level trend. Discussion and conclusion Before interpreting the broader implications of the reviewed literature, it is useful to restate the main comparative finding of this survey. Among the evaluated scopes, FAUD-to-FER (F2F) proposals showed the highest average FER accuracy, outperforming both direct FER-oriented approaches and IFA-based strategies. Within the limits of the reviewed evidence, this suggests that FAUD may be especially effective when implemented as an explicit intermediate stage in the FER pipeline. Artificial Intelligence (AI), and Computer Vision (CV) in particular, are expected to continue evolving rapidly in the coming years. In this context, the reviewed literature suggests that progress in FER depends not only on improving predictive performance, but also on how structural facial information is incorporated into the recognition process. This becomes especially relevant when considering interpretability, dataset heterogeneity, and deployment constraints, all of which shape the practical value of FAUD-based intermediate modeling. From the FAUD perspective, the overall average F1-score observed across the analyzed studies is 0.66. Although this value indicates meaningful progress, it also suggests that AU detection remains a challenging task, particularly under heterogeneous evaluation conditions. The comparative analysis carried out in Section "Facial action unit detection research comparison" indicates that strong FAUD performance can be achieved through different modeling strategies, including CNN-based architectures with attention mechanisms, probabilistic formulations such as Latent Regression Bayesian Networks (LRBN), and Restricted Boltzmann Machine-based designs. Rather than supporting a single dominant family, the evidence suggests that FAUD benefits most when local AU-sensitive representations, inter-AU dependencies, and context-aware processing are effectively combined. For FER, the overall average accuracy across the analyzed proposals is 79.7%. As discussed in Section "Facial expression recognition research comparison", competitive performance is observed in Bayesian, CNN-based, graph-based, and Transformer-enhanced approaches. However, the relative strengths of these families appear to depend less on backbone choice alone than on how effectively the model captures AU locality, inter-region dependencies, temporal or multi-scale context, and task-specific structural information. In this sense, the analyzed studies suggest that AU-informed FER gains are more strongly associated with representational design than with any single architectural label. Another relevant issue concerns evaluation datasets. In studies that report both FAUD and FER results, it is common to find different datasets used for each task (see Section "Comparison of FER research that include FAUD"). The analyzed literature does not show a consistent evaluation strategy regarding whether both tasks should be assessed on matched datasets, and in most cases this decision is not explicitly justified. More broadly, the dataset comparison performed in Section “Datasets comparison” suggests that dataset choice is influenced not only by performance considerations, but also by factors such as acquisition setting, labeling protocol, accessibility, and research objective. Therefore, differences in reported averages across datasets should be interpreted as descriptive trends rather than as evidence of intrinsic dataset superiority. Real-time FER also deserves special attention. Although most analyzed studies rely entirely on benchmark datasets, a small subset explores real-time or edge-oriented deployment (see Section "Real-time research comparison"). These studies show that practical FER is already technically feasible, especially when lightweight or compact architectures are combined with region-aware processing and deployment-conscious optimization. At the same time, the limited number of such proposals indicates that real-time implementation remains underexplored relative to the broader methodological literature. Another notable observation is the limited use of optimization methods beyond standard deep learning training procedures. Only a few optimization-related techniques were identified, such as Adam, the Lovász extension, and the Alternating Direction Method of Multipliers, mainly for weight adjustment, balancing, or optimization subproblems. No explicit metaheuristic methods were identified in the analyzed proposals. Although this absence may be partially explained by computational cost, the survey suggests that optimization-oriented approaches could still offer value in determining complex architectural or probabilistic parameters, especially during offline design stages where inference efficiency is not yet the primary constraint. Beyond reporting which architectures achieved higher scores, a more informative interpretation is to examine why certain model families tend to perform better under specific conditions. CNN-based models remain strong baselines because they efficiently capture local appearance changes associated with facial muscle activity, especially when coupled with attention mechanisms that constrain the analysis to AU-relevant regions. This pattern is consistent with several of the strongest FER and F2F results, where attention modules appear to improve the linkage between localized AU evidence and global expression inference. Graph-based methods, including GCN- and GGNN-oriented formulations, appear particularly advantageous when the method explicitly models dependencies among action units or between facial regions. Their strength is not merely feature extraction, but relational encoding: they are better aligned with the structured and interdependent nature of facial activity. This may explain why graph-based or graph-enhanced approaches remain competitive even when not built on the deepest visual backbones. Transformer-based and hybrid architectures provide a different advantage. Their contribution becomes more evident in settings requiring multi-scale integration, long-range contextual reasoning, or temporal modeling, such as video-based AU detection or cross-region feature fusion. However, these benefits are typically associated with higher computational cost and stronger dependence on backbone quality, training scale, and optimization stability. For this reason, their superiority should not be interpreted as universal, but rather as conditional on data richness and computational availability. Bayesian and probabilistic formulations, although less dominant in recent literature volume, deserve separate attention because they offer an interpretable mechanism for encoding AU-expression dependencies. In the analyzed studies, these approaches showed competitive performance, suggesting that explicitly modeling uncertainty and conditional structure can still be highly effective, particularly when AU knowledge is incorporated as prior information. From a deployment perspective, the survey also suggests a clear trade-off between predictive sophistication and operational feasibility. Lightweight or reduced-complexity pipelines -including HOG/XGBoost, compact CNNs, or knowledge-distilled models- are more compatible with real-time and edge scenarios, where latency and hardware constraints are decisive. By contrast, richer hybrid, Transformer, and multi-branch architectures may achieve stronger representational power, but often at the expense of scalability and inference efficiency. Taken together, the evidence analyzed in this survey suggests that performance gains are less attributable to a single architectural family than to the successful integration of four design principles: localized AU-sensitive feature extraction, explicit modeling of inter-AU dependencies, multi-scale or temporal context fusion, and computational adaptation to the intended deployment scenario. Under this interpretation, F2F approaches appear advantageous not simply because they include FAUD, but because they more often operationalize these principles in a structurally coherent pipeline. Accordingly, the relative strengths discussed here should be interpreted as survey-level trends emerging from heterogeneous studies, not as definitive head-to-head evidence of intrinsic architectural superiority. Future directions Facial Expression Recognition (FER) remains an active and expanding area of Computer Vision, with significant opportunities for both methodological refinement and practical deployment. Although the present survey covers more than 70 tools across 50 studies, several research directions remain insufficiently explored and could support meaningful progress in future work. First, the evidence analyzed in this survey suggests that further advances may come from improving the structural integration between FAUD and FER rather than treating both tasks independently. In particular, future models could explore more effective ways to combine AU-sensitive local representations with temporal reasoning, graph-based dependency modeling, and multi-scale contextual analysis. This is especially relevant for F2F-oriented pipelines, which showed the strongest comparative FER results in the analyzed literature. Second, attention-based mechanisms remain a promising direction. Guided Attention, regional attention, and graph-aware attention designs may further improve the relationship between localized facial activity and global expression inference. Likewise, Transformer-based architectures may offer additional benefits when long-range dependencies, temporal consistency, or multimodal fusion are relevant, although their use should continue to be balanced against computational cost and deployment constraints. Third, weak supervision and data limitations remain central challenges. Self-supervised learning, pseudo-labeling strategies, domain adaptation, and multimodal learning could help reduce dependence on fully labeled AU datasets. In the same direction, better methods are still needed to handle uncertainty, cross-dataset transfer, and heterogeneous annotation protocols. Since the existing FER datasets cover only a limited portion of real-world demographic and behavioral variability, future work should also explore more adaptive learning schemes, including meta-learning and domain-generalization approaches. Fourth, privacy-aware and deployment-oriented FER deserve greater attention. Federated learning may offer an alternative for training FER systems under stricter privacy constraints, while lightweight architectures, knowledge distillation, and compact region-aware pipelines remain especially relevant for edge and mobile implementation. Given the comparatively small number of real-time studies identified in this survey, practical deployment remains an important opportunity for future research. In addition, future studies should report deployment-oriented benchmarks more systematically, including latency, accuracy under continuous video conditions, and hardware-dependent computational requirements, so that real-time FER can be assessed not only as a modeling problem but also as an implementation problem. This is especially important if the field is to move from promising prototype-level results toward more mature and reproducible deployment scenarios. Finally, the survey suggests that optimization-oriented methods are underrepresented in this field. Automated hyperparameter optimization, evolutionary strategies, and metaheuristic approaches could be explored not necessarily for online inference, but for offline design decisions involving architecture configuration, parameter estimation, or probabilistic structure refinement. Such methods may become particularly useful in hybrid models where the interaction between modules is too complex to tune manually. Overall, future progress in FER is likely to depend not on a single breakthrough architecture, but on the careful integration of interpretability, structural facial knowledge, data efficiency, and deployment feasibility. In this sense, the most promising path forward may be the development of solutions that remain both methodologically robust and practically applicable in real-world scenarios. Appendix A: Supplementary tables See Tables 12,13,14. Table 12Research Papers Selected via PRISMA (by Acronym). Acronym Description Acronym Description AdaFERAction units guided unsupervised Domain-Adaptive Facial Expression Recognition37AEL-NNAttention-Enabled Lightweight Neural Network architecture73AFFDEXAFFective Facial DEtection67akay2022The authors did not define any acronym68ATFAdversarial Training Framework66AUCaps++Action Units Capsule Networks Extended72AUD-EA(gBN)Expression-Augmented Action Units Detection61AUE-CRLAction Units Expression knowledge Constrained Representation Learning14AWS-NetAdaptively Weights Shared Network64belhouchette 2019The authors did not define any acronym91DAUGNDeep Action Units Graph Network79DGANDual learning Generative Adversarial Network13DSGANDual Semi-supervised Generative Adversarial Network42EfficientNet-B0The name was assigned by the authors; it is not an acronym15FE-AURDGCNFacial Expression and Action Unit Recognition augmented by their Dependencies on Graph Convolutional Networks62FER(rt)Real-Time Facial Expression Recognition88FER-IK(gBN)Knowledge-augmented Image-based Facial Expression Recognition61FG-NetFeature extraction from StyleGAN2 Network76FRBM+Four Layers Restricted Boltzmann Machine Extended41GAGuided Attention69GTLE-NetGlobal-to-Local Expression-aware Network12he2019The authors did not define any acronym38HOG-XGBoostHistograms of Oriented Gradients - XGBoost89hong2020The authors did not define any acronym82kim2023The authors did not define any acronym84li2019The authors did not define any acronym24LibreFaceThe name was assigned by the authors; it is not an acronym23MiniExpNet+The name was assigned by the authors; it is not an acronym46MSAU-NetMulti-Scale Action Units based Network80MTAAMulti-Task Affective Analysis65MTACMulti-Task Assisted Correction92MTLRTMulti-Scale Region with Local Relationship Learning71perveen2020The authors did not define any acronym90PMVTProgressive Multi-scale Vision Transformer11POSTERPyramid crOSs-fusion Transformer nEtwoRk85POSTER-AUPyramid crOSs-fusion Transformer nEtwoRk for Action Units Detection44R-FENetRegion-based convolutional neural network for Facial Expression recognition Network83SAANetSiamese Action-units Attention Network81SituTechThe name was assigned by the authors; it is not an acronym43SJMTSelective Joint Multi-Task loss63SRERLSemantic Relationship Embedded Representation Learning25TESGCNTransformer Embedded Spectral-based Graph Convolution Network86ULC-AGUncertain Label Correction via Auxiliary action units Graphs36wang2018The authors did not define any acronym26wang2019The authors did not define any acronym77wang2019BThe authors did not define any acronym78yang2024The authors did not define any acronym75yao2021The authors did not define any acronym87zhang2023The authors did not define any acronym74zhao2023The authors did not define any acronym70 Research Papers Selected via PRISMA (by Acronym). Table 13Artificial intelligence tools utilized in included research papers. Tool Description Tool Description AAMActive Appearance ModelAdamAdam optimization methodADMMAlternating Direction Method of MultipliersALAsymmetric loss functionAttention ModAttention ModuleBCE lossBinary Cross-Entropy lossBMatchingBlock-Matching algorithmBNBayesian NetworksCAMCoordinate Attention ModuleCapsNetCapsule NetworkCCCConcordance Correlation CoefficientCE lossCross-Entropy loss functioncGANConditional Generative Adversarial NetworkChebConvChebyshev Graph Convolution LayerCVCross-ValidationDAEDenoising AutoencoderDeep-BlockDeep-Block NetworkDenseNetDensely Connected Convolutional NetworkDNNDeep Neural NetworkDRMFDiscriminative Response Map FittingEncoderTransformer EncoderEnsembleEnsemble LearningFPNFeature Pyramid NetworkFRBM+Four Layers Restricted Boltzmann Machine ExtendedGAMGuided Attention MaskGANGenerative Adversarial NetworkGCNGraph Convolutional NetworkGdGaussian distributionGFGaussian FilterGGNNGated Graph Neural NetworkGoogLeNetGoogLeNetHLBPHistogram Local Binary PatternHOGHistogram of Oriented GradientsKDKnowledge DistillationKL LossKullback-Leibler divergence loss functionLAMLinear Attention ModuleLDALinear Discriminant AnalysisLovászLovász extensionLRBNLatent Regression Bayesian NetworkLRBPLow-Rank Bilinear PoolingLSTMLong Short-Term MemoryMAEMasked Auto-encoderMediaPipeMediaPipeML lossMulti-Label lossML-CEMulti-Label Cross-EntropyMLMLMulti-Label Multi-Level learningMLPMultilayer PerceptronML-ROSMulti-label Random Over-SamplingMMIB-SFMutual Information Based Feature Selection FrameworkMSAMulti-Head Self-AttentionMSE lossMean Squared Error lossMTCNNMulti-task Cascaded Convolutional NetworksPCAPrincipal Component AnalysisPyramid CNNPyramid CNN InterpreterRAERegional Attention Encoder (RAE)R-CNNRegion-based Convolutional Neural NetworkRNNRecurrent Neural NetworkSGDStochastic Gradient DescentSoftmaxSoftmax activation functionSOMSelf-Organizing MapSquare lossSquare Error loss functionSSHSingle Stage HeadlessTC&FRTemporal Feature Combination and Feature Reassignment ModuleTMFTemporal and Multi-modal FusionTriplet lossTriplet loss functionViola-JonesViola-Jones AlgorithmViTVision TransformersWA lossWeighted Asymmetric loss functionXGBoostExtreme Gradient Boosting Artificial intelligence tools utilized in included research papers. Table 14Pre-trained models and backbone architectures reported in the included research papers.ProposalBackboneSituTech43IResnet100PMVT11ResNet-34kim202384ResNet-18AUE-CRL14ResNet-101AdaFER37ResNet-18hong202082VGGFacewang201977AlexNetR-FENet83ResNet-18GA69DenseNet-121POSTER85MobileFaceNet/IR50POSTER-AU44MobileFaceNet/IR50zhao202370MT-CNNDGAN13TensorFlowwang2019B78AlexNetFG-Net76StyleGAN2GTLE-Net12InceptionResnetV1/StarGANV2yang202475ResNetMTAC92SwinTransformer Pre-trained models and backbone architectures reported in the included research papers. Appendix B: Acronyms For quick reference, the following list summarizes the acronyms defined in the main body of the manuscript. AcronymMeaningABAWAffective Behavior Analysis in-the-WildAFEWActed Facial Expressions in-the-WildAIArtificial IntelligenceANNArtificial Neural NetworkAPMAttentive Pooling ModuleAUAction UnitBNBayesian NetworkBP4DBinghamton–Pittsburgh 4D Spontaneous Facial Expression DatabaseCNNConvolutional Neural NetworkCVComputer VisionCVPRIEEE/CVF Conference on Computer Vision and Pattern RecognitionDBNDynamic Bayesian NetworkDISFADenver Intensity of Spontaneous Facial ActionDLDeep LearningDSGANDual Semi-supervised Generative Adversarial NetworkEMFACSEmotional Facial Action Coding SystemF2FFAUD-to-FERFACSFacial Action Coding SystemFAUDFacial Action Unit DetectionFERFacial Expression RecognitionFER2013Facial Expression Recognition 2013FLFacial LandmarkGAGuided AttentionGAUAMGlobal-AU Attention ModuleGCNGraph Convolutional NetworkGFTSayette Group Formation TaskIFAImproving FER AccuracyKLKullback–LeiblerLRBNLatent Regression Bayesian NetworkMHIMotion History ImageMLMachine LearningML-ROSMulti-label Random Over-SamplingMLPMultilayer PerceptronMMIMMI Facial Expression DatabaseMSAMulti-Head Self-AttentionNIRNear-InfraredPRISMAPreferred Reporting Items for Systematic Reviews and Meta-AnalysesRAF-DBReal-world Affective Face DatabaseRBMRestricted Boltzmann MachineROIRegion of InterestSDMSelf-Distillation MechanismSFEWStatic Facial Expressions in-the-WildSOMSelf-Organizing MapSVMSupport Vector MachineTMFTemporal and Multi-modal FusionVISVisible Author contributions M. P. Methodology, Investigation, Figures, Plots, Tables, Writting original paper I. A. Methodology, Investigation, Formal analysis, Supervision, Writting original paper D. O. Methodology, Investigation, Formal analysis, Supervision, Validation, Writting original paper A. V.-G. Investigation, Formal analysis, Supervision, Writting original paper M. A. N. Methodology, Investigation, Formal analysis, Supervision, Writting original paper M. P.-C. Investigation, Funding, Supervision, Writting original paper Data availability “The data used to support the findings are cited within the article. Also, the datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.” Declarations Competing interests The authors declare no competing interests. Informed consent Informed consent was obtained from all individual participants included in the study. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors contributed equally to this work: Mauricio Priego, Itzel Aranguren, Diego Oliva, Arturo Valdivia-G and Mario A. Navarro. Contributor Information Itzel Aranguren, Email: itzel.aranguren@academicos.udg.mx. Diego Oliva, Email: diego.oliva@cucei.udg.mx. Marco Pérez-Cisneros, Email: marco.perez@cucei.udg.mx. References Associated Data Data Availability Statement “The data used to support the findings are cited within the article. Also, the datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.”