Bridging the gap in FER: addressing age bias in deep learning

F. Xavier Gaya-Morey, Julia Sanchez-Perez, Cristina Manresa-Yee, Jose M. Buades-Rubio · 2025

Read Bridging the gap in FER: addressing age bias in deep learning by F. Xavier Gaya-Morey, Julia Sanchez-Perez, Cristina Manresa-Yee, Jose M. Buades-Rubio (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07638.txt.

Introduction

Xavier Gaya-Morey and co-authors probe age bias in a 2025 study of emotion reading software. Systems that read facial expressions now score high on standard tests. Yet their errors may not fall evenly: older faces could be read worse, which would make the tools unfair in care and service settings.

The team measured accuracy across age groups expression by expression. They also used explanation tools to see where the models look on a face. Gaps did appear, worst for neutral, sad, and angry looks on older people. Attention maps shifted with age too. The study closes with ways to train and test such systems so the elderly are not left behind.

Catalog record

CREATORF. Xavier Gaya-Morey, Julia Sanchez-Perez, Cristina Manresa-Yee, Jose M. Buades-Rubio
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging the gap in FER: addressing age bias in deep learning F. Xavier Gaya-Morey, Julia Sanchez-Perez, Cristina Manresa-Yee, Jose M. Buades-Rubio Abstract Facial Expression Recognition (FER) systems based on deep learning have achieved impressive performance in recent years. However, these models often exhibit demographic biases, particularly with respect to age, which can compromise their fairness and reliability. In this work, we present a comprehensive study of age-related bias in deep FER models, with a particular focus on the elderly population. We first investigate whether recognition performance varies across age groups, which expressions are most affected, and whether model attention differs depending on age. Using Explainable AI (XAI) techniques, we identify systematic disparities in expression recognition and attention patterns, especially for “neutral”, “sadness”, and “anger” in elderly individuals. Based on these findings, we propose and evaluate three bias mitigation strategies: Multi-task Learning, Multi-modal Input, and Age-weighted Loss. Our models are trained on a large-scale dataset, AffectNet, with automatically estimated age labels and validated on balanced benchmark datasets that include underrepresented age groups. Results show consistent improvements in recognition accuracy for elderly individuals, particularly for the most error-prone expressions. Saliency heatmap analysis reveals that models trained with age-aware strategies attend to more relevant facial regions for each age group, helping to explain the observed improvements. These findings suggest that age-related bias in FER can be effectively mitigated using simple training modifications, and that even approximate demographic labels can be valuable for promoting fairness in large-scale affective computing systems. [type=editor, auid=000, bioid=1, orcid=0000-0003-1231-7235] [type=editor, auid=001, bioid=2, orcid=0009-0008-1274-031X] [type=editor, auid=002, bioid=3, orcid=0000-0002-8482-7552] [type=editor, auid=003, bioid=4, orcid=0000-0002-6137-9558] keywords Facial Expression Recognition ,Deep Learning ,Computer Vision ,Elderly ,Bias Mitigation , † † credit: Conceptualization, Methodology, Validation, Investigation, Writing - Original Draft, Writing - Review & Editing Preparation, Software, Visualization † † credit: Conceptualization, Methodology, Validation, Investigation, Writing - Original Draft, Writing - Review & Editing Preparation, Software, Visualization † † credit: Conceptualization, Methodology, Writing - Review & Editing Preparation, Supervision, Project administration, Funding acquisition † † credit: Conceptualization, Methodology, Writing - Review & Editing Preparation, Supervision, Project administration, Funding acquisition † † corresponding: Corresponding author 1 Introduction Facial expressions play a key role in conveying emotions and intentions in human communication [ 11 ] . Because of their relevance in a wide range of applications—such as assistive robotics, healthcare, driver monitoring, or other human-computer interaction systems—automatic facial expression recognition (FER) has become a prominent research area in computer vision. Over the years, many FER systems have been developed to extract emotional information from facial features, with recent progress being driven largely by deep learning techniques [ 39 ] . Thanks to the unprecedented advances in the machine learning field, many techniques for tackling this task now use deep learning approaches [ 39 ] . These data-driven approaches rely heavily on large annotated datasets, which can inadvertently reflect imbalances present in the collected data [ 12 ] . One of the most common and impactful biases in FER datasets is related to age: children and elderly individuals tend to be underrepresented, while adults are typically overrepresented. Moreover, the facial morphology and expressive patterns of older adults differ from those of younger individuals, as highlighted in prior studies [ 4 ] . For example, wrinkles and skin folds can interfere with expression decoding and introduce challenges for both human and automated systems [ 18 ] . Without proper handling of these differences, FER models risk learning spurious correlations and producing inaccurate predictions for minority age groups [ 60 , 44 ] . To better understand and mitigate such issues, researchers have increasingly turned to eXplainable Artificial Intelligence (XAI) [ 5 , 1 ] , which provides tools to interpret the internal decision-making processes of deep learning models. By offering insights into the features and regions that guide model predictions [ 25 ] , XAI can be used to detect and characterize biases, as shown in studies examining gender bias [ 43 ] and differences between neurotypical users and those with intellectual disabilities [ 54 ] . However, to date, little attention has been given to leveraging XAI to explore age-related bias in FER. In this work, we aim to address this gap by conducting a comprehensive analysis of age-related bias in deep learning models for FER. Using XAI techniques, we examine how such biases manifest across age groups, which expressions are most affected, and how model attention differs depending on the subject’s age. Based on these insights, we implement and evaluate three bias mitigation strategies: Multi-task Learning, Multi-modal Input, and Age-weighted Loss. These approaches are tested on AffectNet, using automatically estimated age labels for training, and validated on age-balanced datasets such as FACES [ 14 ] , RaFD [ 38 ] , and DDCF [ 10 ] . The main contributions of this work are as follows: • We perform an in-depth analysis of age-related biases in deep FER models using XAI tools, identifying how expression recognition and model attention differ across age groups. • We propose and evaluate three bias mitigation strategies—Multi-task Learning, Multi-modal Input, and Age-weighted Loss—and show that they improve performance, particularly for elderly individuals. • We demonstrate that models trained on AffectNet with automatically estimated age labels can generalize well to other datasets, outperforming previous methods such as Huang et al. [ 31 ] on the elderly group from FACES. • We show, through heatmap analysis, that age-aware training strategies lead models to focus on more relevant facial regions for each age group, helping to explain the observed improvements in accuracy. All code is made publicly available at https://github.com/Xavi3398/FER-for-elderly . This paper is structured as follows. First, we review related work, focusing on how age influences FER and existing approaches for mitigating age-related bias. We then formulate our research questions, followed by a detailed description of our experimental setup, including models, datasets, XAI tools, and debiasing strategies. Finally, we present our results, discuss key findings, and outline directions for future research. 2 Related work Many psychological studies have investigated the impact of aging on facial expression recognition, revealing that the individuals perceiving the expressions are affected by age-related changes. For instance, older adults often experience difficulties in accurately recognizing certain emotions [ 32 , 57 ] . However, beyond perception, the structural changes in aging faces also affect how expressions are displayed and interpreted. Fölster et al. [ 18 ] analyzed how age-related facial changes, such as wrinkles and folds, impact the recognition of emotional expressions. Their findings emphasized that facial aging significantly alters the way emotions are conveyed and perceived. This conclusion is reinforced by Ko et al. [ 36 ] , who observed that older adults not only display more negative emotions but also engage more lower-face muscles than younger individuals. These differences in facial muscle activation suggest that age-related changes can lead to variations in expression intensity, making recognition more complex. These age biases in expression recognition can also be found in computational approaches. For instance, Grondhuis et al. [ 24 ] applied generative adversarial networks to analyze the increasing difficulty of recognizing expressions in older faces. Their study attributed this challenge to the gradual decline in facial muscle function, which affects the clarity and distinctiveness of expressions. Similarly, Sönmez [

Read the complete text