Progressive Multi-Source Domain Adaptation for Personalized Facial Expression Recognition

Muhammad Osama Zeeshan, Marco Pedersoli, Alessandro Lameiras Koerich, and Eric Granger · 2025

Read Progressive Multi-Source Domain Adaptation for Personalized Facial Expression Recognition by Muhammad Osama Zeeshan, Marco Pedersoli, Alessandro Lameiras Koerich, and Eric Granger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04252.txt.

Introduction

Faces differ from person to person, so expression reading software works best when it adapts to each user. Muhammad Osama Zeeshan and co-authors present a method in 2025 that adapts a model to a new person using many labeled examples from others, none from the target. The catch is that some source people differ too much, and learning from them can hurt.

Their method adds sources in stages, starting with the people most similar to the target and moving outward. A memory store keeps the most useful past samples so earlier learning is not lost as new sources arrive. Tests across several expression datasets, and across datasets, showed the staged approach beat methods that use all sources at once.

Catalog record

CREATORMuhammad Osama Zeeshan, Marco Pedersoli, Alessandro Lameiras Koerich, and Eric Granger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04252

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Progressive Multi-Source Domain Adaptation for Personalized Facial Expression Recognition Muhammad Osama Zeeshan, Marco Pedersoli, Alessandro Lameiras Koerich, and Eric Granger Abstract Personalized facial expression recognition (FER) involves adapting a machine learning model using samples from labeled sources and unlabeled target domains. Given the challenges of recognizing subtle expressions with considerable interpersonal variability, state-of-the-art unsupervised domain adaptation (UDA) methods focus on the multi-source UDA (MSDA) setting, where each domain corresponds to a specific subject, and improve model accuracy and robustness. However, when adapting to a specific target, the diverse nature of multiple source domains translates to a large shift between source and target data. State-of-the-art MSDA methods for FER address this domain shift by considering all the sources to adapt to the target representations. Nevertheless, adapting to a target subject presents significant challenges due to large distributional differences between source and target domains, often resulting in negative transfer. In addition, integrating all sources simultaneously increases computational costs and causes misalignment with the target. To address these issues, we propose a progressive MSDA approach that gradually introduces information from subjects (source domains) based on their similarity to the target subject. This will ensure that only the most relevant sources from the target are selected, which helps avoid the negative transfer caused by dissimilar sources. During adaptation, the source domains are introduced in a curriculum manner. We first exploit the closest sources to reduce the distribution shift with the target and then move towards the furthest while only considering the most relevant sources based on the predetermined threshold. Furthermore, to mitigate catastrophic forgetting caused by the incremental introduction of source subjects, we implemented a density-based memory mechanism that preserves the most relevant historical source samples for adaptation. Our extensive experiments 1 1 1 https://github.com/osamazeeshan/P-MSDA show the effectiveness of our proposed method on challenging FER datasets: Biovid, UNBC-McMaster, Aff-Wild2, and BAH. Further, performance is evaluated on a cross-dataset setting (UNBC-McMaster → \rightarrow BioVid) , showing the importance of gradually adapting to source subjects. I Introduction In recent years, there has been a growing demand for deep learning (DL) models that can perform well on FER across various industrial sectors such as in detecting suspicious or criminal behavior, automated emotion recognition, or the estimation of pain in health care [ 1 , 2 , 3 , 4 ] . Addressing the significant variability of facial expressions between individuals due to cultural and ethnic differences or varying capture conditions is a challenging problem because of the subtle expression in real-world applications that leads to a substantial disparity between the data used to train and test DL models [ 5 ] . Therefore, adapting a deep FER model to a specific individual (i.e., personalization) is important to maintain a high level of performance. Personalized FER has been extensively studied in the literature, primarily through supervised learning approaches and fine-tuning techniques [ 6 , 7 , 8 ] to capture individual-specific nuances. These approaches mostly rely on fully or weakly labeled data to adapt and create a personalized model for each subject. Fine-tuning a model using fully labeled data requires a costly annotation of samples, and this annotation may be ambiguous due to variability among the annotators [ 9 , 10 ] . Unsupervised domain adaptation (UDA) [ 11 , 12 ] is a promising alternative for leveraging unlabeled data in FER. Nevertheless, SOTA UDA techniques treat datasets as domains containing samples from mixed subjects across source and target domains, which limits the model’s ability to perform fine-grained adaptation [ 13 , 14 , 15 ] . Defining each subject as a domain can address this issue yet blending source data to perform UDA restricts its capacity to handle variations and diversity within the target domain. To address this challenge, multi-source (unsupervised) domain adaptation (MSDA) [ 16 , 17 , 18 ] has gained significant popularity as it incorporates information from multiple source domains to enhance model resilience to different target variations. In [ 18 ] , the authors introduced a subject-based domain adaptation method, where each domain consists of a distinct subject, thus multiple sources and a single target subject. The target domain consists of samples belonging to a single person, where the data are captured in a stationary environment and incorporate less effective diversity. This approach differs from the traditional MSDA methods that adapt to a target domain of mixed individuals. The subject-based method focused on adapting to a single individual enables more precise and targeted adaptation strategies to create a personalized FER model. Fig. 1: Comparison between subject-based MSDA and our proposed progressive MSDA. (a) Subject-based MSDA aligns all source domains simultaneously with the target using a Discrepancy-based, Self-supervised, Contrastive-learning, or Adversarial-based approaches. (b) Our Progressive MSDA first rank and gradually adapts source domains (subjects) based on their similarity to the target domain, optimizing the transfer process through sequential adaptation. Secondly, we construct a replay memory (domain) that retains key samples from previously adapted source domains, which are re-accessed after each adaptation. The discrepancy-based approach is applied to align the source and target domains. MSDA methods address the challenge of transferring knowledge across diverse domains (datasets) by aligning multiple source domains with a target domain representation. By leveraging data from multiple sources, MSDA aims to enhance model robustness and generalization by minimizing domain shift [ 19 , 20 ] and create a shared feature space where source and target distributions are closely matched. While effective in some cases, this approach forces all sources to align with the target data, often overlooking individual domain variability. In contrast, the subject-based domain adaptation method in [ 18 ] struggles with target adaptability under such alignment, as the inherent differences among source subjects can deviate significantly from the target domain. Although the adaptation to all the source subjects is beneficial for learning a better representation of diverse subjects, adapting to a single target subject does not always require incorporating every source subject. This is due to the subject’s negative transfer [ 21 , 22 ] caused by the large difference in distributions of source and target domains. For instance, if the target domain corresponds to a young caucasian woman and the source domains consist of a diverse group of individuals from varying personal characteristics such as age, ethnicity, and gender (illustrated in the Fig. 1 (a)). Integrating all the source domains will generate a large domain shift between source and target, causing a model to deviate, and creating difficulties in the adaptation process. Furthermore, it would not be feasible in real-world applications due to high computing and memory consumption. To address the challenges of developing a subject-based MSDA method for FER, we gradually introduce only the most relevant source domains (subjects) during the adaptation process. We introduce a method that integrates curriculum learning (CL) [ 23 ] and self-paced learning (SPL) [ 24 ] , leveraging prior knowledge from CL to prioritize easier source subjects during target domain adaptation. Simultaneously, SPL is employed to dynamically adapt the learning process, allowing the model to recalibrate and select relevant sources as training progresses. To effectively capture contextual information from each source subject while avoiding domain corruption, we adopt a progressive approach by adapting to one source at a time gradually. Drawing

Read the complete text