PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer Vision CCS: Computing methodologies Activity recognition and understanding

Ibtissam Saadi, Abdenour Hadid, Douglas W. Cunningham, Abdelmalik Taleb-Ahmed, Yassin El Hillali · 2025

Read PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer Vision CCS: Computing methodologies Activity recognition and understanding by Ibtissam Saadi, Abdenour Hadid, Douglas W. Cunningham, Abdelmalik Taleb-Ahmed, Yassin El Hillali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16945.txt.

Introduction

Ibtissam Saadi, Abdenour Hadid, Douglas Cunningham, Abdelmalik Taleb-Ahmed, and Yassin El Hillali wrote this paper in two thousand twenty-five. Their topic is how to teach a large vision and language model to read faces in motion, a task called dynamic facial expression recognition.

They present PE-CLIP, a method that adapts the CLIP model while training far fewer parts than full fine-tuning. It uses two small adapters, one that tracks how a face changes over time and one shared by the text and image sides. Learnable prompts help the two sides line up. Tests on two benchmark data sets show results close to the best known methods, with much less training cost.

Catalog record

CREATORIbtissam Saadi, Abdenour Hadid, Douglas W. Cunningham, Abdelmalik Taleb-Ahmed, Yassin El Hillali
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16945

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer Vision CCS: Computing methodologies Activity recognition and understanding Ibtissam Saadi, Abdenour Hadid, Douglas W. Cunningham, Abdelmalik Taleb-Ahmed, Yassin El Hillali Abstract . The emergence of Vision-Language Models (VLMs) like CLIP (Contrastive Language-Image Pretraining) provides appealing solutions to various vision problems including Dynamic Facial Expression Recognition (DFER). However, most of the proposed approaches face major challenges, particularly related to inefficient full fine-tuning of the encoders and the complexity of the models. Moreover, some of the proposed methods seem to struggle with suboptimal performance due to (i) poor alignment between textual and visual representations, and (ii) ineffective temporal modeling. To address these challenges, we propose PE-CLIP, a parameter-efficient fine-tuning (PEFT) framework that elegantly adapts CLIP for dynamic facial expression recognition, requiring significantly reduced number of trainable parameters while maintaining high accuracy. At its core, to enhance efficiency and performance, PE-CLIP introduces two specialized adapters namely a Temporal Dynamic Adapter (TDA) and a Shared Adapter (ShA). The temporal dynamic adapter is a GRU-based module with a dynamic scaling mechanism, capturing sequential dependencies while adaptively modulating the contribution of each temporal feature to emphasize the most informative ones while mitigating irrelevant variations. The shared adapter is a lightweight adapter refine representations within both textual and visual encoders, ensuring consistent feature processing while maintaining parameter efficiency. Additionally, we leverage Multi-modal Prompt Learning (MaPLe), which introduces learnable prompts to both visual and action unit-based textual description inputs, further improving the semantic alignment between modalities and enabling the efficient adaptation of CLIP for dynamic tasks. We evaluate our proposed PE-CLIP on two benchmark datasets, namely DFEW and FERV39K, achieving competitive performance compared to state-of-the-art methods while requiring fewer trainable parameters. By striking an optimal balance between parameter efficiency and performance, PE-CLIP sets a new benchmark in resource-efficient DFER. The source code of the proposed PE-CLIP will be publicly available at https://github.com/Ibtissam-SAADI/PE-CLIP . 1. Introduction Early research on Facial Expression Recognition (FER) mainly focused on static FER (SFER), which analyzes individual images to identify facial expressions ( Chattopadhyay et al. 2020 ; Saadi et al. 2023 ) . Fundamentally, static images may fail capturing the temporal dynamics of facial movements - an essential factor in understanding human emotions. Dynamic FER (DFER), on the other hand, utilizes video sequences to analyze temporal changes in facial expressions, capturing the progression of facial dynamics over time. This enables a more nuanced understanding of emotional states, making DFER better suited for real-world applications. In particular, the availability of in-the-wild DFER datasets, such as DFEW ( Jiang et al. 2020 ) , FERV39k ( Wang et al. 2022a ) , and MAFW ( Liu et al. 2022a ) , which closely reflect real-world conditions, has advanced the field but also presents inherent challenges. These include occlusions, motion blur, non-frontal faces, and low-resolution images, all of which hinder the accurate analysis of dynamic facial features. Researchers have explored various approaches to address these challenges. Early methods used Convolutional Neural Networks (CNNs) for spatial feature extraction, combined with Recurrent Neural Networks (RNNs) for temporal modeling (CNN-RNN) ( Sun et al. 2020 ; Lee et al. 2020 ) . Other approaches leveraged 3D Convolutional Neural Networks (3D CNNs) ( Wang et al. 2023a ; Fan et al. 2016 ; Khanna et al. 2024 ) to jointly model spatial and temporal information, providing a unified representation of facial dynamics. Transformer-based architectures ( Zhao and Liu 2021 ; Li et al. 2023 ; Xia and Jiang 2023 ) have also been proposed to leverage global attention and long-range dependency modeling to improve spatio-temporal feature learning. Despite of the notable advances in the proposed works, some challenges still remain. The primary issue lies in the effective modeling of the complex temporal correlations within facial expression sequences, especially in the presence of subtle and hidden expressions. Furthermore, the scarcity of the training data in existing DFER datasets limits the development of deep models. To tackle these challenges, recent works like MAE-DFER ( Sun et al. 2023 ) and MARLIN ( Cai et al. 2023 ) have utilized large-scale self-supervised pretraining on unlabeled facial videos to enhance learning from limited data and improve temporal modeling through techniques such as mask modeling. However, despite achieving promising results, such methods usually involve high computational costs due to the heavy pretraining and inadequate representation of subtle temporal variations in facial expressions. More recently, researchers have begun exploring vision-language models due to their exceptional generalization capabilities across various downstream tasks. These models, such as Contrastive Language-Image Pretraining (CLIP) ( Radford et al. 2021 ) , represent a significant shift from vision-only approaches by integrating textual and visual modalities. By leveraging the alignment between visual and textual input, vision-language models have shown great promise in enhancing the understanding of facial expressions. Recent works like CLIPER ( Li et al. 2024a ) and DFER-CLIP ( Zhao and Patras 2023a ) have adapted CLIP for DFER, introducing textual prompts that provide additional semantic information to visual features and achieving notable improvements in performance. However, while these CLIP-based methods have made significant progress, they also face new challenges. Many of the approaches require full fine-tuning of the image encoder, leading to high computational costs and resource-intensive training. This limits their practicality, especially in resource-constrained scenarios. Furthermore, temporal modeling in these methods often relies on temporal average pooling or single-layer Transformers, which struggle to capture complex short- and long-term dependencies crucial for dynamic facial expression analysis. Additionally, while textual prompts provide valuable semantic context, existing methods often fail to fully exploit enriched and task-specific textual descriptions tailored to dynamic facial sequences. This limitation reduces the alignment between visual and textual modalities, leading to suboptimal performance in capturing dynamic facial expressions. To mitigate the high computational cost associated with fine-tuning large pre-trained models, recent efforts have explored parameter-efficient fine-tuning (PEFT) strategies ( Houlsby et al. 2019 ; Chen et al. 2022 ) . However, these approaches primarily focus on single-modality adaptation and employ simplistic temporal modeling techniques. They lack dedicated temporal adapters specifically designed for CLIP in DFER. From the observations above, we believe that there is a need for methods that leverage advancements in PEFT while incorporating specialized temporal modeling mechanisms. Based on this motivation, we propose PE-CLIP, a novel parameter-efficient adaptation framework designed to adapt CLIP for DFER. PE-CLIP introduces specialized components to efficiently model temporal dynamics, improve alignment between the two modalities’ representations, and achieve competitive performance with reduced computational cost. Unlike prior works that rely on computationally expensive architectures or inadequate temporal modeling, PE-CLIP integrates adapters and prompt learning to provide a more efficient and robust solution. Figure 1. Performance comparison of dynamic facial expression recognition on the DFEW dataset. This chart highlights the trade-off between the number of tunable parameters (in millions) and model accuracy (measured as weighted average recall, WAR). The bubble size represents the model size, indicating the total number of trainable parameters. Our proposed framework, PE-CLIP (highlighted in red), achieves superior performance with significantly fewer tunable parameters ( < 6% of the whole model’s parameters) compared to state-of-the-art methods. Models included in the comparison are IAL ( Lee et al. 2023 ) , CEFLNet

Read the complete text