Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model

Muzammil Behzad, Guoying Zhao Information & Computer Science Department, King Fahd University of Petroleum & Minerals, Saudi Arabia Center for Machine Vision and Signal Analysis, University of Oulu, Finland Email: [email protected] , [email protected] · 2025

Read Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model by Muzammil Behzad, Guoying Zhao Information & Computer Science Department, King Fahd University of Petroleum & Minerals, Saudi Arabia Center for Machine Vision and Signal Analysis, University of Oulu, Finland Email: [email protected] , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19739.txt.

Introduction

Muzammil Behzad and Guoying Zhao present a vision-language model in two thousand twenty-five for reading facial emotions in three and four dimensions. Standard emotion tools use flat video, which can miss depth and motion cues. Their model, AffectVLM, merges several views for a richer read of expressions.

A joint learning framework pairs the multi-view input with a new loss function that speeds convergence to good features. Augmented text prompts strengthen the model's language side, and mixed view augmentation expands the visual data. The team also built a web app for real-time use and enabled distributed learning. Tests across several benchmarks showed top performance, the authors report. The work targets emotion tools that handle dynamic faces.

Catalog record

CREATORMuzammil Behzad, Guoying Zhao Information & Computer Science Department, King Fahd University of Petroleum & Minerals, Saudi Arabia Center for Machine Vision and Signal Analysis, University of Oulu, Finland Email: [email protected] , [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19739

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model Muzammil Behzad, Guoying Zhao Information & Computer Science Department, King Fahd University of Petroleum & Minerals, Saudi Arabia Center for Machine Vision and Signal Analysis, University of Oulu, Finland Email: [email protected] , [email protected] Abstract In this paper, we introduce AffectVLM, a vision-language model designed to integrate multiviews for a semantically rich and visually comprehensive understanding of facial emotions from 3D/4D data. To effectively capture visual features, we propose a joint representation learning framework paired with a novel gradient-friendly loss function that accelerates model convergence towards optimal feature representation. Additionally, we introduce augmented textual prompts to enhance the model’s linguistic capabilities and employ mixed view augmentation to expand the visual dataset. We also develop a Streamlit app for a real-time interactive inference and enable the model for distributed learning. Extensive experiments validate the superior performance of AffectVLM across multiple benchmarks. I Introduction Facial Expression Recognition (FER) is a key research area within affective computing, focusing on analyzing and interpreting human emotions through facial analysis, with several applications in human-computer interaction [ 1 ] , mental health [ 2 ] , education [ 3 ] , and more [ 4 ] . Given the complexity of the human emotions and the pioneering emotion theory proposed by Ekman & Friesen [ 5 ] , researchers have developed various models to identify and resolve potential research gaps [ 6 ] . Fig. 1: Overview of our proposed AffectVLM: Affective Vision-Language Model. The literature presents various methods to learn from the underlying 3D facial geometry. In this regard, the most widely used approaches include local feature-based methods [ 7 , 8 , 9 ] , template-based methods [ 10 , 11 ] , curve-based methods [ 12 , 13 ] , and 2D projection-based methods [ 14 , 15 ] . In recent years, 3D/4D FER has gained a significant attention as it enables deep learning models to extract additional discriminative facial features given the facial depth axis. For example, Yin et al. [ 16 ] and Sun et al. [ 17 ] employed Hidden Markov Models (HMM) to capture temporal facial features from 4D facial scans. Likewise, Ben Amor et al. [ 18 ] demonstrated the effectiveness of a deformation vector field based on Riemannian analysis using a random forest classifier. Similarly, Sandbach et al. [ 19 ] utilized Hidden Markov Models (HMM) and GentleBoost to learn free-form representations of 3D frames. Additionally, the authors in [ 20 ] represented geometric coordinates and their normals as feature vectors, while another study [ 21 ] utilised dynamic local binary patterns (LBP) to recognize facial expressions using a support vector machine (SVM). In a related approach, the authors in [ 22 ] extracted features from polar angles and curvatures, proposing a spatio-temporal LBP-based feature extractor for recognition. Conversely, Li et al. [ 23 ] introduced an intriguing framework for automatic 4D Facial Emotion Recognition (FER) using a dynamic geometrical image network. They generated geometrical images by calculating differential quantities from the given 3D facial point clouds. The emotion prediction is achieved through score-level fusion of the probability scores derived from various geometrical images. I-A Motivations While effective, these traditional methods often rely on manually extracted features and localized cues, limiting performance and adaptability in the real-world scenarios. Inspired by the scaling victory of large language models (LLMs) [ 24 ] and large vision language models (LVLMs) [ 25 ] , we work towards developing a VLM with joint representation learning capabilities to benefit from the underlying facial patterns stored as muscle movements in 3D/4D faces [ 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] , for a significantly better facial expression recognition (FER). Specifically, instead of using conventional 2D faces only (e.g., [ 34 , 35 , 36 , 37 ] ), this emotion recognition involves classifying facial expressions from 3D/4D faces with added spatio-temporal features, and the substantial results [ 38 , 39 , 40 , 9 ] have validated its advantages. VLMs offer a promising solution by learning joint representations, improving generalization across diverse datasets without manual feature engineering. This highlights the need for advanced approaches that leverage VLMs, which can learn from both visual and textual data [ 41 ] . By aligning visual cues with semantic labels, VLMs enhance the robustness of FER systems in varying conditions. However, existing 3D/4D datasets have limited capacity, which hinders the performance of deep learning models. This underscores the importance of improved augmentation strategies, such as spatial transformations, temporal variations, and multiview combinations, to address data scarcity and overfitting. I-B Contributions To the best of our knowledge, existing literature lacks research on 3D/4D FER models utilizing VLMs. This gap arises primarily because implementing such models is not straightforward, accompanied by the complexity and variability in the 3D data structure. In this context, we introduce AffectVLM, a VLM based architecture [ 42 ] . The key features of our model are as follows: 1. We propose a joint embedding space for coherent feature representation to integrate multiview embeddings. 2. We also propose augmented textual prompts to extend the model’s understanding of emotional semantics by augmenting the training labels with extended textual propmts. 3. We introduce mixed view augmentation to generate diverse training samples by combining augmentations from multiple views, thereby enriching the dataset. 4. We employ a novel gradient-friendly loss function for model’s smoother convergence during training. 5. We design AffectVLM with the capability to leverage distributed training for improved scalability. 6. We develop a Streamlit application for user inference, enabling interactive access to the model for real-time emotion classification. Furthermore, it must be noted that AffectVLM is equally applicable to other downstream tasks like, face recognition, anti-spoofing and identity recognition. II AffectVLM: Affective Vision-Language Model Our AffectVLM model’s ability to incorporate multiviews offers a significant advantage with its robust and scalable implementation, making it highly effective and ready for deployment with minimal adjustments. II-A Learning Joint Embedding Space Representations As shown in Fig. 1 , we tailor our model to learn joint embedding space to facilitate coherent feature representation across both visual and textual modalities. We use contrastive language-image pre-training (CLIP) [ 42 ] as a backbone VLM model and extend it for multiviews in our problem statement. Specifically, we aim to achieve joint embedding of visual features from multiple facial views and corresponding textual prompts. The model processes multiview data, thereby capturing frontal, left, and right perspectives, allowing it to learn a rich, holistic representation of emotions from different facial angles by optimizing the loss function jointly with the features from all views in the embedding space. Simultaneously, we integrate augmented textual prompts into the learning process as discussed in the next sub-section. This joint learning framework ensures that different views of the same emotion are aligned in a shared embedding space with their textual counterparts, promoting a deeper understanding of the emotion. This approach not only improves the model’s ability to generalize across unseen data but also enhances its robustness in recognizing emotions from varying viewpoints and contextual diversity. AffectVLM supports multiple model engines such as ViT(16/32) [ 43 ] , and ResNet

Read the complete text