Causal-Ex : Causal Graph-based Micro and Macro Expression Spotting
Pei-Sze Tan*, Sailaja Rajanala, Arghya Pal, Raphaël C.-W. Phan, Huey-Fang Ong · 2025
Read Causal-Ex : Causal Graph-based Micro and Macro Expression Spotting by Pei-Sze Tan*, Sailaja Rajanala, Arghya Pal, Raphaël C.-W. Phan, Huey-Fang Ong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09098.txt.
Introduction
Pei-Sze Tan and four co-authors spot hidden facial expressions in twenty twenty-five. Macro and micro expressions reveal feelings a person may be concealing. Spotting them means marking when an expression starts, peaks, and ends in a video. Such skill can flag mental health issues early and guide timely support.
Facial action units, codes for muscle movements, help accuracy. But data sets can bake in biases, linking some movements to certain emotions by accident. The authors replace the usual adjacency data between action units with causal graphs. That strips out spurious links while keeping true drivers. Their system, Causal-Ex, aims for cleaner, fairer spotting. Readers get a method that watches for dataset bias as part of the task.
Catalog record
| CREATOR | Pei-Sze Tan*, Sailaja Rajanala, Arghya Pal, Raphaël C.-W. Phan, Huey-Fang Ong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09098
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal-Ex : Causal Graph-based Micro and Macro Expression Spotting Pei-Sze Tan*, Sailaja Rajanala, Arghya Pal, Raphaël C.-W. Phan, Huey-Fang Ong Abstract Detecting concealed emotions within apparently normal expressions is crucial for identifying potential mental health issues and facilitating timely support and intervention. The task of spotting macro and micro-expressions involves predicting the emotional timeline within a video, accomplished by identifying the onset, apex, and offset frames of the displayed emotions. Utilizing foundational facial muscle movement cues, known as facial action units, boosts the accuracy. However, an overlooked challenge from previous research lies in the inadvertent integration of biases into the training model. These biases arising from datasets can spuriously link certain action unit movements to particular emotion classes. We tackle this issue by novel replacement of action unit adjacency information with the action unit causal graphs. This approach aims to identify and eliminate undesired spurious connections, retaining only unbiased information for classification. Our model, named Causal-Ex ( Causal -based Ex pression spotting), employs a rapid causal inference algorithm to construct a causal graph of facial action units. This enables us to select causally relevant facial action units. Our work demonstrates improvement in overall F1-scores compared to state-of-the-art approaches with 0.388 on CAS(ME) 2 and 0.3701 on SAMM-Long Video datasets. keywords Micro-Expression ,Causality ,Graph-based networks † † Preprint uploaded to arXiv. The paper is under consideration at Pattern Recognition Letters. † † corresponding: Corresponding author 1 Introduction Facial expression spotting and recognition are intricately woven tasks. However, spotting forms the foundation of any recognition system [ 1 ] . Expression spotting entails the challenging task of identifying the frames that are most likely to convey emotions. These expressions can be very obvious, otherwise dubbed as macro expressions, or subtly hidden, namely micro-expressions. The occurrence of micro-expressions is momentary, and the corresponding intensity of facial muscle movements is low. Consequently, it is highly challenging to identify the onset, offset, and apex frames at which micro-expressions reach their peak [ 2 ] . Tasks involving the spotting of both macro and micro-expressions have applications in psychology, human-computer interaction, emotion recognition, lie detection, and various other domains. Figure 1: The image displayed above illustrates two sample videos labeled as (a) and (b). Accompanying them are their respective graphs depicting Region of Interest (ROI) adjacency and ROI causal relations. These causal graphs effectively represent the causal links between different ROIs. They play a crucial role in analyzing and comprehending the ways specific ROIs impact others, ultimately contributing to the visualization of emotional expressions. On the other hand, the adjacency graphs primarily showcase patterns of ROI co-occurrence, albeit with less informative value. These patterns might encompass considerable spurious or duplicated information. Previous literature has demonstrated the use of long short-term memory (LSTM) [ 3 ] to capture the temporal correlations within sequences of videos and optical flow features to detect the rate of change of these features [ 4 , 5 , 6 ] . Although the emerging graph-based methods [ 7 ] aims to uncover the intrinsic semantic connections within facial action units, they overlook the potential causal relationships that could establish cause-and-effect associations between groups of action units and resulting emotions. Figure 1 illustrates an instance showcasing the contrast between the action unit (AU) co-occurrence-based adjacency matrix and the AU causal graph. The adjacency-based approaches solely emphasize co-occurrence patterns, which could arise by chance, without accounting for the cause-and-effect associations among the action units. Moreover, the conventional techniques can be prone to noise, struggle with capturing delicate motions, and disregard the inadvertent inclusion of biases linked to data [ 8 ] . The dataset biases have been known to be associated with imbalanced distributions of the training data w.r.t an emotion class, gender, age or any other latent factors [ 8 ] . In this study, we address two challenges. Firstly, we use structural causal graphs to identify the causal semantics among facial regions’ feature space. Secondly, we devise an effective approach to assess the model bias using strategic counterfactuals, which effectively reveal the hidden biases. The proposed counterfactual debiasing process is conducted post-training and is notably faster and simpler to execute. To overcome the challenges and limitations in macro and micro-expression spotting of the previous works, we introduce a model named Causal-Ex , a Bias-aware AU Causal Graph-based facial Ex pression detector. This model possesses the capability to unveil causal relationships among AUs using minimal visual data, while also having the potential to mitigate dataset bias. Our contributions in this regard are three-fold: • Causal AU Relation Graph : We present an approach to generate AU-based causal relation graphs from a provided video, employing a causal structure discovery algorithm [ 9 ] . This stands in contrast to existing methods that depend on experts and pre-existing information to manually construct the action unit relation graph [ 7 ] . • Improved Spatial-Temporal Embedding Architecture : We introduce an enhanced framework for macro and micro-expression spotting, featuring a spatiotemporal graph embedding module designed to capture the complexities of emotional transitions. To improve the accumulation of temporal knowledge, we incorporate a graph convolutional network (GCN). The model’s ability to grasp subtle micro-expressions is further supported by maxpooling layers within the convolutional network. • Counterfactual Debiasing : We frame the process of debiasing as counterfactual inference [ 10 ] . Initially, counterfactuals are employed to interrogate the model, exposing its inherent biased decisions. Subsequently, a debiasing procedure is applied conditionally to rectify the model’s learned biases. After ablation analysis, our experiments show that the proposed model learns effectively with the causal graph structures that are computed from the input videos. The results outperform the baselines and the robustness of the model is demonstrated by testing with cross-database generalization capability. Figure 2: Illustration of all the proposed Causal-Ex and types of counterfactual inputs in this work. 2 Methodology Our formal architecture consists of three modules, namely the data pre-processing stage, the causal modeling stage and the final debiasing stage. Figure 2 gives a brief overview of our method. The data pre-processing stage is largely inspired by [ 7 ] and it consists of cropping for face alignment facial landmark detection followed by extracting the optical flow and mean directional mean optical flow (MDMO) features [ 11 ] . MDMO features track optical flow changes in essential Regions of Interest (ROI) on the face, which are closely connected to facial action units, thus enhancing performance. We detail the rest of the modules below. 2.1 The Proposed Causal-Ex The data pre-processing stage outputs an adjacency graph of facial ROIs that are a combination of a set of AUs. This adjacency graph is an undirected graph D and purely captures the co-occurrence patterns. During stage two, the Causal-Ex module aims to learn the causal graph from the AU adjacency graph using the approach of [ 9 ] . Table 1: Probability distribution of expression type and frames difference between two types of frames where f_{onset} , f_{apex} , f_{offset} indicate the onset, apex and offset frames respectively. For instance, SAMM-LV contains 68.33% micro expressions (ME) and 31.67% macro expressions (MaE). On average, 31.7% of videos have offset and onset frames within <101 frames (column 2), while in 59.2% of videos,