Enhancing cross-subject emotion recognition precision through unimodal EEG: a novel emotion preceptor model Authors: Yihang Dong, Changhong Jing, Mufti Mahmud, Michael Kwok-Po Ng, Shuqiang Wang Published: 2024 ABSTRACT AbstractAffective computing is a key research area in computer science, neuroscience, and psychology, aimed at enabling computers to recognize, understand, and respond to human emotional states. As the demand for affective computing technology grows, emotion recognition methods based on physiological signals have become research hotspots. Among these, electroencephalogram (EEG) signals, which reflect brain activity, are highly promising. However, due to individual physiological and anatomical differences, EEG signals introduce noise, reducing emotion recognition performance. Additionally, the synchronous collection of multimodal data in practical applications requires high equipment and environmental standards, limiting the practical use of EEG signals. To address these issues, this study proposes the Emotion Preceptor, a cross-subject emotion recognition model based on unimodal EEG signals. This model introduces a Static Spatial Adapter to integrate spatial information in EEG signals, reducing individual differences and extracting robust encoding information. The Temporal Causal Network then leverages temporal information to extract beneficial features for emotion recognition, achieving precise recognition based on unimodal EEG signals. Extensive experiments on the SEED and SEED-V datasets demonstrate the superior performance of the Emotion Preceptor and validate the effectiveness of the new data processing method that combines DE features in a temporal sequence. Additionally, we analyzed the model’s data flow and encoding methods from a biological interpretability perspective and validated it with neuroscience research related to emotion generation and regulation, promoting further development in emotion recognition research based on EEG signals.Keywords: Emotion recognition, EEG, Temporal causal network Introduction Affective computing is a significant research direction in computer science, neuroscience, and psychology, aiming to enable computers to recognize, understand, and respond to human emotional states. As affective computing technology matures, human sensory, cognitive, and operational capabilities will be significantly expanded, providing unprecedented depth and breadth in human-computer interaction. Accurate identification of the true emotional state of the target is paramount for effective affective computing tasks. To achieve accurate detection and recognition of the target’s true emotional state, researchers have increasingly focused on emotion recognition methods based on physiological signals. Among these, EEG signals, which directly measure brain activity, are seen as especially promising. EEG can detect changes in brain activity across different emotional states and offers advantages like being non-invasive, affordable, and convenient for data collection [1, 2]. Consequently, using EEG signals for emotion recognition tasks has become a research hotspot and has shown broad application prospects in affective computing. With the rapid proliferation of multimodal data and learning methods, emotion recognition tasks based on EEG signals have also introduced auxiliary modalities such as eye movement features, facial expressions, and electrocardiograms. Multimodal data indeed improve the accuracy of emotion recognition tasks and promote research development in this field [3–11]. However, in practical applications, the synchronous collection of multimodal signals imposes high equipment and environmental requirements, hindering the practical transition from research to industry and significantly affecting the convenience of using EEG signals. Research on emotion recognition based on EEG signals faces challenges. Each individual has unique physiological and anatomical structures, and non-invasive EEG signals inevitably introduce noise interference from these physical layers, resulting in non-stationary characteristics that affect emotion recognition performance [12, 13]. This also means that an emotion recognition model trained on one subject’s EEG signals cannot be directly transferred to another subject’s data without severely impacting recognition performance. Therefore, this study proposes the Emotion Preceptor, a model for precise cross-subject emotion recognition based on unimodal EEG signals. This model innovatively introduces a Static Spatial Adapter to maximize the integration of spatial dimension information in EEG signals, reducing individual differences in EEG data and extracting robust and effective encoding information. The Temporal Causal Network then utilizes the rich information in the time dimension to extract the most beneficial temporal causal features for emotion recognition tasks, achieving precise emotion recognition based on unimodal EEG signals. Extensive experiments on the SEED and SEED-V datasets have demonstrated the superior performance of the Emotion Preceptor and validated the effectiveness of the new data processing method that combines DE features in a temporal sequence. Specifically, our contributions are as follows: We propose the Emotion Preceptor, a precise emotion recognition model based on unimodal EEG signals, which shows performance in cross-subject emotion recognition tasks comparable to or even surpassing methods using multimodal signals. We propose the Emotion Preceptor, a precise emotion recognition model based on unimodal EEG signals, which shows performance in cross-subject emotion recognition tasks comparable to or even surpassing methods using multimodal signals. We designed the Static Spatial Adapter specifically for handling cross-subject tasks, which extracts unified features and effective encoding from EEG signals with different subject differences. The Temporal Causal Network integrates the temporal causal information of these features and uses it for the final emotion recognition task. We designed the Static Spatial Adapter specifically for handling cross-subject tasks, which extracts unified features and effective encoding from EEG signals with different subject differences. The Temporal Causal Network integrates the temporal causal information of these features and uses it for the final emotion recognition task. We conducted extensive experiments on different datasets and analyzed the data flow and encoding methods in the model from the perspective of biological interpretability, while also validating it with neuroscience research related to emotion generation and regulation. We conducted extensive experiments on different datasets and analyzed the data flow and encoding methods in the model from the perspective of biological interpretability, while also validating it with neuroscience research related to emotion generation and regulation. Related works Research on emotion recognition based on human signals has been rapidly developing in recent years. EEG, as a type of brain signal that is relatively easy to obtain and can directly measure electrical activity on the scalp, can accurately capture changes in neural activity in the brain, making it one of the most powerful data types for emotion recognition projects. Since emotional changes are the result of the body’s comprehensive response to external variations [14, 15], it is natural to develop emotion recognition research using multiple data modalities simultaneously. Empirical research has demonstrated that the concurrent use of multimodal signals can significantly enhance emotion recognition performance [16]. EmotionMeter, introduced by Zheng et al. [3], is a multimodal emotion recognition framework that combines EEG and eye movements, achieving high accuracy and demonstrating effectiveness across sessions. The MMResLSTM network, proposed by Ma et al. [4], effectively learns correlations between EEG and other physiological signals for emotion recognition, achieving high classification accuracy on the DEAP dataset. Li et al. employed a multimodal attention-based BLSTM framework that transforms physiological signals into spectrograms and utilizes attention mechanisms to automatically learn temporal features, achieving efficient emotion recognition on the AMIGOS dataset [5]. Jia et al. introduced HetEmotionNet [6], a two-stream heterogeneous graph recurrent neural network that fuses multimodal physiological signals for emotion recognition, achieving superior performance on real-world datasets. Liu et al. compared the performance and robustness of DCCA and BDAE for multimodal emotion recognition [7], demonstrating that DCCA achieves state-of-the-art results and greater robustness across multiple datasets. Vazquez et al. demonstrated the suitability of a Transformer-based approach for multimodal emotion recognition from physiological signals [8], showing that pre-training improves performance on a state-of-the-art dataset. Jiang et al. introduced a novel multimodal emotion dataset combining EEG and eye movement signals, along with the Multimodal Adaptive Emotion Transformer (MAET) [9], which demonstrated superior performance in emotion recognition through extensive experiments. Dong et al. introduced a pre-trained model-based Multimodal Mood Reader for cross-subject emotion recognition [10], utilizing masked brain signal modeling and interlinked spatial-temporal attention mechanisms to achieve superior performance on public datasets. Despite the fact that the assistance of multimodal data such as eye movement features can indeed improve the performance of EEG-based emotion recognition tasks to some extent, obtaining these synchronously collected multimodal data may face considerable challenges in practical applications and deployment environments. Therefore, research on precise emotion recognition based on single-modal EEG signals is also continuously evolving. The R2G-STNN, introduced by Li et al., is a neuroscience-inspired EEG emotion recognition method that utilizes spatial and temporal neural networks with regional to global hierarchical feature learning, achieving state-of-the-art performance on the SEED database [17]. SST-EmotionNet, presented by Jia et al., is a spatial-spectral-temporal-based attention 3D dense network for EEG emotion recognition. This method effectively integrates diverse features and explores discriminative local patterns, achieving superior performance on real-world datasets [18]. Gong et al. developed ASTDF-Net, an attention-based spatial-temporal dual-stream fusion network for EEG-based emotion recognition. It effectively captures and integrates crucial discriminative features from complex spatiotemporal EEG patterns, outperforming state-of-the-art methods on public datasets [19]. To augment imbalanced EEG emotion datasets, Zhang et al. introduced Emotional Subspace Constrained Generative Adversarial Networks (ESC-GAN), which enhance emotion recognition performance and robustness against adversarial attacks [20]. Methodology Overview We propose Emotion Preceptor, a model designed for cross-subject precise emotion recognition tasks based on single-modal EEG signals. This model optimally integrates the spatial dimension information of EEG signals through the introduced Static Spatial Adapter. Subsequently, the Temporal Causal Network leverages the rich temporal information to extract the most beneficial temporal causal features for emotion recognition tasks, thereby achieving precise emotion recognition using single-modal EEG signals.The overview of Mood Preceptor is presented in Fig. 1. Overview of Emotion Preceptor. The entire Emotion Preceptor consists of three main modules: the Static Spatial Adapter, the Temporal Causal Network, and the Classify Network. These modules are respectively used to extract or enhance the corresponding feature information and are ultimately employed for the emotion classification task Static spatial adapter As a common brain signal with high temporal resolution, EEG’s spatial resolution is not prominent. To address this, we designed a Static Spatial Adapter aimed at efficiently integrating the spatial dimension information of EEG signals for precise emotion recognition tasks. First, we performed Short-Time Fourier Transform (STFT) on the preprocessed EEG signals, following the same specifications as previous studies, to calculate Differential Entropy (DE) features. DE features have been proven to be the most effective in emotion recognition of EEG signals, as they not only capture the variability of EEG signals but also encapsulate brain activity information relevant to emotional states. To further emphasize the temporal dimension in DE features, we refer to study [10] and employ the sliding window combination (SWC) method to increase the number of sliding windows, resulting in a new representation of EEG signals, X∈RN×C×F, where N represents the number of sliding windows used to calculate DE features, C represents the number of electrode channels, and F represents the number of signal frequency bands. Next, we performed a dimensional transformation aimed at channel aggregation to better integrate spatial information and obtain D^∈RN×(C·F). Here, D^ represents the spatial information. This transformation is significant as it enhances the separation and capture of spatial and temporal features in EEG signals, thereby providing richer information for subsequent emotion recognition tasks. For the features obtained after channel aggregation, we used stacked Spatial Units to further encode information in the spatial dimension, resulting in enhanced EEG features D. Each Spatial Unit is based on a multi-head attention mechanism, and the specific calculation process is described as follows: Generate Queries, Keys, and Values: Generate queries Q, keys K, and values V through projection,where LN represents LayerNormalization and Proj means LinearProjector: Subsequently, the feature attention is computed and projected back into the feature space, supplemented by skip connection: Through the aforementioned processing, the Static Spatial Adapter not only maximizes the integration of spatial information from EEG signals but also fully utilizes temporal information, thereby providing a more comprehensive and accurate feature representation for emotion recognition tasks. Temporal causal network We introduce a Temporal Causal Network (TCN) specifically designed to handle EEG features with integrated spatial information. This network aims to capture the temporal causal relationships between encoded EEG spatial signals while further refining the spatial information within individual time segments. The main structure of the TCN includes N serially connected Causal Units, with each sliding window’s features processed by a dedicated Causal Unit. For the Static Spatial Adapter output S with dimensions RN×(C·F), we divide it into N segments according to the sliding window dimensions, with each segment Si having dimensions RC·F. This method allows for a more comprehensive representation of the temporal complexity of EEG signals and provides a foundation for subsequent temporal causal relationship modeling. To capture the temporal causal relationships of EEG signals and refine the spatial brain region information, we designed the Causal Unit. Each Causal Unit comprises a Gated Recurrent Unit(GRU Cell) and trainable neurons. Specifically, in the i-th Causal Unit, the EEG feature segment Si at the current time point is analyzed alongside the intermediate driving information from the previous time point, i.e., the hidden variable ht-1 from i-1. These are jointly encoded through the GRU Cell, producing the intermediate information ht at the current time point i: Further causal relationship transformation Vi is achieved through a linear transformation and Tanh activation function: Finally, the feature representation Si^ incorporating temporal causal relationships at this time point is calculated: The output features of each Causal Unit are concatenated in the order of segmentation to obtain a feature representation E with dimensions RN×(C·F). The Temporal Causal Network effectively captures the temporal dependencies and causal relationships between brain regions in EEG signals, thereby improving the accuracy of emotion recognition. This method not only handles the complex temporal information of EEG signals but also further encodes spatial information, significantly enhancing the precision of emotion recognition. Experiment and analysis Datasets Extensive experiments are conducted on two commonly used emotion datasets, SEED [21] and SEED-V [7]. The SEED dataset includes EEG monitoring data and corresponding emotional labels (sad, happy, and neutral) from 15 subjects (seven males and eight females). Each subject completed three sessions, with each session comprising 15 trials, resulting in a total of 45 trials. Each trial included a 5 secs cue, a 4 mins movie clip, a 45-second self-assessment, and a 15-second rest period. Similarly, the SEED-V dataset was completed by 20 subjects (ten males and ten females), with each subject participating in three sessions totaling 45 trials. Each trial included a 15 secs cue, a 2 to 4 mins movie clip, and a 15 or 30-second self-assessment. The dataset includes EEG data, eye movement data, and corresponding labels for five emotions (disgust, fear, sad, happy, and neutral). Further analysis was conducted on the preprocessed EEG data provided by the dataset. Similar to the baseline method, EEG segments of varying lengths were segmented in reverse order using a 4-secs non-overlapping Hanning window. The 256-point Short-Time Fourier Transform (STFT) was then applied to calculate the DE features across five frequency domains(delta: 1–4 Hz, theta: 4–8 Hz, alpha: 8–14 Hz, beta: 14–31 Hz, gamma: 31–49 Hz): Notably, the results of every four sliding window calculations were concatenated in sequence. A detailed description of this process can be found in section 3.2. Baseline model and settings We compared our experimental results with those of other state-of-the-art models, both single-modal and multi-modal, trained on the SEED and SEED-V datasets. The baseline models include SVM [22], DGCNN [23], RGNN [24], SOGNN [25], BFE-Net [26], and Mood Reader [10]. Specifically, the experimental model was built and deployed using the PyTorch framework, with the NVIDIA TITAN RTX GPU. In each experiment, the dataset was randomly divided into training and testing sets in an 8:2 ratio. The model’s performance on the testing set was recorded for each training session, and the experimental results presented in this paper are the average performance metrics of multiple stable training models. Result Table 1 shows the performance of the Emotion Preceptor and baseline models on the SEED and SEED-V datasets, along with annotations regarding the categories of data used. The results indicate that, based solely on EEG unimodal signals, the Emotion Preceptor trained on different datasets consistently achieved superior performance in cross-subject emotion recognition tasks, with the highest classification accuracy for various emotions and excellent and stable classification standard deviations. Furthermore, our experiments further validate the effectiveness of the data processing method that combines DE features calculated from continuous sliding windows in EEG-based cross-subject emotion recognition tasks, greatly advancing the development and deployment of EEG unimodal-based emotion recognition tasks in real-world environments. Table 1Subject-independent classification performance (Acc/Std%) on SEED and SEED-V, where SWC represents whether to combine DE features in sliding window orderMethodModalitySEEDSEED-VDE Feature Raw EEGEyeAcc ↑Std ↓Acc ↑Std ↓w/o SWC w SWCSVM✓56.73 16.29 34.4513.67SVM✓70.2310.4256.4912.51DGCNN✓79.95 09.02 ––RGNN✓85.30 06.72 66.28 16.71 SOGNN✓86.81 05.79 74.53 07.90 BFE-Net✓92.29 04.65 ––Mood Reader✓✓91.6505.4278.7408.97Mood Reader✓✓93.1204.7584.3605.23Emotion Preceptor✓95.3504.2994.2807.51“Bold” indicates the best performance Subject-independent classification performance (Acc/Std%) on SEED and SEED-V, where SWC represents whether to combine DE features in sliding window order “Bold” indicates the best performance Figure 2 shows the experimental results of the classification accuracy and standard deviation of the model emotion preceptor for five emotions (disgust, fear, sadness, neutral, happy) in the cross-subject EEG emotion recognition task. The radar chart on the left displays the classification accuracy of each emotion with green lines. It can be seen from the figure that the accuracy of the model for all emotions is above 80%, with the highest accuracy for fear, approximately 99%. The radar chart on the right depicts the classification standard deviation for each emotion using blue lines, illustrating the model’s performance variability in recognizing different emotions. It can be observed that the standard deviation of the model for fear, neutral, and sadness emotions is relatively small, indicating stable performance. Similarly, the experimental results on the SEED dataset are shown in Fig. 3. Additionally, we also plotted the confusion matrix between the true emotions and the predicted results on SEED-V, as shown in Fig. 4. Overall, the above results strongly demonstrate the superior performance of the emotion preceptor in the cross-subject emotion recognition task. The results of the emotion recognition classification task on the SEED-V dataset using Emotion Preceptor clearly illustrate the classification accuracy and corresponding standard deviations for the five emotions: Disgust, Fear, Sadness, Neutral, and Happiness The results of the emotion recognition classification task on the SEED dataset (Sad, Neutral, and Happy) The confusion matrix (%) calculated based on the predicted results of the Emotion Preceptor on the SEED-V dataset and the actual emotion labels in the dataset Interpretation As shown in Fig. 5, part (a) displays the electrode positions for collecting EEG signals, while parts (b) to (e) and (f) to (i) visualize the changes in two sets of EEG signals after processing by the Spatial Unit. In the first group (b) to (e), the EEG signals correspond to the emotion ’fear’, with attention areas gradually concentrating in the prefrontal region, the right parietal and right parieto-occipital regions, and the right central region, exhibiting high levels of activity. This further corroborates existing neuroscience research, indicating that increased activity in the right prefrontal cortex is typically associated with emotion processing [27–29], particularly the regulation of negative emotions [30]. In contrast, the second group of EEG signals (f) to (i) corresponding to the emotion ’happy’ shows more focus on the frontal and left temporal regions, which also aligns with existing neuroscience research [31]. Additionally, since the right parietal and right parieto-occipital regions are usually associated with visuospatial processing, attention allocation, and visual information integration, this to some extent explains that the network captures information from visual-related regions for emotion recognition tasks, consistent with the SEED-V dataset’s video-based emotional stimuli. The distribution map of electrode positions, subplot (a), for the datasets used in the experiment is shown, along with the spatial aggregation changes and activation level changes of EEG features as processed by the Spatial Unit.The subplots (b to e) show the encoding changes in EEG signals corresponding to the emotion ’fear’, while subplots (f to i) correspond to ’happy’ Ablation To demonstrate the contribution of each module in the emotion preceptor to the emotion recognition task, we conducted detailed ablation experiments on the SEED-V dataset, showing the impact of different module combinations on the final performance. The specific experimental results are annotated in Table 2, proving the effectiveness of our adopted multi-module combination architecture. Table 2Ablation Study Results (Acc/Std%) on SEED and SEED-V dataset, where SU stands for Spatial Unit and TCN stands for Temporal Causal NetworkMethodModalitySEEDSEED-VDE FeatureAcc ↑Std ↓Acc ↑Std ↓w/o SWC w SWCSU × 2✓87.9506.4782.8413.55SU × 6✓88.6306.8383.2411.31SU × 4✓88.7506.2985.5010.72SU × 2 + TCN✓93.5305.4190.9808.77SU × 6 + TCN✓95.0705.3292.1608.42SU × 4 + TCN✓95.3504.2994.2807.51“Bold” indicates the best performance Ablation Study Results (Acc/Std%) on SEED and SEED-V dataset, where SU stands for Spatial Unit and TCN stands for Temporal Causal Network “Bold” indicates the best performance Conclusion This study proposes and validates a precise emotion recognition model based on unimodal EEG signals, termed Emotion Preceptor. By introducing a dedicated Static Spatial Adapter and a Temporal Causal Network, Emotion Preceptor effectively captures and integrates spatial and causal temporal features in EEG signals, achieving superior performance in cross-subject emotion recognition tasks. Compared to studies leveraging multimodal signals for emotion recognition, our method achieves superior performance with fewer devices in more streamlined, real-world environments, while also demonstrating high efficiency in emotion classification. Furthermore, we conducted a detailed exploration of different stages of the model from the perspective of biological interpretability and validated it with high confidence from a neuroscience perspective. This provides more references for the development of emotion recognition research based on EEG signals and creates more possibilities for detailed mechanistic research on brain signals such as EEG. Acknowledgements Not applicable. Funding This work was supported by the National Natural Science Foundations of China under Grant 62172403, the Distinguished Young Scholars Fund of Guangdong under Grant 2021B1515020019; M.Ng's research is funded by HKRGC GRF 17300021, HKRGC CRF C7004-21GF, and Joint NSFC and RGC N-HKU769/21. Availability of data and materials The datasets SEED and SEED-V used in this study are publicly available at https://bcmi.sjtu.edu.cn/~seed/index.html. Declarations Ethics approval and consent to participate Not applicable. Consent for publication All the authors agree consent for publication. Competing interests There is no any Conflict of interest. Footnotes Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets SEED and SEED-V used in this study are publicly available at https://bcmi.sjtu.edu.cn/~seed/index.html.