Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing
Qingzhu Zhang, Jiani Zhong, Zongsheng Li, Xinke Shen, Quanying Liu, [2pt] Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China · 2025
Read Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing by Qingzhu Zhang, Jiani Zhong, Zongsheng Li, Xinke Shen, Quanying Liu, [2pt] Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22197.txt.
Introduction
Qingzhu Zhang, Jiani Zhong, Zongsheng Li, Xinke Shen, and Quanying Liu pretrain brain-wave models for emotion. General-purpose pretraining, they argue, teaches the wrong features. Emotion datasets disagree about where feelings live in the signal. They label emotions inconsistently. They also shift wildly between people.
Their framework pools several datasets for joint pretraining. A loss aligns the covariance structure across datasets. The representation steadies without heavy labeling or per-person calibration. A hybrid encoder pairs a Mamba-style linear attention channel with a space-time dynamics model. Long, tangled brain patterns can be followed. The result beat large-scale general EEG models. The average margin reported is over four percent. The work points toward emotion systems that travel across studies and subjects.
Catalog record
| CREATOR | Qingzhu Zhang, Jiani Zhong, Zongsheng Li, Xinke Shen, Quanying Liu, [2pt] Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22197
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing Qingzhu Zhang, Jiani Zhong, Zongsheng Li, Xinke Shen, Quanying Liu, [2pt] Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China Abstract Task-specific pre-training is essential when task representations diverge from generic pre-training features. Existing task-general pre-training EEG models struggle with complex tasks like emotion recognition due to mismatches between task-specific features and broad pre-training approaches. This work aims to develop a task-specific multi-dataset joint pre-training framework for cross-dataset emotion recognition, tackling problems of large inter-dataset distribution shifts, inconsistent emotion category definitions, and substantial inter-subject variability. We introduce a cross-dataset covariance alignment loss to align second-order statistical properties across datasets, enabling robust generalization without the need for extensive labels or per-subject calibration. To capture the long-term dependency and complex dynamics of EEG, we propose a hybrid encoder combining a Mamba-like linear attention channel encoder and a spatiotemporal dynamics model. Our method outperforms state-of-the-art large-scale EEG models by an average of 4.57% in AUROC for few-shot emotion recognition and 11.92% in accuracy for zero-shot generalization to a new dataset. Performance scales with the increase of datasets used in pre-training. Multi-dataset joint pre-training achieves a performance gain of 8.55% over single-dataset training. This work provides a scalable framework for task-specific pre-training and highlights its benefit in generalizable affective computing. Our code is available at https://github.com/ncclab-sustech/mdJPT_nips2025 . * * footnotetext: Corresponding authors 1 Introduction In domains like neuroscience, data is often limited, heterogeneous, and intrinsically linked to specific experimental tasks. This makes task-specific pre-training a particularly promising paradigm, especially for modalities such as EEG. Unlike generic large-scale pre-training, which aims to learn broad, task-agnostic representations, task-specific pre-training focuses on inducing representations tailored to a coherent problem class, incorporating task-relevant inductive biases and domain structure. Recent EEG foundation models [ 1 ; 2 ; 3 ; 4 ; 5 ; 6 ] adopt generic pre-training by aggregating heterogeneous datasets across multiple tasks. While effective on broad benchmarks like sleep staging or anomaly detection, such models often struggle with tasks that involve more complex and nuanced neural representations [ 1 ] . This performance gap stems from severe inter-task representation mismatch and the dilution of task-relevant signals during large-scale heterogeneous pre-training. Figure 1: Cross-dataset generalization through multi-dataset pre-training. (left) Challenges of task-specific EEG pre-training. (right) Challenges of inference. In contrast, task-specific multi-dataset pre-training, which leverages multiple datasets targeting the same cognitive function, offers a focused alternative. EEG-based emotion recognition is a natural candidate for this paradigm: it has growing availability of labeled datasets (e.g., SEED [ 7 ] , DEAP [ 8 ] , FACED [ 9 ] ) and strong practical value in affective computing and BCI applications. However, developing such a framework faces several technical challenges: large inter-dataset distribution shifts, heterogeneous EEG montages, inconsistent emotion category definitions, and substantial inter-subject variability. Existing approaches typically rely on one-to-one dataset adaptation and subject-specific fine-tuning [ 10 ; 11 ; 12 ; 13 ; 14 ; 15 ] (Table 1 ), limiting their generalization to unseen datasets or emotion categories (Fig. 1 ). These limitations call for a more principled task-specific pre-training strategy that aligns across datasets without requiring extensive labels or per-subject calibration. Inspired by the evidence that EEG signals exhibit similar second-order statistical properties (such as correlations or covariance) across datasets, which can be aligned through simple transformations [ 16 ; 17 ] , we develop a label-free, task-specific pretraining framework that learns transferable emotion representations by aligning statistical patterns, while maintaining robustness across both datasets and subjects. Our framework harmonizes cross-dataset feature structures and captures discriminative neural dynamics through tailored spatiotemporal modeling (Fig. 2 ). Our main contributions are as follows: • Multi-dataset joint pre-Training (mdJPT) : We develop a scalable multi-dataset pre-training framework for EEG-based emotion recognition, called mdJPT. We validated the superior performance of mdJPT over generic EEG foundation models in cross-dataset generalization. • Cross-dataset alignment (CDA) loss : We introduce a novel CDA loss that aligns second-order feature statistics across datasets, effectively mitigating inter-dataset and inter-subject distribution shifts. This enables zero-shot generalization to new emotion categories and unseen datasets. • Hybrid spatiotemporal encoder : We design a Mamba-like linear attention (MLLA)-based encoder [ 18 ] augmented with spatial transition convolutions and dynamic attention, enabling robust extraction of emotion-related long-term EEG dynamics and inter-channel dependencies. 2 Method 2.1 Task-specific multi-dataset joint pre-training Our work aims to develop a unified framework for task-specific multi-dataset EEG training and cross-dataset generalization (Fig. 2 ). We introduce two complementary alignment losses: the Covariance Distribution Alignment (CDA) loss and the Inter-Subject Alignment (ISA) loss. The CDA loss aligns global second-order statistics (i.e., inter-channel covariance structures) across subjects and datasets (Fig. 2 B). The ISA loss complements the CDA loss by a more fine-grained inter-subject alignment (Fig. 2 C). For the EEG encoder, we aim to design a physiologically plausible and computationally efficient architecture. It consists of an MLLA-based channel encoder (Fig. 2 D) and a spatiotemporal dynamics model (Fig. 2 E). The former captures long-term temporal dynamics from each EEG channel, and the latter integrates information across channels and time, enabling the model to learn coordinated patterns of brain activity. Our approach is validated under two settings: (i) cross-dataset subject-independent (few-shot) classification and (ii) zero-shot generalization. The few-shot classification consists of three stages: multi-dataset joint pre-training, classifier fine-tuning, and testing. In the first stage, an EEG encoder is pre-trained on multiple datasets using CDA and ISA loss. In the second stage, the encoder is frozen, and a classifier is fine-tuned on the extracted representations from a few labeled subjects in the target dataset. In the final stage, the trained model is evaluated on the remaining subjects of the target dataset (Fig. 2 A). The zero-shot generalization setting has no fine-tuning stage, with the pre-trained model directly generalized to a new dataset. Table 1: Comparison with existing cross-dataset emotion recognition methods. Multi-dataset Generalizable to Generalizable to Methods joint pre-training? new emotion categories? new subjects w/o finetuning? PESD [ 13 ] \times \times \times E 2 STN [ 14 ] \times \times \times JCFA [ 15 ] \times \times \times Imtiaz & Khan [ 10 ] \times \times \times SCMM [ 12 ] \times ✓ \times DBDG [ 11 ] \times \times ✓ mdJPT (Ours) ✓ ✓ ✓ Figure 2: The overall framework, model architectures, and loss designs of mdJPT. A ) The overall framework. Our approach comprises three stages: multi-dataset pre-training, optional classifier fine-tuning, and testing. The EEG encoder integrates an MLLA channel encoder with a spatiotemporal dynamics model. Pre-training combines CDA and inter-subject alignment (ISA) losses. The resulting model supports either zero-shot generalization or fine-tuning with a lightweight classifier on limited target data. Final evaluation is performed on held-out subjects. B) The cross-dataset alignment loss. The covariance is calculated from the latent representations, and covariance centroids of subjects from different datasets are aligned. C ) The inter-subject alignment loss. EEG samples from two subjects corresponding to the same time segment within a trial are regarded as positive samples, and those corresponding to different trials are regarded as negative samples. A contrastive loss is employed to pull together the positive samples and push apart the negative samples. D ) MLLA channel encoder. A channel-independent Mamba-like