Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification

Ruimin Peng, Zhenbang Du, Changming Zhao, Jingwei Luo · 2024

Read Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification by Ruimin Peng, Zhenbang Du, Changming Zhao, Jingwei Luo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15224.txt.

Introduction

Ruimin Peng and co-authors classify seizure subtypes from brain wave recordings in a 2024 paper. Naming the exact subtype matters for precise diagnosis of epilepsy, but deep learning tools usually need far more labeled data than clinics have.

Their transformer has several branches that teach each other. One branch reads the raw brain signal, while others read its wavelet views at different frequency bands. Knowledge passes between the branches so a small labeled set goes further. Every second encoder block is replaced by one of these multi branch blocks. On two public data sets, the model beat both classic machine learning and recent deep learning tools.

Catalog record

CREATORRuimin Peng, Zhenbang Du, Changming Zhao, Jingwei Luo
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15224

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification Thanks: R. Peng, Z. Du, W. Liu and D. Wu are with the Belt and Road Joint Laboratory on Measurement and Control Technology, Huazhong University of Science and Technology, Wuhan 430074, China. Thanks: C. Zhao is with the AI Platform, Software Engineering Research Center, Dongfeng Corporation Research & Development Institute, Wuhan 430072, China. Thanks: J. Luo is with the China Electronic System Technology Co., Ltd., Beijing 100089, China. Thanks: X. Chen is with the Guangdong Provincial Key Laboratory of Human-Augmentation and Rehabilitation Robotics in Universities, Southern University of Science and Technology, Shenzhen, 518055, China. Thanks: X. Chen and D. Wu are the corresponding authors. E-mail: [email protected], [email protected]. Ruimin Peng, Zhenbang Du, Changming Zhao, Jingwei Luo Abstract Cross-subject electroencephalogram (EEG) based seizure subtype classification is very important in precise epilepsy diagnostics. Deep learning is a promising solution, due to its ability to automatically extract latent patterns. However, it usually requires a large amount of training data, which may not always be available in clinical practice. This paper proposes Multi-Branch Mutual-Distillation (MBMD) Transformer for cross-subject EEG-based seizure subtype classification, which can be effectively trained from small labeled data. MBMD Transformer replaces all even-numbered encoder blocks of the vanilla Vision Transformer by our designed multi-branch encoder blocks. A mutual-distillation strategy is proposed to transfer knowledge between the raw EEG data and its wavelets of different frequency bands. Experiments on two public EEG datasets demonstrated that our proposed MBMD Transformer outperformed several traditional machine learning and state-of-the-art deep learning approaches. To our knowledge, this is the first work on knowledge distillation for EEG-based seizure subtype classification. I Introduction Epilepsy is a widespread neurological disorder characterized by the rapid and early abnormal electrical activity of neurons in the brain, affecting more than 50 million people globally [ 1 , 2 ] . Among them, over 30% have intractable epilepsy, which significantly impacts the patients’ emotional, behavioral, and cognitive functions, severely limiting their ability to engage in daily activities [ 3 , 2 ] . Furthermore, disruptions in cognition and consciousness in severe cases impose significant risks to the patient’s safety and well-being. Consequently, precise diagnosis and effective treatment for epilepsy are very important. Clinical diagnosis of epilepsy heavily relies on the expertise of medical professionals to analyze patients’ electroencephalogram (EEG), which is demanding and time-consuming. Therefore, an automatic seizure diagnosis system, which analyzes EEG recordings automatically and rapidly, is highly desirable. Seizure detection, i.e., recognizing and marking the ictal fragments in EEG recordings, has been extensively studied in the literature; however, seizure subtype classification, which is critical in determining the appropriate therapies with medicine or surgery [ 4 ] , has not received enough attention. The 2017 International League Against Epilepsy guideline [ 5 , 6 ] categorizes epileptic seizures into generalized seizures [e.g., absence seizures (ABSZ), tonic seizures (TNSZ), and tonic-clonic seizures (TCSZ)], focal seizures (FSZ), and a combination of generalized and focal seizures. Different seizure types may have different prorogation patterns, e.g., ABSZ diffuses to the entire brain, whereas FSZ only affects a local area. Additionally, many seizures are caused by specific and personalized diseased tissue. These characteristics make it very challenging for automatic cross-patient seizure subtype classification. Conventional seizure subtype classification approaches usually involve three steps: data pre-processing, feature extraction, and classification [ 7 ] . Previous studies have extracted a large number of features to be used in machine learning models, e.g., support vector machine (SVM) [ 8 ] , ridge classifier (RC), logistic regression (LR) [ 9 ] , and gradient boosting decision tree (GBDT). However, these manually extracted features may not be optimal. Deep neural networks, e.g., convolutional neural networks (CNNs), recurrent neural networks, and autoencoders, have also been extensively used for automatic EEG feature extraction [ 10 ] . Recently, Transformer [ 11 ] based models have achieved great success in numerous tasks. A deep learning model may be very large. To reduce the model size and enhance the training efficiency, various techniques for model compression have been proposed, e.g., pruning [ 12 ] , low-rank approximation [ 13 ] , quantization [ 14 ] , compact network design [ 15 ] , and knowledge distillation [ 16 ] . Knowledge distillation typically employs a large teacher model to provide soft labels, guiding the training of a more compact student network. By leveraging the teacher’s knowledge, the smaller student model mimics the teacher’s output, achieving comparable or even better performance. Different from the conventional use of a fixed teacher model, Zhang et al. [ 17 ] proposed mutual learning, where every student model can learn from others. Fig. 1 illustrates the general ideas of knowledge distillation and mutual learning. Fig. 1: (a) Knowledge distillation; and, (b) mutual learning. Most model compression techniques require a large teacher model trained on big data. However, for data scarcity scenarios like seizure subtype classification, a large teacher model may not be available. In this situation, self-distillation could be used to distill a network’s knowledge and guide its training. Self-distillation can be implemented by using data augmentation or auxiliary structure [ 18 ] , as illustrated in Fig. 2 . Data augmentation based self-distillation enforces consistent predictions among augmented copies of the same instance or two instances from the same class. Auxiliary structure based self-distillation designs additional branches for the backbone network, and enforces them to be similar. Fig. 2: Self-distillation strategies. (a) Data augmentation; and, (b) auxiliary structure. This paper proposes a novel multi-branch mutual-distillation (MBMD) Transformer for EEG-based seizure subtype classification, which can be effectively trained from small labeled data. Our main contributions are: 1. We design a novel multi-branch Transformer. It uses multi-branch encoder blocks, which employ a multi-branch feedforward network (FFN) to process the wavelets decomposed from the raw EEG signals with different frequency bands. All wavelets use the same class label as the raw data, and the block output is an ensemble of all branches. 2. We propose a novel mutual-distillation strategy to facilitate knowledge transfer between the raw EEG data and the wavelets. It enables the model to uncover more hidden information and achieve improved performance. 3. To our knowledge, this study is the first attempt to introduce self-distillation in cross-patient seizure subtype classification. The remainder of this paper is organized as follows. Section II briefly reviews related works. Section III describes the details of our proposed MBMD Transformer. Section IV presents the performance of MBMD Transformer on two seizure subtype classification datasets. Finally, Section V draws conclusions and outlines some future research directions. II Related works This section reviews prior works on EEG-based seizure subtype classification and self-distillation. II-A EEG-based Seizure Subtype Classification Both traditional machine learning and deep learning have been used in EEG-based seizure subtype classification. An important consideration in traditional machine learning is to extract meaningful features from EEG signals. Vanabelle et al. [ 19 ] employed 22 features from the time and frequency domains to train an XGBoost model. Tian et al. [ 20 ] used time, frequency, and time-frequency domain features in multiple classifiers for seizure detection. Zhao et al. [ 21 ] extracted 41 time/spectral/time-frequency domain and nonlinear features to train semi-supervised and unsupervised transfer boosting algorithms for seizure subtype classification. Many

Read the complete text