Improving SSVEP BCI Spellers with Data Augmentation and Language Models
Joseph Zhang, Ruiming Zhang, Kipngeno Koech, David Hill, Kateryna Shapovalenko · 2024
Read Improving SSVEP BCI Spellers with Data Augmentation and Language Models by Joseph Zhang, Ruiming Zhang, Kipngeno Koech, David Hill, Kateryna Shapovalenko (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20052.txt.
Introduction
Joseph Zhang and co-authors improve brain-computer speller systems that read steady visual brain signals. These spellers let people with disabilities pick letters by gaze. Flickering targets evoke distinct brain patterns for each choice.
Deep networks for the task often fail on new users. Brain data vary a lot between people and sessions. The team tried data augmentation to harden the models. Masking and noise methods expanded the training data cheaply. A character language model then added word context to decoding. Letter guesses improved when language sense guided them. Gains over baseline reached a few points in tests. The hybrid design suits real communication aids for patients.
Catalog record
| CREATOR | Joseph Zhang, Ruiming Zhang, Kipngeno Koech, David Hill, Kateryna Shapovalenko |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-20052
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving SSVEP BCI Spellers with Data Augmentation and Language Models Joseph Zhang, Ruiming Zhang, Kipngeno Koech, David Hill, Kateryna Shapovalenko Abstract Steady-State Visual Evoked Potential (SSVEP) spellers are a promising communication tool for individuals with disabilities. This Brain-Computer Interface utilizes scalp potential data from (electroencephalography) EEG electrodes on a subject’s head to decode specific letters or arbitrary targets the subject is looking at on a screen. However, deep neural networks for SSVEP spellers often suffer from low accuracy and poor generalizability to unseen subjects, largely due to the high variability in EEG data. In this study, we propose a hybrid approach combining data augmentation and language modeling to enhance the performance of SSVEP spellers. Using the Benchmark dataset from Tsinghua University, we explore various data augmentation techniques, including frequency masking, time masking, and noise injection, to improve the robustness of deep learning models. Additionally, we integrate a language model (CharRNN) with EEGNet to incorporate linguistic context, significantly enhancing word-level decoding accuracy. Our results demonstrate accuracy improvements of up to 2.9 percent over the baseline, with time masking and language modeling showing the most promise. This work paves the way for more accurate and generalizable SSVEP speller systems, offering improved communication solutions for individuals with disabilities. Keywords: SSVEP, Brain-Computer Interface, EEG, Deep Learning, Letter Frequency, Language Models, RNN 1 Introduction Brain-Computer Interfaces (BCIs) are systems that allow direct communication between the brain and an external device. BCIs have transformative potential, particularly for individuals with severe physical disabilities, offering applications in communication, environmental control, and mobility. Our group is primarily interested in steady-state visually-evoked potential (SSVEP), which is a neural signal that occurs when a person looks at flickering stimuli at specific frequencies. This signal is commonly used in a speller task, where a user will try to spell words by looking at a computer screen with a matrix of characters each flickering at specific frequencies, and their SSVEP data is recorded using electroencephalography (EEG). This data is preprocessed and then decoded to determine what the user was looking at, and recently, the use of deep neural networks (DNNs) for decoding SSVEP has become more viable and effective, specifically the use of EEGNet, a CNN that is commonly used in EEG research. However, this process is still prone to errors due to the noisy and highly variable nature of EEG data. One especially difficult area is generalization, as EEG-based DNNs tend to not generalize well into new data. Therefore, our group experimented with various components of the BCI data pipeline to improve the efficacy of SSVEP DNN decoders. To improve SSVEP BCI performance, we explored data augmentation and language modeling. Data augmentation aims to increase the accuracy and stability of a model by creating more data to be used in training which is derived from the original dataset. This new data can be transformed from the dataset using a variety of functions. For our project, we plan to combine the common augmentation of noise addition and Fourier domain random phase shifting to improve our baseline model’s performance. We also incorporated letter frequency into character classification as the real-world application of these speller tasks would involve writing words and phrases which can be predicted using prior inputs. A character-based Recurrent Neural Networks (RNN) would dynamically adjust EEGNet’s output, emphasizing linguistically-probable characters. The final result would be a hybrid model containing both EEGNet and our language model named "CharRNN", which would work in parallel to decode each letter observed in the SSVEP speller. 2 Literature Review • [ Lashgari et al.(2020)Lashgari, Liang, and Maoz ] is a very useful overview paper of various augmentation techniques used in the field of deep learning for EEG. It reviews the common augmentations as well as their effectiveness on different types of problems for BCI-related tasks. It splits the types of data augmentation into six main categories: noise addition, GAN for generating new data, sliding or overlapping windowing, sampling, Fourier transform transformations, and recombination after segmentation. The authors discuss the current state of the art in each of these techniques and which techniques generally perform better for different tasks such as motor imagery, motor tasks, and visual tasks. We plan to use this paper as well as the papers referenced in the article to determine a new augmentation technique. • [ Ding et al.(2024)Ding, Liu, Guan, and Chen ] introduces an innovative data augmentation technique known as EEG mask encoding (EEG-ME) for deep learning (DL)-based classification in SSVEP-based BCIs. This method addresses a common challenge in DL models for BCIs: overfitting caused by the limited availability of EEG data. The proposed EEG-ME method improves model generalization by masking portions of the EEG data, thereby encouraging the model to learn more robust features. To validate the effectiveness of EEG-ME, three distinct network architectures were employed: CNN-Former (a hybrid model integrating convolutional neural networks and Transformer), time-domain-based CNN (tCNN), and a lightweight architecture known as EEGNet. Testing was conducted on publicly available benchmark and BETA datasets. The study demonstrates that EEG-ME substantially improves the average classification accuracy across these architectures and datasets. • [ Graves(2013) ] introduced a comprehensive framework for RNNs in sequence-based tasks, particularly emphasizing their ability to capture temporal dependencies over long sequences. By using a hidden state that evolves as the network processes each time step in the input, RNNs can retain information about previous elements, allowing for context-based predictions of characters, words, and phrases. For our project, we implemented a character-driven RNN to augment the classification probabilities of specific characters to aid the accuracy of EEGNet. • [ Wan et al.(2023)Wan, Cheng, Li, Zhu, and Duan ] presents GDNet-EEG, an attention-aware deep neural network based on group depth-wise convolution for recognizing SSVEP stimulation frequencies, with potential applications in early glaucoma diagnosis. The model employs group depth-wise convolution to extract temporal and spectral features from EEG signals across brain regions. It also integrates an attention mechanism to highlight essential brain regions for improved feature extraction. The model is validated on two public SSVEP datasets (Benchmark and BETA), as well as their combined dataset. With 1-second input signals, GDNet-EEG achieves average classification accuracies of 84.11%, 85.93%, and 93.35% on the Benchmark, BETA, and combined datasets, respectively. Compared to baseline models, accuracies improve by 1.96% to 18.2%. This paper is especially important as we will use it to compare our baseline implementation. The remaining literature was useful in researching approaches to improving SSVEP-based DNNs but ultimately was not used for our final implementations. • [ Guney et al.(2022)Guney, Oblokulov, and Ozkan ] introduces a novel DNN architecture for steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) that significantly improves information transfer rates (ITRs) for BCI speller systems. The BCI speller task involves looking at various characters/symbols on a grid on a computer screen and attempting to spell words/phrases using just the input of sight, which is very beneficial to people with paralysis. The network involves multiple convolutional layers that captures the spatial and spectral information from the EEG signal. When tested on publicly-available Benchmark and BETA datasets, it reached ITRs of 265.23 bits/min on the Benchmark dataset and 196.59 bits/min on the BETA dataset, both achieved with just 0.4 seconds of stimulation. • [ Liu et al.(2020)Liu, Hu,