Decoding Covert Speech from EEG Using a Functional Areas Spatio-Temporal Transformer

Muyun Jiang, Yi Ding, Wei Zhang, Kok Ann Colin Teo, LaiGuan Fong, Shuailei Zhang, Zhiwei Guo, Chenyu Liu, Raghavan Bhuvanakantham, Wei Khang Jeremy Sim, Chuan Huat Vince Foo, Rong Hui Jonathan Chua, Parasuraman Padmanabhan, Victoria Leong, Jia Lu, Balázs Gulyás, Cuntai Guan · 2025

Read Decoding Covert Speech from EEG Using a Functional Areas Spatio-Temporal Transformer by Muyun Jiang, Yi Ding, Wei Zhang, Kok Ann Colin Teo, LaiGuan Fong, Shuailei Zhang, Zhiwei Guo, Chenyu Liu, Raghavan Bhuvanakantham, Wei Khang Jeremy Sim, Chuan Huat Vince Foo, Rong Hui Jonathan Chua, Parasuraman Padmanabhan, Victoria Leong, Jia Lu, Balázs Gulyás, Cuntai Guan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03762.txt.

Introduction

Muyun Jiang and a large team build a system to decode imagined speech from brain wave recordings, in a 2025 paper. Imagined speech has no sound or movement, so decoding it is hard. Brain signals are noisy, and how the brain maps imagined sounds is not well understood.

They gather a large data set from fifty-seven right-handed native English speakers. Each person repeated the same word in five utterances, both imagined and spoken, within ten seconds. Their transformer model treats brain signals as tokens and uses the layout of functional brain areas over space and time. The work aims toward silent communication aids.

Catalog record

CREATORMuyun Jiang, Yi Ding, Wei Zhang, Kok Ann Colin Teo, LaiGuan Fong, Shuailei Zhang, Zhiwei Guo, Chenyu Liu, Raghavan Bhuvanakantham, Wei Khang Jeremy Sim, Chuan Huat Vince Foo, Rong Hui Jonathan Chua, Parasuraman Padmanabhan, Victoria Leong, Jia Lu, Balázs Gulyás, Cuntai Guan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03762

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decoding Covert Speech from EEG Using a Functional Areas Spatio-Temporal Transformer Muyun Jiang, Yi Ding, Wei Zhang, Kok Ann Colin Teo, LaiGuan Fong, Shuailei Zhang, Zhiwei Guo, Chenyu Liu, Raghavan Bhuvanakantham, Wei Khang Jeremy Sim, Chuan Huat Vince Foo, Rong Hui Jonathan Chua, Parasuraman Padmanabhan, Victoria Leong, Jia Lu, Balázs Gulyás, Cuntai Guan Abstract Covert speech involves imagining speaking without audible sound or any movements. Decoding covert speech from electroencephalogram (EEG) is challenging due to a limited understanding of neural pronunciation mapping and the low signal-to-noise ratio of the signal. In this study, we developed a large-scale multi-utterance speech EEG dataset from 57 right-handed native English-speaking subjects, each performing covert and overt speech tasks by repeating the same word in five utterances within a ten-second duration. Given the spatio-temporal nature of the neural activation process during speech pronunciation, we developed a Functional Areas Spatio-temporal Transformer (FAST), an effective framework for converting EEG signals into tokens and utilizing transformer architecture for sequence encoding. Our results reveal distinct and interpretable speech neural features by the visualization of FAST-generated activation maps across frontal and temporal brain regions with each word being covertly spoken, providing new insights into the discriminative features of the neural representation of covert speech. This is the first report of such a study, which provides interpretable evidence for speech decoding from EEG. The code for this work has been made public at https://github.com/Jiang-Muyun/FAST I Introduction Covert speech is the process of internally articulating speech units, such as words or sentences, without producing audible sounds or movements [ 1 ] . This method is particularly beneficial for individuals with verbal communication impairments due to conditions like stroke, trauma, and amyotrophic lateral sclerosis (ALS), which affect speech production and comprehension [ 2 , 3 ] . This task also helps provide insights into how the brain forms speech patterns and prepares them for verbal articulation, even when no physical speech occurs. Decoding covert speech has made tremendous progress in using invasive recordings [ 4 , 5 ] . Decoding from EEG has gained recognition as a method for assisting individuals with severe communication barriers to establish a channel for thought-based interaction [ 6 ] . However, the task of decoding covert speech from non-invasive neural recordings such as EEG remains challenging. Currently, most covert speech BCIs are based on invasive methods, their feasibility has been demonstrated through various studies. This includes utilizing electrocorticography (ECoG) data to synthesize speech using densely connected 3D CNN as shown in [ 4 ] . Moreover, [ 7 ] demonstrated that the finer details captured by these recordings play a crucial role in understanding and interpreting speech-related neural activity. Additionally, [ 8 ] has shown that imagined speech could be decoded from low and cross-frequency features in intracranial electroencephalography (iEEG) signals. Non-invasive technologies have gradually garnered attention for their potential in BCIs. Notably, the use of Magnetoencephalography, as explored in [ 9 ] , has successfully decoded imagined and spoken phrases. Despite these advancements, studies based on EEG remain scarce due to the challenge of small signal magnitudes, which hinder the ability of decoding methods to learn effective features. In recent years, with the incorporation of deep learning technologies, particularly CNNs like DeepConvNet [ 10 ] and EEGNet [ 11 ] , the capability to accurately interpret EEG data has markedly improved in the tasks of motor imagery, emotional cognition, and more [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] . Building on the success of the transformer model in the fields of computer vision and natural language processing, researchers have also attempted to adopt the transformer structure for the identification and classification of brain activities, leveraging their ability to effectively model long sequences [ 21 , 22 , 23 , 24 ] . However, challenges persist in effectively integrating transformer-based architectures with EEG data. Issues such as high computational demand, the need for large datasets to train effectively, and the sensitivity to the highly variable nature of EEG signals complicate the application. From the existing literature, numerous studies have focused on EEG classification tasks using pure convolutions or convolutional layers combined with transformer architecture approaches. However, a comprehensive insight into the limiting factors of these approaches reveals a common set of challenges: 1. Lack of a high-quality large-scale, multi-utterance, EEG-based covert speech dataset. 2. Lack of effective decoding algorithm to tackle challenging covert speech EEG signals. 3. Lack of understanding of the covert speech intrinsic information in EEG to enable speech decoding. Addressing the challenges outlined, we have collected a multi-utterance dataset aimed at exploring the brain’s mechanisms during covert speech activities. This dataset is collected from 57 right-handed adult males who engaged in both covert and overt speech tasks. Each participant contributed 1000 utterances, allowing for a detailed analysis of speech processing and brain function. We developed FAST, a Functional Areas Spatio-temporal Transformer model that integrates a convolutional neural network with a transformer architecture. The model begins with a series of brain functional area convolutional tokenizers designed to extract spatially sensitive features from the data. Tokens from each brain region are first processed through a spatial projection layer to integrate information across different brain regions. These aggregated features then undergo deeper analysis through multiple transformer layers, enhancing the model’s ability to decode and interpret complex brain signals. Detailed feature visualization reveals distinct and interpretable speech-neural patterns in the frontal and temporal brain regions for covertly spoken words, providing new insights into the discriminative features underlying the neural representation of covert speech. Our main contributions can be summarized as follows: 1. Proposed FAST, a novel Spatio-temporal Transformer Network inspired by cerebral functional systems for covert speech EEG modeling. 2. Provided in-depth feature analysis, revealed discriminative features during covert speech. 3. We evaluate the model on a large-scale multi-utterance covert and overt speech dataset from 57 subjects totaling 1000 utterances per participant. To the best of the authors’ knowledge, this is the first study in the literature showing features learned from EEG-based covert speech BCI approaches. This model merges neuroscience-driven feature extraction with a multimodal dataset and a flexible learning strategy. The structure of this work is outlined as follows: Section II reviews prior research closely related to our study. Section III elaborates on the detailed methodology of our model. Section IV focuses on the procedures for data collection. Section V expressed the training scheme of FAST. Section VI outlines the computational experimental setup and the evaluation of our novel method as well as our insights derived from in-depth feature analysis. Finally, Section VII concludes the study and proposes directions for future research. II Related Work II-A Neural Foundations and Decoding Advancements of Covert Speech Recent studies provide neural evidence that covert speech involves an articulatory component [ 25 ] . Activation in Broca’s area, associated with speech production, and supplementary motor areas underscores its link to articulatory processes, while engagement of auditory imagery networks highlights its reliance on both auditory and motor functions [ 26 , 27 ] . Research has identified distinct neural signatures for covert speech compared to listening to speech, with overlapping yet specialized activation in key brain regions [ 28 ]

Read the complete text