A Silent Speech Decoding System from EEG and EMG with Heterogenous Electrode Configurations
Masakazu, Motoshige, Kenichi, Nathania, Eri, Kai, Ilya, Shuntaro · 2025
Read A Silent Speech Decoding System from EEG and EMG with Heterogenous Electrode Configurations by Masakazu, Motoshige, Kenichi, Nathania, Eri, Kai, Ilya, Shuntaro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13835.txt.
Introduction
Masakazu Inoue and seven co-authors present a silent speech decoding system in two thousand twenty-five. Silent speech reads unspoken words from brain and muscle signals. Data are hard to gather, and setups vary from lab to lab. Mixed electrode layouts thus block one big clean data set.
Their networks take in signals from varied electrode setups as they come. Multi task training on large pooled sets drives the learning. Word accuracy hit high marks for healthy users in tests. A speech impaired patient scored far past single subject models too. Cross language calibration also gained from the pooled route.
Catalog record
| CREATOR | Masakazu, Motoshige, Kenichi, Nathania, Eri, Kai, Ilya, Shuntaro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13835
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Silent Speech Decoding System from EEG and EMG with Heterogenous Electrode Configurations Masakazu, Motoshige, Kenichi, Nathania, Eri, Kai, Ilya, Shuntaro Abstract Silent speech decoding, which performs unvocalized human speech recognition from electroencephalography/electromyography (EEG/EMG), increases accessibility for speech-impaired humans. However, data collection is difficult and performed using varying experimental setups, making it nontrivial to collect a large, homogeneous dataset. In this study we introduce neural networks that can handle EEG/EMG with heterogeneous electrode placements and show strong performance in silent speech decoding via multi-task training on large-scale EEG/EMG datasets. We achieve improved word classification accuracy in both healthy participants (95.3%), and a speech-impaired patient (54.5%), substantially outperforming models trained on single-subject data (70.1% and 13.2%). Moreover, our models also show gains in cross-language calibration performance. This increase in accuracy suggests the feasibility of developing practical silent speech decoding systems, particularly for speech-impaired patients. \interspeechcameraready Inoue Sato Tomeoka Nah Hatakeyama Arulkumaran Horiguchi Sasai nocounter]Araya Inc.Japan Accepted for presentation at Interspeech 2025 keywords silent speech recognition, transfer learning, brain-machine interface 1 Introduction Silent speech decoding from electroencephalography (EEG) and electromyography (EMG) signals offers a promising communication solution for individuals with conditions like amyotrophic lateral sclerosis (ALS) or post-laryngectomy status. While invasive brain activity measurements have achieved remarkable decoding accuracy [ 1 , 2 , 3 , 4 , 5 ] , their widespread adoption is hindered by the requirement for cranial surgery. Conversely, non-invasive methods such as EEG and EMG are more accessible alternatives. Whilst there is considerable research on EEG-based speech decoding, these approaches have not yet achieved practical accuracy levels due to lower signal-to-noise ratios and spatial resolution, as compared to invasive methods [ 6 , 7 , 8 , 9 ] . For individuals with sufficient muscle control, EMG-based silent speech decoding has demonstrated adequate accuracy for silent speech command input and conversation, using subject-specific trained models [ 10 , 11 , 12 , 13 ] . Considering that patients with neurodegenerative diseases may retain some degree of muscle control depending on their disease progression, it is practical to explore models that utilize EEG and EMG signals to achieve optimal accuracy. However, there exists a lack of standardized equipment, as there is a variety of EEG headsets available with differing numbers of electrodes and variable placements, and EMG electrodes must be placed manually. While achieving high accuracy in deep learning is known to require substantial amounts of data [ 14 , 15 , 16 ] , building large EEG datasets specifically designed for a particular task is challenging due to these variations in electrode placement protocols and measurement devices. Therefore, developing architectures capable of handling diverse electrode arrangements is crucial for building generalizable EEG/EMG encoders. Recent studies have made significant progress in developing generalizable decoders by using EEG data from diverse electrode configurations to train deep neural networks (DNNs) through self-supervised learning approaches, such as signal reconstruction [ 17 , 18 , 19 ] . However, the variability in shapes of these latents due to different electrode configurations remains, making it challenging to train a single model on diverse electrode configuration data in an end-to-end manner for downstream tasks. This has been addressed by incorporating average pooling [ 20 , 21 ] or implementing subject/electrode-specific layers [ 22 , 23 ] to convert the latents into predefined sizes. While these studies demonstrated the utility of handling heterogeneous electrode configurations within a single model to increase available data for tasks such as motion/imagined motion detection, the effectiveness for relatively complex tasks such as silent speech decoding remains unclear, as does the efficacy of knowledge transfer between healthy individuals and patients. Therefore, in this work we investigate four types of EEG/EMG tokenizers, including a novel on-the-fly kernel, that enable our DNN models to handle data from heterogeneous electrode configurations. We evaluate how training models with these tokenizers on EEG/EMG data collected from eight healthy participants and a patient with a neurodegenerative disease during silent speech, along with additional pretraining on large data with heterogeneous electrode configurations and tasks, would improve silent speech decoding performance. Our results demonstrate higher word classification accuracy compared to models trained exclusively on individual data. Furthermore, we validate calibration performance on silent speech data for novel subjects, sessions from different days, and on a separate EMG dataset in a different language [ 24 ] , demonstrating the value of pretraining on large datasets with heterogeneous tasks and electrode placements. Figure 1 : Our model architecture. To handle heterogeneous electrode configurations, the tokenizer can be selected by inserting with one of the following modules: (a) global average pooling, (b) electrode-specific, (c) subject-specific, or (d) on-the-fly kernel. 2 Materials and methods 2.1 Data acquisition In previous work, [ 16 ] collected a large EEG and EMG dataset recorded while participants spoke, using three types of EEG recording devices: eego sports 1 1 1 https://www.ant-neuro.com/products/eego-sports , g.Pangolin 2 2 2 https://www.gtec.at/product/g-pangolin-electrodes/ and g.SCARABEO 3 3 3 https://www.gtec.at/product/g-scarabeo-eeg-electrodes/ . Using a similar setup, we collected EEG and EMG data with the eego sports device from eight healthy participants during silent speech and vocalized speech, and one patient during silent speech. Due to a progressive neuromuscular disease, the patient is unable to perform sufficient muscular movements for speech and cannot produce any vocalization due to dependence on a ventilator. For healthy participants, based on electrode positioning in [ 24 ] , we recorded the EMG from the cheek, upper mouth, lower mouth, chin, and throat. For the patient, the latter EMG electrode pair was located below and above the eye instead of the throat because the electrode could not be attached to his throat due to the presence of medical equipment. Participants were instructed to read text presented on a screen according to timing cues. Each healthy participant performed 20 sessions of word utterances and 2 sessions of sentence utterances in both vocalized and silent conditions. The patient also performed 20 sessions of silent speech word utterances, though some sessions were concluded prematurely due to fatigue. The vocabulary size for the word reading task was 64 words, selected by the patient based on daily communication needs, and sentence reading sessions consisted of 153 sentences. All participants are native Japanese speakers and spoke Japanese in our experiments. As such, to measure phonetic transcription accuracy, we use moras, which are Japanese phonetic units of speech [ 25 ] . Table 1 : Pretraining datasets. For tasks, V+SS indicates paired vocalized/silent speech data, and VS indicates solely vocalized speech. The dataset “word” refers to subject-specific splits, and “word all” refers to the aggregation across all subjects. Task EEG EMG Sampling Rate Hours Vocab Size Confounds Dataset Device Channels Channels word/word all eego all all VS [ 16 ] g.Pangolin 128 ch 3 ch 1024\text{\,}\mathrm{Hz} 49.1 open audio ✓ VS [ 16 ] g.Pangolin 128 ch 3 ch 1024\text{\,}\mathrm{Hz} 64.2 open audio, text ✓ VS [ 16 ] g.SCARABEO 62 ch 3 ch 1024\text{\,}\mathrm{Hz} 27.7 open audio ✓ V+SS (word) eego sports 63 ch 5 ch ∗ 2048\text{\,}\mathrm{Hz} 16.1 64 audio, text, timing, word ✓ ✓ ✓ V+SS (sentence) eego sports 63 ch 5 ch ∗ 2048\text{\,}\mathrm{Hz} 13.6 267 audio, text, timing ✓ ✓ VS eego sports 63 ch 3 ch 2048\text{\,}\mathrm{Hz} 49.2 open audio ✓