Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding
Ji-Ha Park, Soowon Kim, Seo-Hyun Lee, Seong-Whan Lee · 2025
Read Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding by Ji-Ha Park, Soowon Kim, Seo-Hyun Lee, Seong-Whan Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14790.txt.
Introduction
Ji-Ha Park and three co-authors work toward a speech prosthesis that reads the brain in a 2025 paper. People who cannot speak might one day communicate through devices that decode their brain signals. Past systems mostly decode short intentions or broken fragments, not flowing speech.
The team trained people on grouped speech sounds, called visemes, each tied to a shape of the lips. A diffusion model then read non-invasive brain signals and rebuilt the matching lip movements, for single trials and for whole sentences. The rebuilt faces carried coherent visual speech. The authors present viseme decoding as a step toward live neural communication and speech aids for patients.
Catalog record
| CREATOR | Ji-Ha Park, Soowon Kim, Seo-Hyun Lee, Seong-Whan Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14790
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding Ji-Ha Park, Soowon Kim, Seo-Hyun Lee, Seong-Whan Lee Abstract Decoding text, speech, or images from human neural signals holds promising potential both as neuroprosthesis for patients and as innovative communication tools for general users. Although neural signals contain various information on speech intentions, movements, and phonetic details, generating informative outputs from them remains challenging, with mostly focusing on decoding short intentions or producing fragmented outputs. In this study, we developed a diffusion model-based framework to decode visual speech intentions from speech-related non-invasive brain signals, to facilitate face-to-face neural communication. We designed an experiment to consolidate various phonemes to train visemes of each phoneme, aiming to learn the representation of corresponding lip formations from neural signals. By decoding visemes from both isolated trials and continuous sentences, we successfully reconstructed coherent lip movements, effectively bridging the gap between brain signals and dynamic visual interfaces. The results highlight the potential of viseme decoding and talking face reconstruction from human neural signals, marking a significant step toward dynamic neural communication systems and speech neuroprosthesis for patients. I INTRODUCTION Brain-computer interface (BCI) is a technology that interprets and conveys the human mind by translating brain signals that encapsulate various aspects of user intention and cognition [ 1 , 2 , 3 , 4 , 5 ] . Recent advances in signal processing and generative technologies are being integrated with BCI, giving rise to a novel form of intuitive neural communication represented by speech BCI [ 6 ] . This technology aims to decode speech-related neural signals and directly generate text or speech [ 7 , 8 ] . Speech BCIs are an important and rapidly emerging field with significant potential, not only as assistive technologies for individuals with speech impairments, such as those with aphasia but also as a groundbreaking mode of neural communication for general users [ 9 , 10 , 11 , 2 ] . Fig. 1: Overview of the proposed viseme decoding framework from EEG signals. The recorded EEG signals are preprocessed in two ways for training. Firstly, sentence-level EEG signals are segmented at the phoneme-level based on real audio of uttered sentences to extract phoneme-based viseme feature embeddings. Secondly, single-trial EEG data of viseme classes are preprocessed and given as input features. DDPM in the proposed framework learns relevant features by injecting noise and restoring the data, while the encoder and decoder work together to compensate for the information loss during this process, allowing the accurate reconstruction of EEG signals. The trained model then predicts the viseme classes by passing the compressed information from the final encoder stage through a classifier. Finally, the predicted viseme classes are sequentially reconstructed into articulatory movements of sentences. At the same time, advancements in computer vision and speech processing methods have significantly enhanced the generation of realistic talking faces that synchronize with users’ speech and facial expressions, providing immersive and interactive visual experiences [ 12 , 13 , 14 , 15 , 16 ] . The convergence of these technologies with BCI may open new avenues of exploring how visual speech intentions, such as facial movements and expressions, can be decoded directly from brain signals and transformed into dynamic visual outputs [ 17 , 18 , 19 ] . This opens up the possibility of expressive neural communication for general users, and significant speech neuroprostheses for language-impaired patients that can provide speech-visual feedback to facilitate rehabilitation [ 10 , 9 ] . Despite these promising potentials, most studies have focused on generating fragmented or abstract outputs from neural signals, as challenges remain in translating these signals into dynamic and unconstrained forms [ 20 ] . Especially, extracting relevant features to reconstruct complex lip movements or subtle facial expressions from noisy neural signals poses a significant challenge with great potential for advancement. While signal processing techniques using invasive measurements have demonstrated their potential for robust decoding of speech intentions, their reliance on surgical procedures limits their application [ 10 , 19 ] . Non-invasive methods such as electroencephalography (EEG) remain underexplored due to their low signal-to-noise ratio (SNR), despite offering significantly broader applicability and potential benefits [ 21 , 22 , 23 ] . Therefore, developing techniques that can effectively process non-invasive neural signals despite low SNR is crucial for adaptable neural communication and speech neuroprosthesis [ 17 ] . To achieve this, a novel approach capable of robustly capturing subtle information is necessary to enhance both the fidelity and the ability to generate unconstrained visual representations from non-invasive brain signals. Fig. 2: Experimental paradigm for EEG signals recording. The phonemes and sentences were presented randomly to guide overt, mimed, and imagined speech. Participants were instructed to perform the experiment with a focus on the changes in lip shapes and articulator movements. In this paper, we introduce a diffusion model-based viseme decoding framework that effectively learns visual speech intentions from speech-related EEG signals. The viseme-level decoding demonstrates the potential to generate dynamic outputs from noisy and limited non-invasive neural signals, advancing beyond conventional word- or sentence-level decoding, and offering potential applications for patients with progressive paralysis. Our generative model-based framework goes beyond predefined class decoding, enabling robust decoding through phoneme-level clustering while extending to unconstrained reconstruction by learning small fragments of speech. II METHODS II-A Model Architectures II-A 1 Denoising Diffusion Probabilistic Models The overall structure of the model consists of denoising diffusion probabilistic models (DDPMs), a conditional autoencoder (CAE), and a classifier (Fig. 1 ) [ 24 ] . DDPMs are based on a time-conditional U-Net architecture, designed to learn how to represent EEG data by progressively adding noise in a stepwise manner [ 25 ] . Unlike the conventional technique of training a model with a noisy \mathbf{x}_{t} and predicting the noise it contains by training network \epsilon\theta(\mathbf{x}_{t},t) , our model is trained to reach the original signal \mathbf{x}_{0} : \mathcal{L}_{\text{DDPM}}(\theta)=||\mathbf{x}_{0}-\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t},t)|| (1) II-A 2 Conditional Autoencoder The information loss of the DDPM is identified and corrected by the CAE [ 26 ] . The CAE is composed of an encoder and decoder, denoted as \mathcal{E}_{\phi} and \mathcal{D}_{\psi} , respectively (Fig. 1 ). Instead of connecting to the output of \mathcal{E}_{\phi} , \mathcal{D}_{\psi} is skip-connected with the DDPM layers, enabling \mathcal{D}_{\psi} to be indirectly influenced by the corruption stage of the DDPM. Additionally, to improve the reconstruction of \mathcal{L}_{\text{DDPM}} , the original signal \mathbf{x}_{0} and the DDPM output \hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t},t) are skip-connected to the final layer of \mathcal{D}_{\psi} . To ensure that \mathcal{E}_{\phi} effectively derives meaningful representations related to visemes, channel attentions are applied after each layer block of \mathcal{E}_{\phi} . This structure learns the importance of each channel in the input features, calculates channel weights, and multiplies them with the original input to generate a weighted output. By doing so, the CAE can generate more accurate representations of the original signals. \mathcal{L}_{\text{CAE}}(\psi,\phi)=||\mathcal{L}_{\text{DDPM}}(\theta)-\mathcal{D}_{\psi}(\mathcal{E}_{\phi}(\mathbf{x}_{0}),\hat{\mathbf{x}}_{\theta}(\mathbf{x}_{t},t))|| (2) II-A 3 Viseme Decoding Process The output from \mathcal{E}_{\phi} is compressed into a one-dimensional representation \mathbf{z} through an adaptive average pooling layer. This latent vector is then passed to the linear classifier \mathcal{C}_{\rho} . \mathcal{C}_{\rho} , is composed of Kolmogorov–Arnold Networks (KAN), which incorporate linear layers and group normalization [ 27 ] . \mathcal{C}_{\rho} is co-trained with the CAE to distinguish