Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings
Owais Mujtaba, Pablo Rodríguez San, Zubair Ahmad, Marc, Jose A · 2025
Read Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings by Owais Mujtaba, Pablo Rodríguez San, Zubair Ahmad, Marc, Jose A (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14074.txt.
Introduction
Owais Mujtaba Khanday and co-authors ask in 2025 whether modern language and speech models can rebuild brain activity recorded while people speak. They focus on high-gamma signals, a fast brain rhythm tied to cortex activity during speech. Their data come from recordings made as study participants produced speech.
The team took compact summaries, called embeddings, from large pre-trained models of text and sound. They then mapped those summaries onto the recorded brain signals and checked how close the rebuilt signals came. Scores were high for every participant, with match values from 0.79 to 0.99. The results suggest these model summaries keep much of the pattern and timing of real speech-related brain activity.
Catalog record
| CREATOR | Owais Mujtaba, Pablo Rodríguez San, Zubair Ahmad, Marc, Jose A |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings Owais Mujtaba, Pablo Rodríguez San, Zubair Ahmad, Marc, Jose A Abstract Understanding how neural activity encodes speech and language production is a fundamental challenge in neuroscience and artificial intelligence. This study investigates whether embeddings from large-scale, self-supervised language and speech models can effectively reconstruct high-gamma neural activity characteristics, key indicators of cortical processing, recorded during speech production. We leverage pre-trained embeddings from deep learning models trained on linguistic and acoustic data to represent high-level speech features and map them onto these high-gamma signals. We analyze the extent to which these embeddings preserve the spatio-temporal dynamics of brain activity. Reconstructed neural signals are evaluated against high-gamma ground-truth activity using correlation metrics and signal reconstruction quality assessments. The results indicate that high-gamma activity can be effectively reconstructed using large language and speech model embeddings in all study participants, generating Pearson’s correlation coefficients ranging from 0.79 to 0.99. Khanday Esteban Lone Ouellet Gonzalez-Lopez University of Granada, Granada, 18071Spain University of Granada, Granada, 18071Spain Andhra Pradesh, 530045India keywords Computational neuroscience, large language model, Wav2Vec2, word embedding, brain encoding, sEEG. † † email: {owaismujtaba, prodriguez, moullet, joseangel}@ugr.es, [email protected] 1 Introduction Speech neuroprostheses, also known as speech brain-computer interfaces (BCI), are a class of assistive technology aimed at restoring or increasing communication abilities for individuals with severe speech and motor impairments. These devices decode neural activity associated with speech production and translate it into text [ 1 , 2 , 3 , 4 , 5 , 6 ] , synthesized speech [ 7 , 8 , 9 , 10 , 11 , 12 ] , or other forms of communication, such as control parameters for 3D-model facial animation [ 13 ] . The potential applications of these devices are numerous [ 14 ] , including the promise to restore communication to individuals with conditions such as amyotrophic lateral sclerosis (ALS) [ 15 , 4 , 11 , 6 , 16 ] or other severe speech disorders caused by brain damage (e.g. brain-stem stroke) [ 2 , 3 ] . During the past decade, this area has experienced significant growth, driven by advances in neuroscience, deep learning, and neural interface technologies. Recent studies have demonstrated significant progress in decoding speech from neural signals, with applications ranging from text generation to real-time speech synthesis (see [ 17 , 18 , 19 ] for recent reviews on the topic). This interdisciplinary field, which bridges neuroscience and speech and language technology, has garnered increasing interest from both the scientific community and the public, as it holds the promise of restoring one of the most fundamental human abilities: our capacity to communicate. Most prior research on speech neuroprostheses has relied on invasive neural recording techniques, such as electrocorticography (ECoG) or microelectrode arrays, due to their superior signal quality compared to non-invasive methods like electroencephalography (EEG). Among these, ECoG, which involves placing electrodes directly on the brain’s surface, has been particularly effective in capturing high-resolution neural activity related to speech production [ 1 , 7 , 20 ] . However, these methods are limited in their ability to access deeper brain structures, which may also play a critical role in speech and language processing. In this study, we investigate the potential of stereo EEG (sEEG), a less commonly used but highly promising sensing technology for speech neuroprostheses [ 21 ] . Unlike ECoG, sEEG employs depth electrodes that can record neural activity from both cortical and subcortical regions, offering a more comprehensive view of the brain’s speech network. Although sEEG has primarily been utilized in clinical settings for epilepsy monitoring, its potential for BCI applications, particularly in speech decoding, remains underexplored. This work aims to contribute to the limited but growing body of research on sEEG’s applicability in speech neuroprostheses [ 10 , 22 , 23 , 12 ] . In parallel to the above studies, recent neuroscience research has revealed a striking similarity between neural activity in the human brain during language-related tasks and the internal states of large-scale, self-supervised speech and language models. Models such as Wav2Vec 2.0 [ 24 ] and GPT-2 [ 25 ] have demonstrated a close alignment with brain activity during both speech perception and production, indicating that they encode linguistically relevant information in ways that parallel human neural processing [ 26 , 27 , 28 ] . These findings suggest that these models may serve as powerful tools for advancing neuroscience research, offering new insights into the neural basis of speech and language, as well as for developing more effective speech neuroprostheses. In this work, we aim to bridge these two lines of research by exploring whether embeddings derived from pre-trained language and speech models can effectively reconstruct neural activity captured during speech production using sEEG. To achieve this, we utilize embeddings from large deep-learning models trained on text and audio to represent high-level linguistic features, which are then mapped onto sEEG signals. We assess how well these embeddings preserve the spatio-temporal dynamics of brain activity by comparing the reconstructed neural signals to ground truth recordings, employing correlation metrics and signal reconstruction quality evaluations. Ultimately, this work aims to provide novel insights into the neural mechanisms underlying speech production and to evaluate the potential of large deep learning models to advance neuroscience research and the development of speech neuroprostheses. 2 Method Figure 1: Methodology overview. Fig. 1 presents a block diagram summarizing our methodology for this work. We used high-gamma features extracted from sEEG signals as the target neural features that were predicted from the embeddings computed by large, self-supervised audio and text models. These recordings were obtained from 10 participants implanted with depth electrodes while reading words aloud. The embeddings computed by the models were mapped onto the high-gamma features using an Elastic Net regression model [ 29 ] , applied per subject. We evaluated the embeddings from three models: FastText [ 30 ] and GPT-2.0 [ 25 ] for textual representations, and Wav2Vec 2.0 [ 24 ] for audio representations. Finally, the reconstructed high-gamma features were compared to the ground-truth features extracted from the sEEG signals using objective metrics to assess reconstruction quality. The following sections describe the analysis performed in more detail. 2.1 Dataset We used a publicly available [ 31 ] dataset comprising simultaneous recordings of audio and sEEG from 10 Dutch participants (5 male and 5 female; mean age: 32 years) diagnosed with pharmacoresistant epilepsy. As part of their clinical treatment, participants were implanted with platinum-iridium sEEG electrode shafts (Microdeep intracerebral electrodes; Dixi Medical, Besançon, France). Each electrode shaft had a diameter of 0.8 mm and contained between 5 and 18 electrode contacts, resulting in an average of 131.7 implanted electrodes per participant (SD = 49.92). The placement of the electrodes was determined solely based on clinical requirements, with a focus on regions such as the superior temporal sulcus, hippocampus, and inferior parietal gyrus. Due to individualized clinical needs, the number and location of electrodes varied between participants, providing sparse but widespread coverage of the cortical and subcortical regions. During the experiment, participants read aloud a list of 100 words from the Dutch IFA corpus [ 32 ] .The words were presented in a randomized order to avoid