Teaching Wav2Vec2 the Language of the Brain
Tobias Fiedler, Leon Hermann, Florian Müller, Sarel Cohen, Peter Chin, Tobias Friedrich, Eilon Vaadia, Hasso Plattner Institute, Digital Engineering Faculty, University of Potsdam, {tobias.fiedler,leon.hermann,florian.mueller}@student.hpi.de, [email protected] · 2025
Read Teaching Wav2Vec2 the Language of the Brain by Tobias Fiedler, Leon Hermann, Florian Müller, Sarel Cohen, Peter Chin, Tobias Friedrich, Eilon Vaadia, Hasso Plattner Institute, Digital Engineering Faculty, University of Potsdam, {tobias.fiedler,leon.hermann,florian.mueller}@student.hpi.de, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09459.txt.
Introduction
Speech decoded from brain activity could restore voice for paralyzed people, but brain data sets are small. By contrast, speech models trained on audio are large and ready-made. Tobias Fiedler and co-authors asked in 2025 whether skills learned by one such model, Wav2Vec2, carry over to brain signals.
They swapped the model's audio front end for a fresh brain-signal front end and tried forty-five designs. Three training routes were compared: full tuning with pre-trained weights, training from scratch, and tuning only the new front end. Full tuning won, with a character error rate near nineteen percent, far below both rivals. Knowledge from audio thus does lift brain decoding.
Catalog record
| CREATOR | Tobias Fiedler, Leon Hermann, Florian Müller, Sarel Cohen, Peter Chin, Tobias Friedrich, Eilon Vaadia, Hasso Plattner Institute, Digital Engineering Faculty, University of Potsdam, {tobias.fiedler,leon.hermann,florian.mueller}@student.hpi.de, [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teaching Wav2Vec2 the Language of the Brain Tobias Fiedler, Leon Hermann, Florian Müller, Sarel Cohen, Peter Chin, Tobias Friedrich, Eilon Vaadia, Hasso Plattner Institute, Digital Engineering Faculty, University of Potsdam, {tobias.fiedler,leon.hermann,florian.mueller}@student.hpi.de, [email protected] Abstract The decoding of continuously spoken speech from neuronal activity has the potential to become an important clinical solution for paralyzed patients. Deep Learning Brain Computer Interfaces (BCIs) have recently successfully mapped neuronal activity to text contents in subjects who attempted to formulate speech. However, only small BCI datasets are available. In contrast, labeled data and pre-trained models for the closely related task of speech recognition from audio are widely available. One such model is Wav2Vec2 which has been trained in a self-supervised fashion to create meaningful representations of speech audio data. In this study, we show that patterns learned by Wav2Vec2 are transferable to brain data. Specifically, we replace its audio feature extractor with an untrained Brain Feature Extractor (BFE) model. We then execute full fine-tuning with pre-trained weights for Wav2Vec2, training “from scratch” without pre-trained weights as well as freezing a pre-trained Wav2Vec2 and training only the BFE each for 45 different BFE architectures. Across these experiments, the best run is from full fine-tuning with pre-trained weights, achieving a Character Error Rate (CER) of 18.54%, outperforming the best training from scratch run by 20.46 and that of frozen Wav2Vec2 training by 15.92 percentage points. These results indicate that knowledge transfer from audio speech recognition to brain decoding is possible and significantly improves brain decoding performance for the same architectures. Related source code is available at https://github.com/tfiedlerdev/Wav2Vec2ForBrain . I Introduction As a social being, the ability to communicate is essential to the mental health of human individuals. However, some people have lost theirs e.g. through a brainstem stroke or ALS, limiting their ability to participate socially. The new developments of brain-computer interfaces (BCIs) take the challenge to help these patients. The method involves recordings of neuronal spiking activity via hundreds of micro-electrodes implanted directly in areas of interest in the participant’s brain. By recording this neuronal activity as the participant attempts to speak a given sentence, a dataset of brain data and corresponding attempted speech content is created. The computational challenge is to map the neuronal activity time series data to the corresponding text, essentially discovering patterns within the brain data and mapping them to attempted spoken characters, words, or phonemes, to produce a plain English transcription from unseen brain activity. What makes this task especially challenging is the small amount of available data. In contrast, data and pre-trained models for the closely related task of audible speech recognition are widely available. One such model is Wav2Vec2 which has been pre-trained in a self-supervised fashion to create meaningful representations of speech audio data and finetuned to transcribe speech based on these representations. In this study, we show that patterns learned by Wav2Vec2 for the task of speech recognition are transferable to the task of neuronal spiking activity based on attempted speech decoding. Specifically, we replace the audio feature extractor of Wav2Vec2 with an untrained Brain Feature Extractor (BFE) model. We then execute ”Full Fine-Tuning” with pre-trained weights for Wav2Vec2, ”Training from Scratch” without pre-trained weights as well as ”Frozen Wav2Vec2” training with pre-trained and locked weights for Wav2Vec2 each for 45 different BFE architectures. Across all experiments, the best run is from Full Fine-Tuning with pre-trained weights, achieving a Word Error Rate (WER) of 30.97% and a Character Error Rate (CER) of 18.54% without Language Model (LM) assisted decoding, while the best runs of Training from Scratch and Frozen Wav2Vec2 training achieve significantly worse WERs of 66.21% (39.01% CER) and 53.29% (32.46% CER) respectively. These results show that knowledge transfer from audio speech recognition to neuronal activity decoding works. Additionally, the Frozen Wav2Vec2 training runs having a CER better than a random predictor show that neuronal spiking data can be transformed into a representation understood by models trained on different input domains such as audio signals. II Related Work Recently, there have been major advances in the field of BCIs. This report focuses on related work on decoding spoken sentences from brain activity. This project was initially set up to participate in a deep learning challenge called brain-to-text published by Willett et al. [ 1 ] . As such, the model architectures explained in this paper were trained on the dataset they collected. Willet et al. implemented a GRU [ 2 ] model architecture to predict spoken phonemes from brain activity and added a language model in post processing with a resulting WER of 23.8% on a 125,000-word vocabulary. However, our experiments are executed on a lower training sample count since the provided dataset did not supply transcriptions of data samples in the competition holdout set. Thus, we created a test set from the training samples, reducing the size of the training partition. Benster et al. [ 3 ] developed a multimodal architecture that can be applied to arbitrary pairs of speech modalities. On the brain-to-text dataset, they were able to improve the previous best contribution from 9.8% WER to 8.9%. Similarly to Willet et al. experiments, they used the complete training set without partitioning it further. At the point of submission of the study, we are aware of one competitor of the brain-to-text challenge which achieves a WER of 5.81%. In this work, we explored the ability of the state-of-the-art speech transcription transformer [ 4 ] model ”Wav2Vec2” [ 5 ] to apply its learned patterns to the given task of converting brain recordings to text. To our best knowledge, there are no related works that attempted to apply a pre-trained speech transcription transformer like Wav2Vec2 to the input domain of brain recordings. However, Wu et al. [ 6 ] have successfully adapted the pre-trained LLM called ”Llama” to audio signals, roughly resembling our approach on different domains. III Methods III-A Dataset We used the brain-to-text dataset [ 1 ] as stated in the related work section. The creators collected the data for five days of vocalized speech and three days of silent speech. In total, 10,850 sentences were recorded. The brain activity of a singular patient was measured with four microelectrode arrays. Two of these were implanted in the ventral premotor cortex and the remaining two were in area 44, which is part of Broca’s area. The work of the brain-to-text researchers suggested that area 44 contained little information for speech classification so we also excluded that data from training. The data samples were recorded in blocks. Each day has a fixed amount of blocks. These blocks have already been split by the Brain-to-Text team into a training, test, and competition holdout partition. This project uses the given test set as a validation set during training. Since the targets of the competition holdout set are not supplied, we chose to create our test set by taking the first block of each day of the train set as test data. With this, we decrease the data available for training from 8,800 recorded sentences to 7,920 and therefore take a worse model performance into account. Each block contains a fixed amount of sentences and the corresponding brain activity recordings. The recorded microelectrode data is already available in