Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain

Omer, Mariya · 2025

Read Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain by Omer, Mariya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03832.txt.

Introduction

Omer Moussa and Mariya Toneva study brain-tuned speech models in a 2025 paper on how machines process speech like the brain. Standard speech models learn sound well but stack meaning in the wrong layers versus human brains.

Brain-tuning means fine-tuning a model with human brain recordings from speech tasks. Past work showed this boosts meaning skills. Here, late layers of tuned models align far better with brain meaning areas than plain models. Early layers stay focused on raw sound traits. Late layers also win on hard high-level tasks. The tuned models thus show a clean ladder from sound to meaning, much like human speech processing in the brain.

Catalog record

CREATOROmer, Mariya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03832

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain Omer, Mariya Abstract Pretrained self-supervised speech models excel in speech tasks but do not reflect the hierarchy of human speech processing, as they encode rich semantics in middle layers and poor semantics in late layers. Recent work showed that brain-tuning (fine-tuning models using human brain recordings) improves speech models’ semantic understanding. Here, we examine how well brain-tuned models further reflect the brain’s intermediate stages of speech processing. We find that late layers of brain-tuned models substantially improve over pretrained models in their alignment with semantic language regions. Further layer-wise probing reveals that early layers remain dedicated to low-level acoustic features, while late layers become the best at complex high-level tasks. These findings show that brain-tuned models not only perform better but also exhibit a well-defined hierarchical processing going from acoustic to semantic representations, making them better model organisms for human speech processing. Moussa Toneva nocounter]Max Planck Institute for Software SystemsGermany keywords cognitive models of speech, brain alignment, self-supervised learning, interpretability † † email: {omoussa, mtoneva}@mpi-sws.org 1 Introduction With the rapid development of self-supervised (SSL) language and speech models, research at the intersection of neuroscience, language, and speech understanding has advanced considerably. Current pretrained self-supervised (SSL) speech and language models can predict human brain responses to natural speech with impressive accuracy, outperforming hand-engineered features [ 1 , 2 , 3 , 4 ] . Researchers seek to use language models as model organisms for studying reading and listening in the brain [ 5 ] , aiming to better understand the information processing mechanisms that give rise to brain-like representations. Prior work has also shown that the ability of these models to predict brain responses (i.e., their “brain alignment”) differs substantially across their layers [ 6 , 7 , 8 , 9 , 10 , 2 , 3 ] . For SSL speech models specifically, upper-middle layers align much more with the whole cortex response and the auditory cortex than other model layers [ 2 ] . Some semantic hierarchy is also observed in downstream tasks, where low-level and spectral features are best captured by earlier layers, while phonemic and semantic tasks are best captured by upper-middle layers [ 2 ] . However, these analyses of how semantics evolve through model layers did not test how these models align with the brain’s high-level late language regions, which are thought to be the endpoint of linguistic and speech processing in the brain [ 11 ] . Moreover, SSL models show a downward trend after the upper-middle layers on high-level tasks, suggesting that the late layers (the ones closest to the model output) encode semantics less effectively than the preceding ones. We propose that a model that is more in line with speech processing in the brain would exhibit a better-defined hierarchical processing from acoustic to semantic representations, where early layers excel at low-level tasks and alignment with primary auditory regions while late layers better reflect high-level tasks and alignment with late-language semantic regions. Recent work introduced the framework of brain-tuning (fine-tuning models using brain data as targets) for speech models [ 12 ] . It demonstrated that brain-tuning improves the semantic understanding of speech models, but it did not examine how brain-tuning affects the hierarchy of information processing in the model or how this hierarchy compares to the pretrained version. In this work, we utilize brain-tuning to investigate whether it could lead to models that reflect the speech processing hierarchy in the brain better than their pretrained counterparts. To this end, we build on the methods introduced in [ 12 ] to brain-tune popular speech models and then contrast the layer-wise brain alignment and downstream performance between the original pretrained models and the corresponding brain-tuned models. After testing this approach on two popular model families (Wav2Vec2.0 and HuBERT [ 13 , 14 ] ), we find that: 1) both brain alignment and downstream performance are generally improved in brain-tuned models (consistent with the findings in [ 12 ] ). 2) Pretrained models have a downward trend (i.e., they peak in the middle) when aligning with high-level late language brain areas. This trend is similar to the one reported for the whole cortex and the auditory cortex in [ 2 ] . Hence, upper-middle layers align with late language regions best while late layers perform worse, similarly to middle and early layers. 3) In contrast, brain-tuned models exhibit an upward trend, where late layers predict late language areas best. 4) Similarly to pretrained models, brain-tuned models have a downward trend when aligning with the low-level primary auditory regions. 5) For downstream tasks, phonemic and semantic tasks are best predicted by late layers in the brain-tuned models but by upper-middle layers in the pretrained models. As for low-level speech tasks, early layers perform best in both the pretrained and brain-tuned versions. Based on these findings, we propose that brain-tuned speech models serve as more suitable model organisms for human speech processing, offering both stronger performance and a hierarchy more consistent with human speech processing. 2 Methods 2.1 Speech Language Models We analyze two popular pretrained self-supervised speech language model families: Wav2vec2.0 [ 13 ] and HuBERT [ 14 ] . Both models are transformer-based and rely on a CNN feature extractor to output latent speech representations that are fed to the transformer blocks. We use the base architectures, which were trained on \sim 960 hours of audio in a self-supervised manner, where a portion of the input audio is masked, and the training objective is to predict the correct latent representations of the masked parts. Each model has 12 transformer layers and a fixed embedding dimension of 768. 2.2 Brain fMRI Dataset and Preprocessing We use the largest public fMRI dataset [ 15 ] in terms of the amount of data per participant, which provides 6.4 hours of fMRI recordings per participant. Each participant listened to 27 short stories from the Moth Radio Hour podcast while undergoing a whole-brain scan; there is a total of 8 participants in the dataset. We use 25 stories of this dataset for brain-tuning and the remaining 2 stories as a held-out portion for estimating brain alignment (refer to Sections 2.3 and 2.4 ). We process this data by creating paired fMRI–audio snippets following [ 4 , 2 , 3 , 12 ] . This is done in three steps. First, a sliding window of 16\text{s} and a stride of 0.1\text{s} is applied to each audio recording. Second, we downsample the audio signal to match the slow fMRI acquisition rate using a three-lobed Lanczos filter (similar to [ 15 , 3 ] ). Third, we account for the delay in the hemodynamic response by modeling it as a finite response filter over 10 seconds [ 15 , 5 ] . This process yields a paired audio snippets-fMRI dataset for brain-tuning and computing brain alignment. 2.3 Brain-tuning For the speech model families mentioned in Section 2.1 , we follow the brain-tuning process detailed in [ 12 ] to brain-tune each pretrained model. After constructing the dataset as in Section 2.2 , given some input audio and its corresponding fMRI response, we fine-tune the pretrained speech model using the fMRI responses

Read the complete text