MindVoice: Reconstructing Intelligible Speech from Non-invasive Neural Signals with Pretrained Priors
Guangyin Bao, Taiping Zeng, Jianfeng Feng, Xiangyang Xue · 2026
Read MindVoice: Reconstructing Intelligible Speech from Non-invasive Neural Signals with Pretrained Priors by Guangyin Bao, Taiping Zeng, Jianfeng Feng, Xiangyang Xue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31173.txt.
Introduction
Bao and colleagues rebuild intelligible speech from noisy, non-invasive brain recordings. Direct mapping to tangled speech codes yields sound that resembles speech spectrally but cannot be understood, so MindVoice splits the job: one path recovers meaning, another recovers fine acoustic detail. Fused with speech generation and voice cloning, the two paths produce natural utterances. Tests on EEG and MEG recordings beat prior methods across metrics.
Catalog record
| CREATOR | Guangyin Bao, Taiping Zeng, Jianfeng Feng, Xiangyang Xue |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62699 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31173
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MindVoice: Reconstructing Intelligible Speech from Non-invasive Neural Signals with Pretrained Priors Guangyin Bao, Taiping Zeng, Jianfeng Feng, Xiangyang Xue Abstract Reconstructing continuous speech from non-invasive neural recordings is a fundamental problem for probing human auditory perception and building safe, scalable speech brain-computer interfaces. Despite recent progress, intelligible reconstruction remains elusive, as non-invasive recordings are inherently noisy, spatially blurred, and only partially preserve information about perceived speech. Existing methods directly map neural activity to entangled speech representations before synthesizing waveforms with neural vocoders, resulting in spectral-similar but unintelligible results. To overcome these limitations, we introduce MindVoice , a neuro-to-speech reconstruction framework that uses pretrained models to compensate for the incomplete semantic and acoustic information in neural recordings. MindVoice disentangles reconstruction into two complementary pathways: one recovers high-level semantic content, while the other estimates fine-grained acoustic attributes. These inferred representations are then fused with powerful speech generation models and in-context voice cloning to synthesize natural and intelligible utterances. Extensive experiments on EEG and MEG demonstrate that MindVoice substantially outperforms existing methods on various metrics. These results show that pretrained priors provide a principled way to bridge the gap between noisy neural recordings and natural speech, highlighting a promising attempt for auditory neuroscience research and non-invasive speech brain-computer interfaces. 1 Introduction Auditory perception lies at the core of human cognition. This multi-level neural process not only encompasses the semantic content of speech, but also conveys rich acoustic information such as pitch, intonation, and emotional prosody. For decades, researchers have sought to understand how the brain comprehends complex speech signals 27 ; 57 ; 15 . Reconstructing original speech from neural activity provides direct evidence for uncovering the encoding mechanisms of auditory neural pathways, while also laying the groundwork for next-generation speech brain-computer interfaces. In the context of invasive settings, substantial progress has been made in speech-related decoding and reconstruction 51 ; 66 ; 4 ; 36 ; 47 ; 52 . Unfortunately, when using non-invasive neural signals with safety and broader applicability, such as electroencephalography (EEG) and magnetoencephalography (MEG), speech reconstruction remains a major technical challenge. This difficulty primarily stems from the low spatial resolution and massive noise of non-invasive recordings, which obscure auditory-evoked neural activity. At the same time, speech signals exhibit extremely fine-grained structures and spectrotemporal dynamics, making it difficult to establish precise and reliable mappings from scarce neural data that contain only weak and incomplete auditory information. Figure 1: Decoding of auditory-evoked non-invasive neural recodings . a , most extensive researches focus on decoding paraspeech information, whereas our goal is to directly reconstruct continuous and intelligible speech waveforms. b , existing EEG-to-speech methods align EEG with entangled speech representations and reconstruct speech using neural vocoders. c , our framework adopts a dual-stream architecture comprising semantic- and acoustic-level, which incorporates priors from pretrained models. Extensive prior work has investigated non-invasive speech decoding from auditory-evoked neural activity. As illustrated in Figure 1 a , these studies span a wide range of tasks, including speech envelope reconstruction 37 ; 18 , auditory-attention decoding 54 ; 69 ; 2 , voice activity detection 64 ; 53 ; 38 ; 16 ; 68 , prediction or retrieval of acoustic features such as mel-spectrograms 17 ; 44 , phonemes classification 16 ; 68 , and classification of words from small closed vocabularies 20 ; 50 . Collectively, these efforts establish important foundations and demonstrate the feasibility of decoding speech-related information from EEG or MEG signals. However, most existing studies focus on decoding paraspeech information, rather than directly reconstructing speech waveforms. As a result, they may provide limited insight into how the brain represents the acoustic and linguistic structure of natural speech. More recently, several pioneering studies 40 ; 39 have attempted to reconstruct continuous speech from EEG signals. As shown in Figure 1 b , these methods align neural activity with deep speech features and synthesize speech using neural vocoders. Nevertheless, the direct mappings learned from noisy neural recordings to entangled speech representations remain imprecise and therefore often yield reconstructed speech with limited intelligibility and natural fluency. Non-invasive neural recordings suffer from low signal-to-noise ratios and limited spatial resolution, making continuous speech reconstruction a fundamentally underdetermined problem. Models are required to recover information-dense, structurally complex, and temporally sensitive speech from sparse, noisy, and unstable neural observations. As a result, relying solely on neural recordings is insufficient for achieving this target. Meanwhile, recent advances in generative AI have shown that large-scale pretrained models for speech, text, and multimodal data encode rich acoustic and linguistic priors, enabling strong performance in automatic speech recognition (ASR) 55 ; 7 ; 8 ; 21 ; 65 , text-to-speech generation (TTS) 19 ; 12 ; 46 ; 29 , and language modeling 70 ; 25 ; 23 . Inspired by recent progress in reconstructing images and videos from neural activity 9 ; 13 ; 62 ; 63 ; 43 ; 14 ; 22 ; 48 ; 49 ; 73 , we suggest that continuous speech reconstruction should maximally leverage the priors encoded in such pretrained models to recover missing or semantically ambiguous speech information, thereby assembling intelligible speech from incomplete neural evidence. In light of the above discussion, we propose MindVoice , a neuro-to-speech reconstruction framework. Inspired by classical dual-stream accounts of speech perception 27 and hierarchical auditory processing 15 , our framework adopts a dual-stream architecture comprising semantic-level and acoustic-level streams, as illustrated in Figure 1 c . Rather than directly reconstructing speech from neural activity, this design decomposes the original problem into two lower-dimensional subproblems along complementary information axes, yielding more tractable alignment. Within each stream, we leverage priors encoded in pretrained models to compensate for incomplete speech information in neural signals, and further integrate the recovered semantic content with acoustic characteristics through text-to-speech generative models and in-context voice cloning to produce the final reconstructed speech. We conduct extensive evaluations on widely used EEG 10 and MEG 26 datasets, showing that MindVoice substantially outperforms prior neuro-to-speech reconstruction baselines across semantic accuracy, timbre, and speech quality metrics. Results across multiple datasets and evaluation splits demonstrate its generalizability. Qualitative comparisons and visualizations further corroborate these quantitative gains, while additional analyses and ablation provide insight into the model’s interpretability and reconstruction preference. Together, these results demonstrate the effectiveness of our dual-stream design and the use of pretrained priors to compensate for missing speech information in neural recordings. They further establish MindVoice as a state-of-the-art framework for reconstructing speech from EEG or MEG recordings, marking an important step in advancing neuro-to-speech reconstruction from garbled acoustic approximations toward intelligible speech. 2 Related Works Neural Speech Decoding . Neural speech decoding encompasses a wide range of studies. Neural recordings can be categorized by acquisition method as invasive, such as ECoG and MEA, or non-invasive, such as EEG, MEG, and fMRI. They can also be grouped by the task paradigm used to elicit neural activity, including speech listening, imagined speech, and overt speech. Invasive recordings provide high signal-to-noise ratios and fine spatiotemporal resolution, enabling precise capture of speech cortical dynamics. Accordingly, prior studies achieve high-accuracy and real-time decoding of listened and overt speech with invasive recordings 51 ; 66 ; 4 ; 47 ; 52 ; 67 , and begin to reconstruct imagined inner speech 36 ;