SIREM: Speech-Informed MRI Reconstruction with Learned Sampling

Md Hasan, Nyvenn Castro, Daiqi Liu, Lukas Mulzer, Jana Hutter, Jonghye Woo, Moritz Zaiss, Andreas Maier, Paula A. Pérez-Toro, [7pt] Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, [-1pt] Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany, [-1pt] Institut für Informationsverarbeitung, Leibniz Universität Hannover, Hannover, Germany, [-1pt] Department of Radiology, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA · 2026

Read SIREM: Speech-Informed MRI Reconstruction with Learned Sampling by Md Hasan, Nyvenn Castro, Daiqi Liu, Lukas Mulzer, Jana Hutter, Jonghye Woo, Moritz Zaiss, Andreas Maier, Paula A. Pérez-Toro, [7pt] Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, [-1pt] Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany, [-1pt] Institut für Informationsverarbeitung, Leibniz Universität Hannover, Hannover, Germany, [-1pt] Department of Radiology, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18221.txt.

Introduction

Md Hasan's 2026 paper presents SIREM, a speech-informed system for real-time MRI of speech. Fast vocal-tract imaging trades space detail, time detail, and speed, leaving thin data and poor pictures. SIREM uses synced speech as a prior, since vocal-tract shapes track the sounds made, fusing an audio branch with an MRI branch through a weight map. A learnable weighting over spiral sampling arms tunes how the two branches mix. On the USC speech benchmark it runs far faster than iterative methods while keeping anatomy plausible.

Catalog record

CREATORMd Hasan, Nyvenn Castro, Daiqi Liu, Lukas Mulzer, Jana Hutter, Jonghye Woo, Moritz Zaiss, Andreas Maier, Paula A. Pérez-Toro, [7pt] Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, [-1pt] Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany, [-1pt] Institut für Informationsverarbeitung, Leibniz Universität Hannover, Hannover, Germany, [-1pt] Department of Radiology, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40068
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SIREM: Speech-Informed MRI Reconstruction with Learned Sampling Md Hasan, Nyvenn Castro, Daiqi Liu, Lukas Mulzer, Jana Hutter, Jonghye Woo, Moritz Zaiss, Andreas Maier, Paula A. Pérez-Toro, [7pt] Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, [-1pt] Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg Erlangen, Germany, [-1pt] Institut für Informationsverarbeitung, Leibniz Universität Hannover, Hannover, Germany, [-1pt] Department of Radiology, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA Abstract Real-time magnetic resonance imaging (rtMRI) of speech production enables non-invasive visualization of dynamic vocal-tract motion and is valuable for speech science and clinical assessment. However, rtMRI is fundamentally constrained by trade-offs among spatial resolution, temporal resolution, and acquisition speed, often leading to undersampled k-space measurements and degraded reconstructions. We propose SIREM, a speech-informed MRI reconstruction framework that uses synchronized speech as a cross-modal prior. The central idea is that vocal-tract configurations during speech are correlated with the produced acoustics, making part of the image content predictable from audio. SIREM models each frame as a fusion of an audio-driven component and an MRI-driven component through a spatial weighting map. The audio branch predicts articulator-related structure from speech, while the MRI branch reconstructs complementary content from measured k-space data. We further introduce a learnable soft weighting profile over spiral arms, enabling a differentiable study of how k-space arm usage interacts with speech-informed fusion. This yields a unified multimodal formulation that combines audio-driven prediction, MRI reconstruction, and sampling adaptation. We evaluate SIREM on the USC speech rtMRI benchmark against standard baselines, including gridding, wavelet-based compressed sensing, and total variation. SIREM introduces a speech-informed reconstruction paradigm that operates in a substantially higher-throughput regime than iterative methods while preserving anatomically plausible vocal-tract structure. These results establish an initial benchmark for multimodal speech-informed rtMRI reconstruction and highlight the potential of synchronized speech as an auxiliary prior for fast reconstruction. The source code is available at https://github.com/mdhasanai/SIREM 1 Introduction Real-time magnetic resonance imaging (rtMRI) visualizes the dynamics of speech production by capturing the full midsagittal vocal tract, tongue, lips, velum, pharynx, and laryngeal structures without ionizing radiation or articulatory constraint. Unlike electromagnetic articulography 3 or ultrasound 12 , this makes rtMRI uniquely suited for articulatory and phonetic analysis 21 , as well as clinical assessment of conditions such as velopharyngeal insufficiency, dysarthria, and post-surgical speech impairments 7 . To satisfy temporal-resolution constraints, speech rtMRI reconstructs only a fraction of k-space per frame, turning reconstruction into a strongly ill-posed inverse problem 13 . Standard parallel imaging reconstruction methods such as SENSE 20 and GRAPPA 6 exploit coil and k-space redundancies. Speech-specific dynamic reconstruction methods further exploit temporal structure through low-rank, manifold, or subspace models 5 ; 22 ; 4 ; 13 . These approaches have been effective, but they rely primarily on measurement-domain, image-domain, or temporal priors and do not explicitly use the synchronized speech waveform available during acquisition. Speech rtMRI is, however, intrinsically multimodal: each acquisition includes a synchronized audio waveform in addition to the MRI measurements 14 . The acoustic signal is not merely an annotation, but a physical consequence of the vocal-tract configuration that the MRI seeks to image. Articulatory studies show that the tongue, lips, jaw, and velum shape the spectral envelope, formant trajectories, and voicing structure of speech 21 , and recent audio-conditioned rtMRI synthesis and inversion studies suggest that substantial articulatory information can be recovered from speech alone 27 ; 17 ; 24 ; 19 . Thus, the audio channel contains structured cues about the underlying image content that are typically discarded by reconstruction pipelines. These observations motivate us to formulate speech rtMRI reconstruction as a multimodal estimation problem in which synchronized speech provides an auxiliary prior on anatomically relevant regions. Hence, we propose SIREM, a hybrid framework that fuses an audio-driven image estimate with an MRI-driven reconstruction via a spatial weighting map, and incorporates a learnable soft spiral-arm reweighting profile to study k-space arm usage under speech-informed fusion. Our contributions are: • We formulate speech rtMRI reconstruction as a multimodal inverse problem in which synchronized audio serves as a cross-modal prior on vocal-tract structure. • We propose SIREM, a hybrid framework that fuses audio-driven and MRI-driven estimates via a spatial weighting map and a learnable soft spiral-arm reweighting profile. • We establish an initial benchmark for multimodal speech-informed rtMRI reconstruction on the USC corpus, comparing against gridding, wavelet, and total variation baselines. SIREM does not uniformly outperform classical iterative reconstruction: wavelet retains the strongest \ell_{2} -style distortion scores in every evaluation protocol. Yet demonstrates that multimodal direct reconstruction is feasible, that synchronized audio is a useful auxiliary prior, and that the resulting throughput regime is qualitatively distinct from iterative approaches. 2 Related Work Speech rtMRI has become a central modality for observing the moving vocal tract because it captures the full midsagittal anatomy, including structures that are difficult to measure with electromagnetic articulography or ultrasound. Public speech-rtMRI resources, particularly the USC 75-speaker corpus, provide synchronized audio, raw multi-coil non-Cartesian k-space, and reconstructed image sequences ( 14 ) . In this work, we use the 75-Speaker Annot-16 subset, which provides manual articulatory annotations and supplies the spatial weighting masks used in the reported experiments ( 25 ) . Classical accelerated MRI reconstruction is built on parallel imaging and compressed sensing. SENSE 20 and GRAPPA 6 exploit spatial encoding redundancy across receiver coils, while compressed-sensing MRI combines data consistency with sparsity-promoting priors, such as wavelets and total variation ( 16 ) . Speech-specific rtMRI reconstruction has also explored dynamic priors including temporal finite-difference regularization, low-rank-plus-sparse models, manifold regularization, and subspace-based reconstruction for vocal-tract imaging ( 5 ; 22 ; 4 ) . These approaches improve reconstruction by exploiting measurement-domain or temporal structure, but they do not use the synchronized waveform as an explicit prior. More recent MRI reconstruction has shifted toward physics-informed deep learning and unrolled optimization in accelerated MRI ( 1 ; 8 ; 26 ) . A related direction studies joint optimization of sampling and reconstruction, treating acquisition design as a learnable component rather than a fixed heuristic ( 2 ; 31 ) . SIREM is conceptually related to this line of work, but addresses a distinct setting in which synchronized audio provides side information for reconstruction from raw spiral k-space. Synchronized speech has also been used for vocal-tract modeling in generation, inversion, and segmentation tasks, audio-conditioned rtMRI synthesis, diffusion-based speech-to-rtMRI generation, and audio-assisted vocal-tract segmentation ( 27 ; 17 ; 19 ; 24 ; 11 ; 15 ) . These studies support the premise that speech contains substantial articulatory information, yet the inclusion in speech imaging reconstruction pipelines remains largely unexplored. SIREM instead targets the complementary problem of using synchronized speech as an auxiliary prior for reconstructing undersampled speech rtMRI directly from raw k-space measurements. 3 SIREM We consider speech-informed reconstruction for real-time MRI from undersampled non-Cartesian k-space measurements. Our central hypothesis is that part of the articulatory configuration is predictable from synchronized speech audio, while the remaining anatomy must be recovered from measured MRI data. Following this decomposition, we model each frame as a fusion of an audio-driven estimate and an MRI-driven reconstruction, and we learn a soft sampling policy over spiral arms to study whether acoustically predictable regions permit more aggressive undersampling. Figure 1: Overview of SIREM. The audio branch

Read the complete text