Towards Noise-Robust Speech Inversion through Multi-Task Learning with Speech Enhancement

Saba Tabatabaee, Carol Espy-Wilson · 2026

Read Towards Noise-Robust Speech Inversion through Multi-Task Learning with Speech Enhancement by Saba Tabatabaee, Carol Espy-Wilson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14516.txt.

Introduction

Saba Tabatabaee and Carol Espy-Wilson work on speech technology in this 2026 paper. Speech inversion maps speech sound to the motions of the vocal tract that made it. Recent studies show self-supervised learning speech representations work well for that task. Real-world use stays hard, though, because background noise is everywhere.

The team proposes a unified framework that joins two models. Speech enhancement cleans the signal, and speech inversion reads the tract motions. They are integrated through shared self-supervised speech representations. The title names the approach: multi-task learning. Readers get a design for speech inversion that stays robust when the room is noisy.

Catalog record

CREATORSaba Tabatabaee, Carol Espy-Wilson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14516

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Noise-Robust Speech Inversion through Multi-Task Learning with Speech Enhancement Saba Tabatabaee, Carol Espy-Wilson Abstract Recent studies demonstrate the effectiveness of Self Supervised Learning (SSL) speech representations for Speech Inversion (SI). However, applying SI in real-world scenarios remains challenging due to the pervasive presence of background noise. We propose a unified framework that integrates Speech Enhancement (SE) and SI models through shared SSL-based speech representations. In this framework, the SSL model is trained not only to support the SE module in suppressing noise but also to produce representations that are more informative for the SI task, allowing both modules to benefit from joint training. At a Signal-to-Noise Ratio of – – 5 dB, our method for the SI task achieves relative improvements over the baseline of 80.95% under babble noise and 38.98% under non-babble noise, as measured by the average Pearson product–moment correlation across all estimated parameters. © 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. † † address: 1 Department of Electrical and Computer Engineering, University of Maryland, College Park, USA 1 Introduction Speech production involves precise coordination of the lips, tongue, jaw, velum, and glottis to shape the acoustic signal [ 19 ] . Speech Inversion (SI) aims to recover this hidden articulatory activity by mapping speech onto vocal Tract Variables (TVs), which represent functional synergies among articulators rather than discrete positions [ 4 ] . The oral TVs are characterized by the degree and location of constrictions formed by the lips, tongue body, and tongue tip (see Figure 1). Figure 1: A visual illustration of the vocal tract variables. In addition to oral sounds, speech also includes nasal sounds, which are produced by partial or complete opening of the Velopharyngeal Port (VP). The degree of VP constriction regulates nasality by controlling airflow and acoustic coupling between the nasal and oral cavities through coordinated movements of the velum and pharyngeal walls. Direct measurement of VP activity is invasive, costly, and requires trained specialists. As a non-invasive alternative, nasalance, defined as the ratio of acoustic energy emitted through the nasal and oral cavities, serves as an indirect measure of VP constriction and has been validated through high-speed nasopharyngoscopy imaging [ 17 ] . In this study, nasalance is referred to as the VEL TV, as it serves as an acoustic proxy for velum movement patterns. Our investigation, supported by ablation studies [ 21 , 20 ] , shows that jointly estimating oral TVs, the VEL TV together with source features (SFs), including fundamental frequency (F0), periodicity (periodic energy), and aperiodicity (aperiodic energy) leads to improvements in their estimation. This is because these variables are inherently related and provide complementary information. Therefore, in this study, we adopt the recently developed SI system from [ 21 ] , which simultaneously estimates oral TVs, VEL TV, and three SFs. SI systems have a wide range of applications, including assessment for children with speech disorders [ 2 , 3 , 20 ] . However, deploying SI systems in real-world environments is challenging, as background noise can degrade performance if the system has not been adapted to such conditions. A previous study [ 16 ] demonstrated that adapting a SI system to noisy conditions can be achieved by fine-tuning with noise-contaminated speech signals. However, employing a Speech Enhancement (SE) model as a pre-processing step has been shown to improve robustness across various speech processing tasks [ 22 , 15 , 14 ] and to outperform approaches that rely solely on fine-tuning the target system with contaminated speech [ 15 , 14 ] . Therefore, in this study, we leverage a SE model to improve the adaptation of an SI system to noisy conditions under two different scenarios: (1) fine-tuning the SI system on audio pre-processed by the SE model, and (2) jointly training the SI and SE models within a multi-task learning framework. To the best of our knowledge, this study is the first to investigate multi-task learning of SI and SE models to improve SI robustness under noisy conditions. Recent advances in Self Supervised Learning (SSL) speech models have significantly improved feature extraction for SI systems, surpassing traditional acoustic representations such as MFCCs [ 6 ] . Among these models, WavLM-Large has been found to outperform other SSL models, including HuBERT-Large, on SI tasks [ 6 , 21 ] . Furthermore, several studies have explored the application of SSL models to SE tasks, reporting promising results [ 9 , 10 ] . Motivated by these findings, we employ the WavLM-Large pretrained SSL model [ 5 ] for the SI and SE systems to extract richer, more informative representations from the speech signal. Our key contributions in the current work are as follows: • Introduction of a novel multi-task learning framework jointly modeling SE and SI to estimate enhanced audio along with oral TVs, VEL TV, and three SFs. • Comparative analysis of employing a SE model as a pre-processing step for a SI model versus jointly integrating SE and SI models within a multi-task learning framework. • Evaluation of SI system robustness under babble and non-babble background noise across varying Signal-to-Noise Ratio (SNR) levels. 2 Data prepration 2.1 XRMB dataset The University of Wisconsin XRMB dataset [ 23 ] comprises speech recordings accompanied by point-source trajectories of the lips, tongue tip, and tongue body, captured using a rasterized X-ray microbeam tracking system. The dataset includes recordings from 46 native English speakers (21 males and 25 females), totaling approximately 5.74 hours of speech. The dataset is partitioned into training, development, and test sets in a speaker-independent manner, as detailed in Table 1. The original X–Y coordinates of the pellets were mapped to the TVs using a geometric transformation, as described in [ 1 ] . Because the TVs are based on relative measures, this mapping normalizes variability arising from anatomical differences among speakers. The processed XRMB dataset, following geometric transformation, comprises six oral TVs: LA, LP, TBCL, TBCD, TTCL, and TTCD (details are provided in Figure 1). The VEL TV was extracted from the XRMB speech signals using the method proposed in [ 21 ] , while ground-truth values for the three SFs were obtained from speech signals using the APP detector [ 8 ] . Table 1: Description of the XRMB dataset using speaker-independent splits. Split Number of subjects Number of Utterance Time (hours) Train 36 5801 4.47 Development 5 730 0.63 Test 5 754 0.64 2.2 Data augmentation for XRMB dataset For data augmentation, two categories of environmental noise were employed: (1) babble noise and (2) non-babble background noise. The training and development sets were augmented using the same noise data, whereas the test set was augmented with a separate set of noise data to assess SI and SE models performance under unseen noise conditions. Noise augmentation of train and development sets : For non-babble background noise, we used the MUSAN dataset [ 18 ] , and babble noise was generated from

Read the complete text