Tracking Articulatory Dynamics in Speech with a Fixed-Weight BiLSTM-CNN Architecture
Leena G Pillai, D. Muhammad Noorul Mubarak Elizabeth Sherly · 2025
Read Tracking Articulatory Dynamics in Speech with a Fixed-Weight BiLSTM-CNN Architecture by Leena G Pillai, D. Muhammad Noorul Mubarak Elizabeth Sherly (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18099.txt.
Introduction
Leena Pillai, Muhammad Noorul Mubarak, and Elizabeth Sherly predict how the tongue and lips move from the sound of speech alone. The tongue shapes airflow into distinct sounds, and tracking it normally needs special recording gear worn in the mouth. Writing in twenty twenty-five, their network stacks a bidirectional long short-term memory model with a one-dimensional convolutional stage for cleanup. A fixed-weight initialization scheme is the distinctive choice. Training used two data sets recorded with electromagnetic articulography, differing in speakers' origins, languages, phonetic range, and equipment.
Tests covered speaker-dependent and speaker-independent modes, plus same-corpus and cross-corpus transfer. The fixed-weight model beat the adaptive-weight version while training in relatively few epochs, the authors report. Robust articulation tracking feeds speech science, pronunciation training, and silent-speech interfaces.
Catalog record
| CREATOR | Leena G Pillai, D. Muhammad Noorul Mubarak Elizabeth Sherly |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18099
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tracking Articulatory Dynamics in Speech with a Fixed-Weight BiLSTM-CNN Architecture Leena G Pillai, D. Muhammad Noorul Mubarak Elizabeth Sherly Abstract Speech production is a complex sequential process which involve the coordination of various articulatory features. Among them tongue being a highly versatile active articulator responsible for shaping airflow to produce targeted speech sounds that are intellectual, clear, and distinct. This paper presents a novel approach for predicting tongue and lip articulatory features involved in a given speech acoustics using a stacked Bidirectional Long Short-Term Memory (BiLSTM) architecture, combined with a one-dimensional Convolutional Neural Network (CNN) for post-processing with fixed weights initialization. The proposed network is trained with two datasets consisting of simultaneously recorded speech and Electromagnetic Articulography (EMA) datasets, each introducing variations in terms of geographical origin, linguistic characteristics, phonetic diversity, and recording equipment. The performance of the model is assessed in Speaker Dependent (SD), Speaker Independent (SI), corpus dependent (CD) and cross corpus (CC) modes. Experimental results indicate that the proposed model with fixed weights approach outperformed the adaptive weights initialization with in relatively minimal number of training epochs. These findings contribute to the development of robust and efficient models for articulatory feature prediction, paving the way for advancements in speech production research and applications. keyword Acoustic-to-Articulatory Inversion, Smoothing techniques, Articulatory features, Weight initialization, Bidirectional Long Short-Term Memory (BiLSTM) 1 Introduction Acoustic features play a crucial role as the primary extracted information from speech signals. However, they are the outcomes of the underlying articulatory process. Acoustic-to-Articulatory Inversion (AAI) is a specific research area that focuses on the inverse mapping from acoustic features to their corresponding source articulatory features [ 1 ] . In simpler terms, AAI aims to identify or estimate ”how a speaker produces specific speech sounds?”. The relationship between acoustic and articulatory features is inherently one-to-many, and the mapping between them is nonlinear in nature [ 2 ] . The advancement of AAI has the potential to enhance various speech technologies, including speech analysis and synthesis [ 3 , 4 ] , speech and speaker recognition [ 5 , 6 , 7 , 8 ] , evaluation of pathological subjects [ 9 ] , and the diagnosis of depression from speech [ 10 ] . Articulatory movement tracking is essential for understanding the intricacies of speech production. The most common technologies utilized for tracking articulatory movements, including Magnetic Articulography Methods (e.g., EMA, PMA), Palatograph Methods (e.g., EPG, OPG), and Imaging methods (e.g., MRI, UTI, X-ray) [ 11 , 12 , 13 , 14 , 15 , 16 ] . All these technologies typically require physical contact with the subject. EMA has gained significant prominence in the fields of linguistics, pathological subject analysis and the researches in AAI, due to its advanced sensor-tracking technology. Its effectiveness in capturing and analyzing articulatory movements makes it particularly suitable for research focused on studying specific articulatory aspects. AAI mapping using EMA generated articulatory features serves as a bridge between the acoustic properties of speech and the underlying articulatory mechanisms, contributing to various fields such as linguistics, speech technology, clinical research, and language learning. Therefore, the EMA sensor dataset was chosen as the preferred technology for tracking and analyzing tongue and lip movements. Speech is a result of intricate coordination of active and passive articulators within the oral cavity. Estimating the articulatory involvement in speech production remains a crucial area of research. The tongue is the most versatile articulator in speech production but it is not directly visible during speech production as well. This work focuses on estimating the place and manner of the tongue and lip during speech production. A stacked Bidirectional Long Short-Term Memory (BiLSTM) model is employed to map articulatory features from acoustic features. Since articulatory features exhibit smooth sequential patterns, a Convolutional Neural Network (CNN) is employed to enhance the predicted trajectories with fixed weight initialization. By combining the strengths of BiLSTM and CNN, this work aims to improve the understanding of tongue and lip movement dynamics during speech production. The primary objectives of this work are: • To develop a robust and reliable methodology to estimate the articulatory movements of the tongue and lip from the acoustic speech signal by applying advanced sequential architecture, such as the stacked BiLSTM. • To enhance and smooth the estimated articulatory trajectories using a Convolutional Neural Network (CNN) with fixed weight. By attaining these goals, the effort intends to extend knowledge of tongue and lip articulatory movements, improve the precision of articulatory tracking, and contribute to the creation of novel tools for the detection and treatment of speech disorders. 2 Literature Review Acoustic to Articulatory Inversion (AAI) aims to understand the connection between the acoustic characteristics of speech and the associated articulatory movements involved in it. Due to the scarcity of parallel corpora, researchers explored statistical techniques to establish relationships between articulatory and acoustic space, including the codebook - based approach [ 17 , 18 ] . These early techniques, nevertheless, had trouble capturing the intricate and non-linear mapping between the two domains. Dynamic constraints were introduced to get around these limitations, which significantly improved AAI performance [ 19 ] . Researchers subsequently suggested various modelling techniques, such as feedforward neural networks [ 20 ] , Gaussian Mixture Models (GMM) [ 21 ] , Hidden Markov Models (HMM) [ 22 ] , and Mixture Density Networks (MDN) [ 23 , 24 ] . Even though these methods shown better performance, their architecture exclusively depended on parallel recordings of acoustic-articulatory data, which limits their ability to generalize. With the increased availability of acoustic-articulatory corpora, Deep Learning (DL) methods have gained popularity in addressing the AAI problem [ 25 , 26 , 27 ] . Despite these advancements, AAI still faces challenges, particularly in subject dependent scenarios with substantial parallel data requirements. To address this, [ 28 ] proposes a subject-adaptive AAI method (SA-AAI) using BiLSTM, achieving comparable performance to subject-dependent AAI (SD-AAI) with significantly less training data ( \sim 62.5%), offering a more cost-effective and practical solution. Additionally, [ 29 ] addresses the challenges of speech inversion with speaker variability by introducing a vocal tract length normalization (VTLN) technique to minimize speaker-specific details. The proposed VTLN approach transforms acoustic features of different speakers to a target speaker’s space, resulting in an 8.15% relative improvement in the correlation between actual and estimated tract-variable (TV) trajectories, indicating the effectiveness of VTLN in achieving a more robust and accurate speaker-independent speech inversion system. A novel AAI neural architecture with 1D convolutional filters for spectral sub-band processing, outperformed traditional models in both speaker-dependent and speaker-independent tasks. Further, a lightweight TCN-based system also outperforms BiLSTM for articulatory feature extraction [ 30 ] . In [ 31 ] , the researchers investigate the application of FastSpeech, a transformer framework that utilizes explicit duration modeling, to achieve consistent alignment between the articulatory movements and phonemes in Phoneme-to-Articulatory (PTA) motion estimation. By leveraging FastSpeech, the researchers aim to improve the performance of phoneme-to-articulatory motion estimation and enhance the understanding of the relationship between speech sounds and articulatory movements. Additionally, [ 32 ] explores the use of pretrained Self-Supervised Learning (SSL) features with transformer neural networks for AAI. Instead of traditional MFCCs, SSL features like TERA and DeCoAR, known for their