Acoustic to Articulatory Inversion of Speech; Data Driven Approaches, Challenges, Applications, and Future Scope
Leena G Pillai D. Muhammad Noorul Mubarak · 2025
Read Acoustic to Articulatory Inversion of Speech; Data Driven Approaches, Challenges, Applications, and Future Scope by Leena G Pillai D. Muhammad Noorul Mubarak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13308.txt.
Introduction
Leena Pillai and Muhammad Mubarak review a decade of work on reading mouth motion from sound. When a person speaks, the tongue, lips, and jaw move in ways that sound alone hides. Acoustic to articulatory inversion tries to rebuild those moves from the speech signal. The review covers studies from twenty eleven to twenty twenty one.
The authors sort the work by speaker type, aim, and the body data used to train the models. Labs pair sound with scans or sensors that track the tongue and lips. Scores rest on how close the rebuilt motion comes to the true one. The review notes uses in speech therapy and language training, where a clear view of tongue motion helps.
Catalog record
| CREATOR | Leena G Pillai D. Muhammad Noorul Mubarak |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Acoustic to Articulatory Inversion of Speech; Data Driven Approaches, Challenges, Applications, and Future Scope Leena G Pillai D. Muhammad Noorul Mubarak Abstract This review is focused on the data-driven approaches applied in different applications of Acoustic-to-Articulatory Inversion (AAI) of speech. This review paper considered the relevant works published in the last ten years (2011-2021). The selection criteria considered are (a) type of AAI – Speaker Dependent and Speaker Independent AAI, (b) objective of the work – Articulatory approximation, Articulatory Feature space selection and Automatic Speech Recognition (ASR), explore the correlation between acoustic and articulatory features, and framework for Computer-assisted language training, (c) Corpus – Simultaneously recorded speech (wav) and medical imaging models such as ElectroMagnetic Articulography (EMA), Electropalatography (EPG), Laryngography, Electroglottography (EGG), X-ray Cineradiography, Ultrasound, and real-time Magnetic Resonance Imaging (rtMRI), (d) Methods or Models – recent works are considered, and therefore all the works are based on machine learning, (e) Evaluation – as AAI is a non-linear regression problem, the performance evaluation is mostly done by Correlation Coefficient (CC), Root Mean Square Error (RMSE), and also considered Mean Square Error (MSE), and Mean Format Error (MFE). The practical application of the AAI model can provide a better and user-friendly interpretable image feedback system of articulatory positions, especially tongue movement. Such trajectory feedback system can be used to provide phonetic, language, and speech therapy for pathological subjects. Keywords: Acoustic-to-Articulatory Inversion (AAI), Medical imaging models, Speech Processing, Acoustic and Articulatory feature space. 1 Introduction Human speech production is a complex process that is stacked with four sequential processes - initiation, phonation, oro-nasal process, and articulation Giegerich 1992 . The air stream that expelled from the lungs is the source of the speech. This air passes through the larynx, where the vocal folds state adds the voiced or voiceless features to the air stream. The velum controls the next stage of the air stream, and its movement is responsible to determines where the air pass through the nasal (nasal sounds-/m/, /n/, etc.) or oral cavity. Articulation is the process that shapes the air stream by chaining the place (where) and manner (how) of articulation. Basically, a phonological feature set, such as +bilabial, +plosive, -voicing, and so on, individually recognizes each phoneme and their phone classification conveys the information about the place and manner of articulation. Therefore, the interesting concern here is to explore the relationship between the articulatory and the acoustic space of the speech signal. The Acoustic-to-Articulatory Inversion (AAI) mapping problem can be defined as a non-linear regression problem that is entitled to inference the articulatory trajectories from the acoustic speech signal Atal et al. 1978 ; Toda et al. 2008 ; Ananthakrishnan and Engwall 2011 . Acoustic to Articulatory inversion is illustrated in Figure. 1 . Many studies were done on this acoustic articulatory correlation and applied in many applications like speech signal analysis and speech synthesis Ling et al. 2009 ; Steiner et al. 2012 , Automatic Speech Recognition Kirchhoff et al. 2002 ; Krishna et al. 2019 , speaker recognition Hong et al. 2020 , speech evaluation of pathological subjects Moro-Velazquez et al. 2021 , diagnosis of depression from speech Seneviratne and Espy-Wilson 2021 , and so on. Figure 1: From the Acoustic Features of the speech wave, the AAI estimate how the speech sounds are produced (Place and manner of the articulation). The forward problem that is identifying the acoustic features from the speech signal is a well-expanded research area. But the reverse problem, that is, identifying the articulatory features by the acoustic features, is really mystifying researchers over three decades. The main challenge in the AAI problem is its one-to-many nature. An articulatory state always gives a single acoustic comprehension, whereas an acoustic signal is the outcome of multiple articulatory states. As similar articulatory states can utter different acoustic signals, the problem is recognized as non-linear. But the feasible solutions and more enhancements are required in this research area [AAI]. The studies reveal that this accurate articulatory feature extraction and its appropriate representation can be a good aid for pathological language and speech development Li et al. 2019 ; Jones 2017 . This work focuses on the review of the selected research papers published from 2011 to 2021. The selection criteria are based on publication time, objective, corpus, features considered, machine learning models, and quantitative outcome of the work. 2 Objective This work focused on the study and discussion of the interesting area of research called Acoustic-to-Articulatory Inversion (AAI) of speech. The work selected is in such a way to familiarize the readers in the concept of AAI, its challenges, application areas, and moreover that the approaches experienced in this area, right from the data source availability, feature extraction methods, modeling, and evaluation methods. 3 Motivation The innovative and effective applications of AAI have high-level relevance in the betterment of society. In many cases, it is not applicable to use medical imaging models to evaluate the articulatory position and movement of the speaker during speech, and such cases we can suggest AAI articulatory feedback system to track the articulatory movements during speech production. Pronunciation, speech, and language learning is a challenging task for the hearing-impaired community. A self-explanatory, visual feedback system can make them understand easily and clearly the trajectory positions they are currently following and the actually required position they should practice to achieve. 4 Literature Review Several studies reveal that the visual feedback of the articulatory position of the speaker is a recommended aid for speech therapy and language training Byun et al. 2014 ; Roxburgh et al. 2015 ; Cleland et al. 2015 . Initial works on AAI was concentrated on speaker-dependent approaches (SD-AAI), which is based on codebook Hogden et al. 1996 ; Ouni and Laprie 1999 . The Suzuki, S., Okadome T, & Honda, M explored in their work that the performance of the AAI is dramatically enhanced by introducing dynamic constraints Suzuki et al. 1998 . An inversion based on feed-forward neural network model was proposed by Richmond K, King S, & Taylor P., Richmond et al. 2003 . To meet the real-world demands, the inversion system needs to be robust and speaker-independent (SI-AAI). Lately, the availability of simultaneously recorded articulatory and acoustic corpus radically increased the scope of SI-AAI. As the training data corpus availability increased, the traditional models uplifted to data-driven methods such as various Deep Artificial Neural Network frameworks Cai et al. 2018 ; Seneviratne et al. 2019 , Mixture Density Networks Uria et al. 2012 ; Illa and Ghosh 2020 , Recurrent Neural Network Liu et al. 2015 ; Xie et al. 2018 ; Biasutto–Lervat and Ouni 2018 , Convolutional Neural Network Yu et al. 2018 ; Mannem et al. 2019 , and so on. 5 Approaches towards the solution This problem area is mostly following data-driven approach. Now a days, machine learning is the dominating approach for any classification and regression problem. Any problem in machine learning should went through a sequence of activities such as (1) corpus collection and preprocessing, (2) feature extraction, (3) training and testing. 5.1 Speech - Articulatory Databases Several approaches are available to simultaneously capture speech signals and its corresponding articulatory data, listed in Table. 1 . Techniques Description Electropalatography