Evaluating Speech Articulation Synthesis with Articulatory Phoneme Recognition

Vinicius Ribeiro †, Yves Laprie · 2026

Read Evaluating Speech Articulation Synthesis with Articulatory Phoneme Recognition by Vinicius Ribeiro †, Yves Laprie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20920.txt.

Introduction

V. Ribeiro and Yves Laprie test speech articulation synthesis with articulatory phoneme recognition. Machine learning advances and speech-motion datasets now let vocal tract synthesis be conditioned on phonetic sequences, a key step. Yet quality scoring lacks a clear meaning. Ranking machine-made models is tricky due to own taste, and this adds special knowledge of vocal tract anatomy and sound. Their proposal uses phoneme recognition as a proxy, on the idea that spotting from motion traits better captures nuances like correct places of speech.

Catalog record

CREATORVinicius Ribeiro †, Yves Laprie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21912
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20920

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Speech Articulation Synthesis with Articulatory Phoneme Recognition Vinicius Ribeiro †, Yves Laprie Abstract Recent advances in machine learning and the availability of articulatory datasets allow vocal tract synthesis to be conditioned on phonetic sequences, a primary task of articulatory speech synthesis. However, quality assessment needs a better definition. Generally, ranking generative models is tricky due to subjectivity. However, articulatory synthesis has the additional difficulty of requiring specialized knowledge in vocal tract anatomy and acoustics. To address this problem, this paper proposes to evaluate speech articulation synthesis using phoneme recognition as a proxy. Our hypothesis is that phoneme recognition using articulatory features better captures nuances in phoneme production, such as correct places of articulation, which traditional metrics (e.g., point-wise distance metrics) do not. We train a neural network with acoustic and articulatory features extracted from a single-speaker RT-MRI dataset. Then, we compare the recognition performance when testing the model with different synthetic articulatory features. Our results show that our articulatory feature set is phonetically rich and helps exploring additional dimensions on speech articulation synthesis. Accepted for publication at the European Signal Processing Conference (EUSIPCO), 2026. $\dagger$ $\dagger$ footnotetext: This work was conducted while the author was a PhD student at the affiliated laboratory. I Introduction Ribeiro et al. [ 15 ] described how to synthesize the shape of the vocal tract conditioned in the sequence of phonemes to be articulated, exploring several methods for synthesizing vocal tract articulators during speech. The baseline method was a phoneme-wise mean-contour, which computes the average contour for each phoneme in a single-speaker real-time MRI (RT-MRI) dataset. Then model-free [ 12 , 13 ] and autoencoder-based [ 14 ] vocal tract shape synthesizers were used to generate complete vocal tract shapes from the sequences of phonemes to be articulated. On the one hand, the two latter models consistently outperformed the baseline and presented indistinguishable performances between each other in terms of point-to-closest-point distance. On the other hand, analysis of tract variables indicated that the autoencoder-based method learns better places of articulation, performing better constrictions than the model-free system. However, assessing the quality of synthesized articulatory features from speech remains a challenge. Point-wise distance metrics are easy to interpret, but the usage is limited due to substantial inter- and intra-speaker variability. In contrast, measuring tract variables associated with each target phoneme fits well consonants but does not suit vowels because the latter is characterized by the resonator’s shape and not by constrictions. A better metric for vowels would be the measurement of formant frequencies [ 18 ] by solving simplified aero-acoustic equations in the synthetic vocal tract, which is computationally expensive. Nevertheless, the two metrics were inconclusive regarding the two proposed models, being satisfactory only for comparing with the phoneme-wise mean-contour, which is a very simplistic model. Subjective analysis indicated that synthetic utterances using the model-free approach seem more stable and temporally consistent than those of the autoencoder-based system, but the traditional metrics implemented fail to reflect this perception. Recent research has paid significant attention to the classification of articulatory characteristics and their use in understanding the relationship between articulations and acoustics and how neural networks map the two. Elie et al. [ 6 ] used phoneme recognition probability as a measure of intelligibility in their cost function. Saha et al. [ 16 ] trained a Long-Term Recurrent Convolutional Network to classify 51 vowel-consonant-vowel (VCV) contexts from RT-MRI films from 17 speakers, obtaining an accuracy of 42\% . Van Leeuwen et al. [ 21 ] trained a CNN to classify sustained phonemes (vowels and fricatives) from static mid-sagittal MRI and obtained an accuracy of 57\% . Interestingly, the model learned representations compatible with the vowel chart, showing that although accuracy is limited, the model is consistent with standard phonetic knowledge. In the problem of evaluating synthesized vocal tract shapes, Engwall [ 7 ] used an articulatory classifier as an evaluation metric for acoustic-to-articulatory inversion of VCV words in Swedish sentences using linear estimation and neural networks. Engwall’s research shows that the articulatory classifier provides a more intelligible metric than the RMS error and correlation coefficients. Inspired by these articles, we took a similar direction by using phoneme recognition to measure phonetic information in mid-sagittal RT-MRI contours. We analyzed speech articulations generated by vocal tract shape synthesizers from the literature using phoneme recognition. We first trained a phoneme recognizer on the acoustic signal as a baseline. Then, we trained the recognizer on the real articulators’ contours (true articulatory features). Since the mid-sagittal RT-MRI does not include vocal fold excitation, we add a categorical encoding representing voicing information. We quantify the information retained by the vocal tract contours by comparing the recognition error with the acoustic signal and the true articulatory features with and without voicing encoding. Next, the vocal tract shapes of the utterances in the test set were synthesized using the phoneme-wise mean-contour, model-free and autoencoder-based systems. These synthetic features with voicing encoding are input into the phoneme recognizer trained with true articulations. The recognition error of this test exhibits how much phonetic information the synthesizer can reproduce. We hypothesize that if the true contours carry enough information, the synthetic articulations should also exhibit a recognition performance comparable to the true articulatory features. II Dataset Our corpus contains 2.5 hours of speech from a single female native French speaker. It is composed of RT-MRI images (50 Hz), de-noised audio signals and manually corrected phonetic annotations. To our knowledge, this database is the largest dynamic MRI database for a speaker, which ensures the relevance of the training process. The acoustic features were obtained by computing Mel spectrograms with 80 frequency bands. The contours of ten vocal tract articulators were extracted from the images using the method described in Ribeiro et al. [ 11 ] . These contours are also used to learn the link between the acoustic signal and the shape of the vocal tract as part of our work on articulatory to acoustic inversion [ 2 ] . The articulators are the arytenoid cartilage, epiglottis, lower incisor, lower lip, pharynx, soft palate, thyroid cartilage, tongue, upper lip, and vocal folds. In addition, the upper incisor is used as a reference for the coordinate system. However, it is kept apart from the experiments. The contours of the ten individual articulators ( x_{i}\in\mathbb{R}^{2\times 50},~i\in[1,10] ) from Figure 1 , where 50 refers to the number of samples in each curve, were concatenated composing a 2-channel articulatory feature vector with the x- and y-axis in the channel dimension ( x^{\prime}\in\mathbb{R}^{2\times 500} ). We trained the vocal tract shape synthesizer from [ 13 , 14 ] with this work’s dataset to obtain the synthetic articulatory features. Arytenoid Cart. Epiglottis Lower Incisor Lower Lip Pharynx Soft Palate Thyroid Cart. Tongue Upper Incisor Upper Lip Vocal Folds Fig. 1 : Articulatory features used for phoneme recognition plus the upper incisor, which is the reference for the coordinate system. Table I summarizes the number of utterances and duration of each dataset split. The phonetic vocabulary comprises 50 tokens, from which 42 are phonetic tokens, and 8 are non-phonetic, representing blank token, silence, unknown token, and noise after /i, e, u, y, ø/. Unvoiced plosives are characterized by two phases: closure and burst. Therefore,

Read the complete text