Confidence intervals for forced alignment boundaries using model ensembles

2025

Read Confidence intervals for forced alignment boundaries using model ensembles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01256.txt.

Introduction

Matthew Kelley, writing alone in 2025, gave speech alignment tools a way to show how unsure they are. Forced alignment matches recordings to transcripts, but most tools give one hard boundary per sound. Real sounds melt into each other, so a range is more honest.

Kelley ran ten trained sound classifiers over the same speech and let the spread of their boundaries set the range. The middle boundary became the estimate, wrapped in a wide confidence band built from order statistics. On two standard speech collections the group estimate slightly beat any single model. Output comes as data files for analysis and as the tier files linguists already use. Uncertain boundaries can now be flagged for human review.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Confidence intervals for forced alignment boundaries using model ensembles Abstract Forced alignment is a common tool to align audio with orthographic and phonetic transcriptions. Most forced alignment tools provide only a single estimate of a boundary. The present project introduces a method of deriving confidence intervals for these boundaries using a neural network ensemble technique. Ten different segment classifier neural networks were previously trained, and the alignment process is repeated with each model. The alignment ensemble is then used to place the boundary at the median of the boundaries in the ensemble, and 97.85% confidence intervals are constructed using order statistics. On the Buckeye and TIMIT corpora, the ensemble boundaries show a slight improvement over using just a single model. The confidence intervals are incorporated into Praat TextGrids using a point tier, and they are also output as a table for researchers to analyze separately as diagnostics or to incorporate uncertainty into their analyses. Confidence intervals for forced alignment \leftheader Kelley \authorsnames [1]Matthew C. Kelley \authorsaffiliations Linguistics Program, Department of English, George Mason University, Fairfax, Virginia, 22030, USA \authornote \addORCIDlink Matthew C. Kelley0000-0002-7218-5599 Correspondence should be addressed to Matthew C. Kelley. Email: [email protected]. keywords forced alignment, acoustic modeling, phonetics, speech segmentation, automatic speech recognition 1 Introduction Forced alignment is a tool used to annotate where words and/or segments occur in an audio signal. Forced aligners take in an audio file, an orthographic transcription, and a so-called “grapheme to phoneme” dictionary, which are used to produce a TextGrid with segment and word boundaries usable in Praat ( Boersma & Weenink 2024a ) . Such tools are still widely used across research in phonetics and speech science ( Mahr et al. 2021a ; Yuan et al. 2023a ) and were historically part of how automatic speech recognition systems were developed with hidden Markov models. These systems only provide a point estimate of the boundaries between segments and do not, traditionally, quantify the uncertainty around the boundary placement. The present paper introduces a method for deriving confidence intervals on forced alignment boundaries by using ensemble methods. A representational format for the confidence intervals using TextGrids in Praat ( Boersma & Weenink 2024a ) is also proposed. Finally, a method for comparing alignments with mismatched numbers of segment boundaries using dynamic time warping is evaluated, with a potential use case for comparisons between “ground-truth” transcriptions based on manual annotation and transcriptions derived from orthography. Ensemble methods—such as model averaging—are not unheard of in machine learning ( Breiman 1996a ; Carney et al. 1999a ) . They are prominent enough that competition-winning models have used such techniques to improve their performance ( Koren 2009a ; Krizhevsky et al. 2012a ) , and they are basic enough to be covered in textbooks ( Goodfellow et al. 2016a ) . Despite their potential to offer uncertainty estimates of the boundaries, ensemble methods do not yet seem to have been used in creating end-user forced alignment systems. A system that produces estimates of uncertainty in forced alignment models has a variety of potential benefits. The first of these is that it allows for more gradient representation of where it would be reasonable to place a boundary. Of course, confidence intervals should not be interpreted as theoretically-derived reasonable ranges for boundary placement. Second, ambiguous boundaries should exhibit greater uncertainty due to uncertainty in the acoustic models, offering a better reflection of the phonetic gradience between segment categories than would a single discrete boundary. Third, having confidence intervals would aid researchers in determining where the alignment model has made outlying predictions and caused significant errors in alignment. 2 The nature of forced alignment boundaries Before deriving confidence intervals, I believe it is important to lay out a solid interpretation of what a forced alignment boundary is. In doing so, the nature of the uncertainty in the boundaries can be made more explicit, which may help avoid misinterpretations over what a confidence interval on a boundary represents. The nature of these boundaries is not straightforward to reason about since the time points for segment boundaries do not come directly from an acoustic model that outputs conditional probabilities of segment classes given some acoustic input. Rather, the time points come from subsequent processing of the model output. For forced alignment systems that make use of boundaries from pre-aligned data ( Gorman et al. 2011a ; Kelley et al. 2024a ; McAuliffe et al. 2017a ; Kisler et al. 2012a ; Yuan & Liberman 2008a ) , an assumption is made that these are “true” boundaries, and the boundaries in the data used for training are taken as such. This assumption is made whether the boundaries are used for training or just comparison during evaluation. Yet, this assumption is clearly wrong since there is no “true” point of demarcation between two segments, trivially so because time is uncountably infinite and a range of segmentations is acceptable, and more meaningfully so because some segment combinations like approximant-vowel, vowel-vowel, and vowel-approximant sequences inherently have no clear, inarguable moment of demarcation. Nevertheless, this assumption is required in a utilitarian and not realist sense to get off the ground, much in the same way that linear regression assumes linearity. With the assumption of true boundaries in mind, for most forced alignment systems, the acoustic model is approximating a function producing a time-varying conditional distribution is learned that outputs P(\psi_{i}=\kappa|x_{i}) over all \kappa\in k , with k as the set of segment classes the acoustic model has. Additionally, \psi is the correct segment category and i represents a discrete time step on the interval [1,\,n] (with n as the total number of time steps in an audio file). In prose, the function outputs a distribution of probability over all segment categories in the system, which are specified by the user during training. Note that, for recurrent neural networks, the way that P is evaluated takes into account the sequential context of the moment where each individual x occurs. When P is evaluated over all n time steps in an audio file, a matrix of probabilities is produced. The logs of the probabilities in this matrix are used in the alignment process to find a path that yields the maximal cumulative probability of labels that yields the specified label sequence l . That is, through dynamic programming, the alignment algorithm ( Kelley et al. 2024a , such as the Viterbi algorithm or the modification in) yields the solution c defined as c=\mathop{\arg\max}_{s\in S}\sum_{\sigma_{i}\in s}\log(P(\psi=\sigma_{i}|x_{i}))\,, (1) where S is the set of all sequences s that map onto the length- m label sequence l . That is, each s will collapse into l (the correct transcription) when duplicates are removed. As a brief example, consider an audio file with five time-steps needing aligned labels, meaning that n=5 and all s\in S are also therefore of length 5. Assume the correct transcription is [las] for Spanish las ‘the. fem.pl ’, implying that m=3 . There are several sequences of segments that would yield [las] after removing duplicates. These include {laaas}, {llaas}, and {lasss}. Each of those examples would be one instance of s\in S , and each symbol in each sequence would be a \sigma_{i}\in s . For instance, for {laaas},

Read the complete text