LCS-CTC: Leveraging Soft Alignments to Enhance Phonetic Transcription Robustness

Zongli Ye 1 , Jiachen Lian 2 , Akshaj Gupta 2 , Xuanru Zhou 1 , Haodong Li 3 , Krish Patel 2 , Hwi Joo Park 2 , Dingkun Zhou 4 , Chenxu Guo 1 , Shuhe Li 1 , Sam Wang 2 , Iris Zhou 2 , Cheol Jun Cho 2 , Zoe Ezzes 5 , Jet M.J. Vonk 5 , Brittany T. Morin 5 , Rian Bogley 5 , Lisa Wauters 5 , Zachary A. · 2025

Read LCS-CTC: Leveraging Soft Alignments to Enhance Phonetic Transcription Robustness by Zongli Ye 1 , Jiachen Lian 2 , Akshaj Gupta 2 , Xuanru Zhou 1 , Haodong Li 3 , Krish Patel 2 , Hwi Joo Park 2 , Dingkun Zhou 4 , Chenxu Guo 1 , Shuhe Li 1 , Sam Wang 2 , Iris Zhou 2 , Cheol Jun Cho 2 , Zoe Ezzes 5 , Jet M.J. Vonk 5 , Brittany T. Morin 5 , Rian Bogley 5 , Lisa Wauters 5 , Zachary A. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03937.txt.

Introduction

Zongli Ye and co-authors improve speech to text at the level of single sounds in 2025, using team framing for a long author list. Fine sound records matter for language study and speech tools. A common method is fast but weak on unclear or broken speech. Their method adds a step that finds high trust zones in the sound.

A local matching method builds a cost map between frames and sounds. High trust zones then narrow the paths the main method may take. That cut reduces overfit and boosts work on new data. Tests on clean speech and on speech from patients both beat the base method. The tool also aligns text to sound without a transcript.

Catalog record

CREATORZongli Ye 1 , Jiachen Lian 2 , Akshaj Gupta 2 , Xuanru Zhou 1 , Haodong Li 3 , Krish Patel 2 , Hwi Joo Park 2 , Dingkun Zhou 4 , Chenxu Guo 1 , Shuhe Li 1 , Sam Wang 2 , Iris Zhou 2 , Cheol Jun Cho 2 , Zoe Ezzes 5 , Jet M.J. Vonk 5 , Brittany T. Morin 5 , Rian Bogley 5 , Lisa Wauters 5 , Zachary A.
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LCS-CTC: Leveraging Soft Alignments to Enhance Phonetic Transcription Robustness Zongli Ye 1 , Jiachen Lian 2 , Akshaj Gupta 2 , Xuanru Zhou 1 , Haodong Li 3 , Krish Patel 2 , Hwi Joo Park 2 , Dingkun Zhou 4 , Chenxu Guo 1 , Shuhe Li 1 , Sam Wang 2 , Iris Zhou 2 , Cheol Jun Cho 2 , Zoe Ezzes 5 , Jet M.J. Vonk 5 , Brittany T. Morin 5 , Rian Bogley 5 , Lisa Wauters 5 , Zachary A. Miller 5 , Maria Luisa Gorno-Tempini 5 , Gopala Anumanchipalli 2 Abstract Phonetic speech transcription is crucial for fine-grained linguistic analysis and downstream speech applications. While Connectionist Temporal Classification (CTC) is a widely used approach for such tasks due to its efficiency, it often falls short in recognition performance, especially under unclear and nonfluent speech. In this work, we propose LCS-CTC, a two-stage framework for phoneme-level speech recognition that combines a similarity-aware local alignment algorithm with a constrained CTC training objective. By predicting fine-grained frame-phoneme cost matrices and applying a modified Longest Common Subsequence (LCS) algorithm, our method identifies high-confidence alignment zones which are used to constrain the CTC decoding path space, thereby reducing overfitting and improving generalization ability, which enables both robust recognition and text-free forced alignment. Experiments on both LibriSpeech and PPA demonstrate that LCS-CTC consistently outperforms vanilla CTC baselines, suggesting its potential to unify phoneme modeling across fluent and non-fluent speech. I Introduction Within-word variation in human speech poses challenges for ASR systems [ 1 , 2 , 3 , 4 , 5 ] that focus solely on word-level transcription. Yet, modeling subword structures like syllables or phonemes is essential for applications in language learning and clinical analysis [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . This work focuses on phoneme transcription—a particularly challenging yet critical task. Several factors contribute to the difficulty of phoneme-level transcription. First, unlike word-level targets, phoneme annotations are inherently non-deterministic due to accent, allophony and other context-dependent realizations [ 19 ] . Second, existing training objectives, such as CTC [ 20 ] and attention-based models [ 21 ] , often introduce alignment noise. For example, CTC tends to produce overly peaked label distributions [ 22 ] , which result in inaccurate timing and error-prone alignments. Attention-based models, on the other hand, may hallucinate transcriptions [ 23 ] —an issue that is particularly problematic in clinical applications where precision is critical. Third, most existing remedies are designed for fluent speech and demonstrate limited effectiveness on non-fluent speech. For example, several approaches [ 24 , 25 , 26 ] enhance CTC-based alignment by introducing heuristic constraints on the prior. However, these heuristic alignments have been shown to be incompatible with non-fluent speech [ 27 , 28 , 14 , 15 , 29 ] . Given the aforementioned challenges, a key question arises: Can we design a phoneme transcription objective that performs robustly simultaneously on both fluent and non-fluent speech, while also producing reliable and accurately aligned outputs suitable for clinical applications? A key insight arises from treating phoneme transcription as a speech production process , particularly when speakers are prompted to read reference text. Fluent speech results from accurate production, while deviations introduce non-fluent speech. For example, when given the phoneme sequence “IH N S ER T,” a speaker might produce “IH [S] N S ER [AH] T,” introducing insertions on “IH” and “ER.” This yields an alignment such as “IH–(IH,S), N–(N), S–(S), ER–(ER,AH), T–(T)” (label–spoken). The core idea is that the loss function should account only for effective speech-text alignments . In the case of “IH–(IH,S),” frames aligned to the extraneous “S” should be excluded when computing the loss for “IH.” Without such errors, the method reduces to the standard ASR objective. This alignment strategy follows the Longest Common Subsequence (LCS) principle [ 30 ] , which was recently shown effective for non-fluent speech [ 14 ] . Building on this intuition, we apply the LCS algorithm online to identify effective alignments, which are then used to regularize the vanilla CTC [ 20 ] paths, as shown in Fig. 2 . We refer to this method as LCS-CTC . Unlike previous methods that introduce heuristics to regularize CTC alignments [ 24 , 25 , 26 ] , LCS-CTC is grounded in human speech production, interpretable, and clinically meaningful . Our LCS-CTC framework consists of two main components: a phoneme-aware alignment module based on a modified Longest Common Subsequence (LCS) algorithm, and a constrained CTC training objective that incorporates the resulting alignment masks. The alignment module first predicts a frame-phoneme cost matrix using a lightweight neural network trained with weak supervision. This matrix encodes pairwise acoustic-phonetic similarity and is used to derive a partial alignment path via a similarity-aware LCS dynamic programming procedure. The resulting alignment mask identifies high-confidence frame-phoneme pairs and is used in two complementary ways during training: (1) it anchors specific frames to phoneme labels with cross-entropy supervision, and (2) it restricts the CTC path space via masked emission probabilities. Together, these two components enable robust phoneme recognition by combining interpretable alignment guidance with flexible sequence modeling. We evaluate our method on both fluent (LibriSpeech [ 31 ] ) and non-fluent corpora, including PPA speech [ 32 ] and the largest existing simulated non-fluent corpus, LLM_dys [ 33 ] . Results show that LCS-CTC consistently outperforms vanilla CTC across all metrics, including (weighted) phoneme error rate and duration-aware alignment accuracy. Notably, while LCS-CTC is inspired by non-fluent speech, it also provides stronger regularization for fluent speech. Although we have not yet explored its scalability across data types, linguistic diversity, or domains, the method shows strong potential and may serve as a foundation for universal phoneme recognition. We have open-sourced our model and checkpoints at https://github.com/Auroraaa86/LCS-CTC II Methods (a) (b) Fig. 1 : (a) shows the structure of cost matrix learning model. (b) shows the process from predicted cost matrix to alignment mask: we use cost matrix to construct the dynamic programming matrix and implement modified LCS algorithm to realize a local wise alignment, the dark red dots indicate credible local alignment regions. Fig. 2 : Overview of LCS-CTC framework : (a) Vanilla CTC allows all valid alignment paths that collapse to the target sequence, which can result in peaky distributions. (b) Our LCS-CTC introduces alignment constraints from the LCS-based aligner (black nodes), which eliminate implausible paths (red crosses) by fixing ”high-confidence” frame-phoneme matches. (c) The overall training pipeline of LCS-CTC. The LCS aligner uses ground-truth phonemes and predicted emissions to generate alignment masks, which are then used both as supervision (via \mathcal{L}_{\text{CE}} ) and constraint (via masked \mathcal{L}_{\text{CTC}} ) during recognition model training. Algorithm 1 LCS-based Sequence Alignment 1: Input: Label (phoneme labels), C (cost matrix), tol (tolerance), phn\_sim (similarity dictionary) 2: Output: Alignment mask 3: procedure Align ( Label , C , tol , phn\_sim ) 4: valid\_res\leftarrow\emptyset 5: for (i,j)\in[0,n)\times[0,m) do 6: phn1\leftarrow Label[i] , phn2\leftarrow Label[k] \triangleright k=\arg\min C[:,j] 7: sim\leftarrow phn\_sim[(phn1,phn2)] \triangleright Lookup similarity 8: if C[i][j]\leq(1-sim)\times tol then \triangleright Similarity-adjusted threshold 9: valid\_res\leftarrow valid\_res\cup\{(i,j),(j,i)\} 10: end if 11: end

Read the complete text