LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech

Bingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu, Lei Xie · 2026

Read LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech by Bingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu, Lei Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18220.txt.

Introduction

Bingshen Mu and co-authors present a language-model-based forced aligner for speech in many languages. Their 2026 preprint tackles forced alignment, which marks when each word or character starts and ends in a recording. Old methods are built per language and can drift over time, with errors piling up along a file. Speech large language models read many languages and handle long sequences, which suits them to the task. The authors build a non-autoregressive aligner on that base.

Non-autoregressive design predicts the timings in a more global way, which fights drift. One aligner for many languages lowers the cost of subtitles and speech data sets. Long-form speech, from lectures to podcasts, stands to gain most.

Catalog record

CREATORBingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu, Lei Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18220

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech Bingshen Mu, Xian Shi, Xiong Wang, Hexin Liu, Jin Xu, Lei Xie Abstract Forced alignment (FA) predicts start and end timestamps for words or characters in speech, but existing methods are language-specific and prone to cumulative temporal shifts. The multilingual speech understanding and long-sequence processing abilities of speech large language models (SLLMs) make them promising for FA in multilingual, crosslingual, and long-form speech settings. However, directly applying the next-token prediction paradigm of SLLMs to FA results in hallucinations and slow inference. To bridge the gap, we propose LLM-ForcedAligner , reformulating FA as a slot-filling paradigm: timestamps are treated as discrete indices, and special timestamp tokens are inserted as slots into the transcript. Conditioned on the speech embeddings and the transcript with slots, the SLLM directly predicts the time indices at slots. During training, causal attention masking with non-shifted input and label sequences allows each slot to predict its own timestamp index based on itself and preceding context, with loss computed only at slot positions. Dynamic slot insertion enables FA at arbitrary positions. Moreover, non-autoregressive inference is supported, avoiding hallucinations and improving speed. Experiments across multilingual, crosslingual, and long-form speech scenarios show that LLM-ForcedAligner achieves a 69%~78% relative reduction in accumulated averaging shift compared with prior methods. Checkpoint and inference code are available at https://github.com/QwenLM/Qwen3-ASR . 1 Introduction In speech processing, the objective of forced alignment (FA) is to estimate the start and end timestamps of each word or character in a speech signal, given its corresponding transcript. FA is indispensable in numerous applications, including large-scale speech corpus construction and cleaning, automatic subtitling and word-level highlighting, as well as duration modeling and prosody analysis in speech synthesis. With the continuous advancement of multilingual and multimodal applications, efficient and accurate FA has become increasingly essential ( Tseng et al. 2021 ; Liu et al. 2025 ) . Figure 1: Highlights of LLM-ForceAligner. Existing FA methods can be broadly categorized into two main groups: traditional hybrid systems ( McAuliffe et al. 2017 ) and end-to-end models ( Rastorgueva et al. 2023 ; Kürzinger et al. 2020 ; Shi et al. 2023 ; Bain et al. 2023 ) . Montreal forced aligner (MFA) is typically a hybrid Gaussian Mixture Model–Hidden Markov Model (GMM–HMM) framework, computing the timestamps via Viterbi decoding for frame-level phoneme-to-text alignment paths ( McAuliffe et al. 2017 ) . Connectionist temporal classification (CTC) is a common end-to-end FA method that leverages frame-to-token alignment computed by CTC-based automatic speech recognition (ASR) models, employing dynamic programming to identify the optimal path that aligns with the text sequence within constrained search paths ( Kürzinger et al. 2020 ; Rastorgueva et al. 2023 ) . Continuous integrate-and-fire (CIF) predicts a weight for each encoder output frame and integrates these weights over time. When the accumulated weight exceeds a threshold, a fire event is triggered, at which point a weighted sum of the accumulated frame-level acoustic vectors is computed to generate an acoustic embedding aligned with an output token, thereby enabling the assignment of a corresponding timestamp to each token ( Shi et al. 2023 ) . WhisperX employs a lightweight end-to-end phoneme recognition model to perform frame-level phoneme classification on speech, and then aligns the resulting phoneme sequence with the transcript using dynamic time warping (DTW), thereby obtaining word-level timestamps by aggregating phoneme-level timestamps ( Bain et al. 2023 ) . However, the aforementioned FA methods are tied to language-specific phonemes, lexicons, or structural designs, which means that in multilingual scenarios, deployment typically involves a collection of independent systems with disparate structures, leading to engineering costs and maintenance complexity that grow linearly with the number of languages. Furthermore, previous FA methods can be summarized as a process of calculating local acoustic similarities followed by a monotonic path search. While these methods can produce reasonably accurate boundaries for short segments, they frequently accumulate significant systematic temporal shifts in long-form speech. Large language models (LLMs) have demonstrated powerful abilities in multilingual text understanding and long-sequence processing tasks ( Touvron et al. 2023a ; Touvron et al. 2023b ; Yang et al. 2024 ; Yang et al. 2025 ) , offering a new possibility for FA that supports multilingual, crosslingual, and long-form speech. Increasing studies have explored integrating speech encoders with LLMs to build Speech LLMs (SLLMs) that process speech and text within a unified framework. Nevertheless, existing SLLMs have mainly achieved success in high-level semantic tasks such as ASR ( Geng et al. 2024 ; Mu et al. 2026 ) , speech understanding ( Geng et al. 2025b ; Chu et al. 2023 ; Chu et al. 2024 ) , speech synthesis ( Wang et al. 2025a ; Du et al. 2025 ) , and spoken dialogue ( Wang et al. 2025b ; Geng et al. 2025a ) . For FA, which is more sensitive to acoustic characteristics, these SLLMs typically treat it as a by-product of ASR and generate word-level or character-level timestamps via next-token prediction. Such a paradigm is susceptible to temporal non-monotonic hallucinations and incurs substantial inference latency. In this work, we propose a new FA framework named LLM-ForcedAligner that reformulates FA as a slot-filling paradigm: the start and end timestamps of each word or character are treated as discrete time indices, and dedicated special tokens are inserted into the transcript as slots so that, conditioned on the speech embeddings and the transcript augmented with these slots, the SLLM can directly predict the corresponding time indices at the designated slots. This new FA paradigm effectively leverages the LLM’s strengths in slot filling and long-context processing, extends conventional purely acoustic, phoneme-level alignment to semantic-boundary-aware, character-level or word-level alignment. To perform slot filling, we apply causal attention masking during training without introducing any shift between the input and label sequences, allowing each slot to predict its own time index based on itself and the preceding context, and we compute the loss function only at the slot positions. In addition, we adopt a dynamic slot insertion strategy that randomly decides whether to insert special tokens for each word or character in the transcript, enabling LLM-ForcedAligner to predict timestamps for arbitrary words or characters. During inference, LLM-ForcedAligner enables non-autoregressive decoding, completely avoiding hallucinations compared with autoregressive decoding and achieving faster speed. Experimental results show that, in multilingual, cross-lingual, and long-form speech scenarios of up to 300 seconds, LLM-ForcedAligner achieves a 69%~78% relative reduction in accumulated averaging shift (AAS) compared with prior FA methods, while incurring only a slight increase in real-time factor (RTF). Figure 2: Overview of LLM-ForcedAligner. Left: During training, we replace the word-level or character-level start and end timestamps in the transcript with a special token [time] to serve as slots, and use dynamic slot insertion to randomly determine which word or character slots to ignore. The speech embeddings produced by the AuT encoder are then concatenated and fed into the LLM for training with causal attention masking. When computing the cross-entropy loss, the output IDs and the labels are non-shifted, and the loss is computed only at the positions of the retained slots. Right: During inference, users can insert the special token [time] at arbitrary positions in the transcript and rapidly obtain

Read the complete text