In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions
Xulin Fan \sthanks Work done during an internship at IBM Research Vishal Sunder Samuel Thomas Mark Hasegawa-Johnson Brian Kingsbury George Saon · 2026
Read In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions by Xulin Fan \sthanks Work done during an internship at IBM Research Vishal Sunder Samuel Thomas Mark Hasegawa-Johnson Brian Kingsbury George Saon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22817.txt.
Introduction
Fan and five colleagues extend a speech-aware language model so it predicts word-level timestamps together with its transcript. That removes the need for outside alignment tools. Their light training methods strengthen alignment while keeping recognition quality high, aiming at captioning, media search, and syncing speech with other media. Across several data sets the approach improves time-stamp accuracy and also raises overall speech-recognition scores.
Catalog record
| CREATOR | Xulin Fan \sthanks Work done during an internship at IBM Research Vishal Sunder Samuel Thomas Mark Hasegawa-Johnson Brian Kingsbury George Saon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21785 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22817
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions Xulin Fan \sthanks Work done during an internship at IBM Research Vishal Sunder Samuel Thomas Mark Hasegawa-Johnson Brian Kingsbury George Saon Abstract Recent advances in speech–aware language models have coupled strong acoustic encoders with large language models, enabling systems that move beyond transcription to produce richer outputs. Among these, word-level timestamp prediction is critical for applications such as captioning, media search, and multimodal synchronization, yet it is often handled by external alignment tools. In this work, we extend an existing speech-aware language model to predict timestamps directly alongside transcripts. We introduce a set of novel lightweight training strategies that improve alignment robustness while preserving recognition quality. Experiments across multiple datasets show that these strategies not only enhance timestamp accuracy, but also yield gains in overall ASR performance. Together, they demonstrate an efficient and unified approach to speech recognition with precise timestamp prediction. † † address: 1 University of Illinois Urbana-Champaign 2 IBM Research 1 Introduction The field of automatic speech recognition (ASR) has been fundamentally reshaped over the last decade, primarily through self-supervised learning (SSL). This paradigm began with pretrained acoustic encoders trained on massive unlabeled audio. Seminal models such as wav2vec 2.0 [ 2 ] learned representations directly from raw waveforms via contrastive masking, while HuBERT [ 7 ] improved this approach using predictive losses over clustered discrete units. These frameworks proved highly effective at capturing acoustic and phonetic features, establishing pretrain–finetune pipelines that now form the backbone of modern ASR. Building on these advances, recent work connects pretrained acoustic models with large language models (LLMs), giving rise to speech large language models (SpeechLLMs) that combine the perceptual strength of audio encoders with the linguistic knowledge of LLMs. AudioPaLM [ 20 ] extends PaLM-2 to unify speech understanding and generation, SpeechGPT [ 28 ] adapts LLMs for spoken dialogue, SALMONN [ 24 ] augments LLaMA [ 25 ] with audio inputs to create general-purpose “hearing models”, and SLAM-LLM [ 12 ] describes a straightforward method for integrating an acoustic encoder with a LLM. This trend shifts ASR from transcription-only toward holistic spoken language processing and interactive applications. Beyond accurate transcription, however, many practical applications require fine-grained temporal alignment between speech and text. Word-level timestamps are essential for applications such as closed captioning, video indexing, keyword-based audio retrieval, and multimodal synchronization. Traditionally, timestamps have been produced by forced alignment using hidden Markov models (HMMs), implemented in toolkits such as HTK [ 27 ] and in hybrid ASR systems such as Kaldi [ 16 ] . The Montreal Forced Aligner [ 13 ] makes such pipelines more accessible, but they still require separate alignment passes and additional acoustic models depending on the pronunciation dictionary. Similarly, The Nemo Forced Aligner (NFA) [ 19 ] applies Viterbi decoding to the output of CTC-based models to derive timestamps. There are multiple works [ 4 , 3 ] that builds upon a two-pass approach where either ground-truth transcript is given or we first estimate the transcript and then use the transcript and the audio to estimate the timestamps with a second pass. For example, WhisperX [ 3 ] improves upon the popular Whisper ASR model [ 18 ] by adding an forced alignment module that refines timestamps at the word level, achieving good accuracy and efficiency. Another line of work [ 23 , 29 ] shows that speech–text alignment can be effectively learned using attention scores between speech and text modalities. Sunder et al. [ 23 ] propose a pointer network supervised by an ASR encoder–decoder model, alleviating the need for a pronunciation dictionary. CrisperWhisper [ 29 ] proposes to leverage cross-attention score of a whisper model with modified tokenizer to learn accurate word-level timestamps. Figure 1: (a) Overall architecture of the proposed In-Sync framework for joint transcription and timestamp prediction using Granite-speech. (b) Diagram of timestamp embedding space regularization with N{=}5 , where the similarity matrix S is encouraged to match a structured target G . (c) Illustration of reduced teacher forcing during autoregressive generation, where a timestamp token is randomly corrupted to encourage model robustness. Compared to traditional alignment approaches, end-to-end approaches that perform one-pass Speech Recognition With Timestamps (SRWT) directly have emerged as a promising alternative [ 9 , 5 ] . These models predict timestamp information alongside transcription, reducing reliance on external alignment tools or complicated model architecture design. However, such methods could introduce trade-offs, as timestamp prediction may compete with recognition accuracy or require architectural modifications that increase complexity. Qwen-Audio [ 5 ] integrates audio understanding with the Qwen LLM, enabling multimodal reasoning and fine-grained timestamp prediction as part of its broader capability set. These approaches highlight the opportunity to elevate timestamp prediction into a first-class objective within speech–language modeling frameworks. Moreover, one additional potential benefit is that the additional supervision from timestamps might foster better performance of automatic speech recognition and reduce hallucination. We pursue this goal within Granite-speech [ 21 ] , a recently proposed Speech-aware LLM with strong ASR performance. We make extensions to the model that enable joint transcription and word-level timestamp prediction (In-Sync), eliminating the need for external aligners or costly post-processing. To stabilize timestamp training, we introduce three novel techniques designed for LLM adaptation: 1. Speech Length Augmentation. Concatenating consecutive utterances [ 11 ] balances the long-tail timestamp distribution and improves coverage of large timestamp tokens. 2. Timestamp Embedding Regularization. An auxiliary loss enforces structured similarity among timestamp embeddings, encouraging monotonic temporal progression. 3. Reduced Teacher Forcing. Randomly corrupting timestamp inputs mitigates over-reliance on ground-truth history, improving robustness in autoregressive generation. Together, these contributions enable effective timestamp prediction within Granite-speech, advancing toward end-to-end speech recognition with temporal grounding. Model Metric AVG LS-C ∗ LS-O ∗ CV ∗ AMI ∗ VOXP ∗ MLS ∗† TIMIT † BUCK † External Baselines CrisperWhisper [ 29 ] 1 1 1 23 CV, 36 VOX, 3 MLS samples excluded from CrisperWhisper evaluation due to decoding error during inference. WER 5.60 1.71 3.72 7.76 8.43 6.03 5.26 1.29 10.63 AAS 53.65 30.20 33.84 119.37 64.80 54.93 48.36 34.30 43.41 Qwen-Audio [ 5 ] WER 10.27 2.19 4.59 10.70 2 2 2 Undergoes a rule-based post-processing for correcting a fixed error pattern before evaluation. 31.82 7.22 7.54 5.96 12.15 AAS – 16.67 18.87 – 55.64 – 31.87 22.25 23.17 MAL – 0.57 0.54 – 18.64 – 4.51 0.06 0 Granite-Speech ASR Baseline and Ablations with In-Sync ASR-only Baseline WER 6.55 1.72 3.68 8.95 9.95 6.31 6.84 2.85 12.09 Mixed Training WER 6.71 1.81 3.82 10.51 10.09 7.02 5.81 3.13 11.51 AAS 41.66 33.75 24.42 68.22 42.76 76.96 39.72 20.23 27.22 MAL 0.14 0.42 0.2 0.01 0 0.49 0 0 0 Mixed Training + Length Aug WER 6.60 1.72 3.65 8.97 10.69 6.05 5.68 4.18 11.84 AAS 41.41 13.37 17.36 56.03 46.17 116.37 35.13 20.48 26.38 MAL 0.02 0 0 0.02 0 0.11 0.03 0 0 Mixed Training +Length Aug + Timestamp Reg WER 6.34 1.62 3.69 9.40 9.79 6.15 5.69 2.53 11.88 AAS 37.23 12.61 16.55 68.70 43.48 73.72 34.72 20.22 27.81 MAL 0.08 0.08 0.17 0 0 0.33 0.03 0 0 Mixed Training +Length Aug + Reduced TF WER 6.47 1.64 3.65 8.89 10.87 5.95