Text-only adaptation in LLM-based ASR through text denoising

Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Andrés Carofilis 1 , Shashi Kumar 1,2 , Kadri Hacioglu 3 Srikanth Madikeri 4 , Pradeep Rangappa 1 , Manjunath K E 3 , Petr Motlicek 1,5 , Shankar Venkatesan 3 , Andreas Stolcke 3 · 2026

Read Text-only adaptation in LLM-based ASR through text denoising by Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Andrés Carofilis 1 , Shashi Kumar 1,2 , Kadri Hacioglu 3 Srikanth Madikeri 4 , Pradeep Rangappa 1 , Manjunath K E 3 , Petr Motlicek 1,5 , Shankar Venkatesan 3 , Andreas Stolcke 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20900.txt.

Introduction

Sergio Burdisso and a large team of co-authors adapt speech recognition using text alone. Their 2026 paper studies LLM-based automatic speech recognition in new domains. Text-only adaptation is a significant yet underexplored challenge. New domains offer plenty of text but little transcribed speech. Standard fine-tuning trains the language model on that target text. The risk is real, though.

Such tuning can disrupt the alignment between speech and text. That alignment is learned by the projector that joins the two. Once it degrades, recognition gets worse, not better. The authors turn to text denoising as the adaptation signal. Denoising teaches domain language while respecting the speech link. The method promises domain gains without new audio data. It suits call centers and clinics where text is plentiful but recordings are scarce.

Catalog record

CREATORSergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Andrés Carofilis 1 , Shashi Kumar 1,2 , Kadri Hacioglu 3 Srikanth Madikeri 4 , Pradeep Rangappa 1 , Manjunath K E 3 , Petr Motlicek 1,5 , Shankar Venkatesan 3 , Andreas Stolcke 3
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-only adaptation in LLM-based ASR through text denoising Sergio Burdisso 1 , ⋆ {}^{1,^{\star}} , Esaú Villatoro-Tello 1 , ⋆ {}^{1,^{\star}} , Andrés Carofilis 1 , Shashi Kumar 1,2 , Kadri Hacioglu 3 Srikanth Madikeri 4 , Pradeep Rangappa 1 , Manjunath K E 3 , Petr Motlicek 1,5 , Shankar Venkatesan 3 , Andreas Stolcke 3 Abstract Adapting automatic speech recognition (ASR) systems based on large language models (LLMs) to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on target-domain text often disrupts the critical alignment between speech and text modalities learned by the projector, degrading performance. We introduce a novel text-only adaptation method that emulates the audio projection task by treating it as a text denoising task. Our approach thus trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods. † † address: 1 Idiap Research Institute 2 EPFL 3 Uniphore 4 University of Zurich 5 Brno University of Technology 1 Introduction † † footnotetext: ⋆ Corresponding authors: { sergio.burdisso, esau.villatoro } @idiap.ch Recently, there has been growing interest in integrating speech capabilities into large language models (LLMs) to enable seamless voice interaction and advance voice-driven applications, assistive technologies, and conversational AI more broadly [ 9 , 4 , 16 , 22 , 19 , 13 , 14 , 23 , 11 , 27 , 5 ] . In this context, LLM-based ASR systems have emerged as a practical and computationally efficient alternative, focusing on transcription by leveraging a fixed, manually defined prompt during both training and inference [ 16 , 22 , 26 , 12 , 25 , 7 , 20 , 24 , 1 ] . This fixed-prompt setup ensures consistency between the training objective and inference behavior, making it well-suited for applications where high-accuracy transcription is the primary goal. A key advantage of LLM-based ASR is the ease of combining strong pretrained speech encoders with powerful LLMs through a learnable projection layer, thereby leveraging advances from pretraining in both speech and text modalities. This modular design enables scalable, high-performance transcription without the need for costly instruction tuning. Intuitively, the projection layer learns to map speech representations into the text embedding space of the LLM (i.e., learns a speech-to-text alignment). Once projected, the resulting representation can be interpreted by the LLM as a noisy text, which the model reconstructs into a clean transcription through its inherent denoising capability. Despite these advantages, the training of LLM-based ASR typically relies on large amounts of paired audio–text data, which can limit scalability to new domains. In practice, such resources are often scarce or expensive to collect and transcribe. Moreover, existing studies have indicated that performance may degrade when models are applied to domains that differ from the training data [ 12 ] , highlighting the importance of effective adaptation strategies. Compared to collecting additional audio–text pairs, text-only adaptation offers a more practical alternative given the wide availability of text data. Few studies have explored fine-tuning LLM-based ASR with unpaired text data while preserving cross-modal alignment between the speech projector and the LLM. Fang et al. [ 7 ] proposed using a monitoring metric to maintain alignment, but excessive text-only fine-tuning can still degrade recognition, and mitigation strategies only partially address this issue. Ma et al. [ 15 ] introduced a two-step approach using trainable soft prompts as pseudo audio embeddings: first optimizing domain-specific soft prompts, then performing text adaptation with the soft prompts fixed. While effective, this method requires tuning additional hyperparameters, such as the number, initialization, and placement of soft tokens. To address these challenges, we propose a novel text-only adaptation strategy that fine-tunes the LLM within an LLM-based ASR architecture by means of formulating the problem as a denoising task . Our contributions are as follows: (i) We reformulate text-only adaptation as a denoising task, training the LLM to reconstruct inputs that mimic the outputs of a speech projector in LLM-based ASR architectures. (ii) We propose a lightweight training approach that simply consists of a multi-view noise-driven batching strategy, not requiring additional learnable parameters. (iii) We present a thorough evaluation across two datasets, achieving up to 22.1% relative improvement, surpassing the state-of-the-art. (iv) We release our source code for reproducibility. 1 1 1 https://github.com/idiap/llm-asr-text-adaptation Figure 1: Batch composition used for fine-tuning the LLM during text-only adaptation to a target domain. Here, \sigma and \boldsymbol{\tau} denote the proportions of the batch that are drawn from the source domain ( \mathcal{D}_{src} ) and the target domain ( \mathcal{D}_{tgt} ), respectively. 2 Method LLM-based ASR systems consist of three main components: (i) a pretrained speech encoder (e.g., Whisper [ 17 ] , Hubert [ 10 ] , or WavLM [ 2 ] , etc.) that extracts frame-level acoustic representations, (ii) a learnable projector sp(\cdot) that maps these representations into the LLM input embedding space (e.g., pooling-based [ 4 ] ,CNN-based [ 5 , 18 ] , or linear-based [ 16 , 8 , 12 ] ), 2 2 2 Also known as “speech projector” in the literature, with some referring to it as a “speech adapter” or “connector”. and (iii) a pre-trained LLM (e.g., Llama [ 6 ] , Vicuna [ 3 ] ) that acts as a decoder, and generates final transcripts. Prior work has shown that even a simple projector sp(\cdot) (two linear layers with a nonlinearity) combined with a frozen LLM is sufficient to obtain strong transcription performance [ 16 , 12 , 25 ] . This suggests that the projector learns to convert speech into a sequence of soft tokens that approximate entries in the LLM vocabulary. For example, the projector may map the audio utterance “yes that would be” into embeddings resembling “mmy Z YesssS S SGS that Will B be S S”, as reported in prior work [ 12 , 25 ] . This illustrates how the projector outputs resemble a noisy transcript rather than raw speech features. As a result, the LLM is forced to interpret these inputs as a corrupted or noisy version of the transcript, which it then recovers. We interpret this behavior as evidence that LLM-based ASR can be viewed as a denoising task : the LLM learns to reconstruct the clean transcript from the noisy, text-like sequence produced by the projector. Building on this insight, we propose a novel approach to adapt LLM-based ASR models using only text data by explicitly teaching the LLM to denoise distorted transcripts from the target domain, even when no target-domain audio is available. 2.1 Task Formulation Let \mathcal{D}_{src}=\{(a_{i},t_{i})\} denote the source-domain dataset with paired audio a_{i} and transcript t_{i} , and let \mathcal{D}_{tgt}=\{t_{j}\} denote the target-domain dataset with transcripts only. 3 3 3 It is worth mentioning that in all our experiments, the text-only data consists of (ground truth) transcripts from conversational speech, rather than arbitrary text such as that found on web pages. While standard LLM-based ASR training requires large numbers of (a,t) pairs to achieve high recognition performance, we enable text-only

Read the complete text