Text-Utilization for Encoder-dominated Speech Recognition Models

Albert, Tim, Ralf, Hermann · 2026

Read Text-Utilization for Encoder-dominated Speech Recognition Models by Albert, Tim, Ralf, Hermann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26514.txt.

Introduction

Zeyer and three colleagues compared ways to feed text-only data into encoder-heavy speech recognition models. Their study weighs modality matching and dynamic downsampling for reaching text-level representations inside the encoder. On LibriSpeech, a bigger encoder with a smaller decoder matched or beat architectures with bigger decoders. Simple setups, such as random duration models, often beat complex rivals, simplifying training. All code and recipes are public.

Catalog record

CREATORAlbert, Tim, Ralf, Hermann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23971
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26514

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-Utilization for Encoder-dominated Speech Recognition Models Albert, Tim, Ralf, Hermann Abstract This paper investigates efficient methods for utilizing text-only data to improve speech recognition, focusing on encoder-dominated models that facilitate faster recognition. We provide a comprehensive comparison of techniques to integrate text-only data, including modality matching and dynamic downsampling to reach text-level representations within the encoder. Our experiments on the LibriSpeech corpus show that a larger encoder with a smaller decoder can equal or surpass the performance of architectures with larger decoders. We demonstrate that simple configurations, such as random duration models, are often more effective than complex alternatives, significantly simplifying the training pipeline. All code and recipes are made publicly available. Zeyer Posielek Schlüter Ney keywords Speech Recognition, Text Utilization, Encoder-dominated Models, Dynamic Downsampling † † address: 1 Machine Learning and Human Language Technology Group, RWTH Aachen University, Germany 2 AppTek, Germany † † email: {zeyer,schlueter,ney}@cs.rwth-aachen.de, [email protected] 1 Introduction Figure 1: Speech recognition using denoising LM [ 11 , 12 ] . Figure 2: Model architecture including the training part to utilize the text-only data, which is not used during recognition. It remains an open question how to best utilize the large amount of text-only data for improving ASR, in addition to the paired audio-text data. A standard approach is to train a separate language model (LM) on the text-only data, and then use shallow fusion to utilize the LM during decoding. Using text-to-speech (TTS) to convert the text-only data into synthetic speech and then train on that together with the paired audio-text data is another approach, which has been shown to be very effective [ 15 , 11 , 12 ] . We summarize these baselines in Table 1 . These results lead to the main motivation of this paper, that a smaller decoder with a larger encoder is just as good or even better than a large decoder with a smaller encoder. Using TTS to utilize the text-only data directly during the ASR training is a very strong baseline, outperforming the separate LM training and shallow fusion approach. Using both TTS + LM together still gains further. Replacing the LM by a denoising language model (DLM) [ 11 , 12 ] still results in small further improvements. However, the interesting observation is that the CTC + DLM has a smaller decoder and a larger encoder than the CTC + LM, which is more efficient for recognition. Similarly, a simple CTC + AED model with a smaller decoder and a larger encoder results in almost the same performance as the CTC + DLM, even though the training and recognition is much simpler, just utilizing TTS for the text-only data. We note that the DLM is an AED model operating on text, on top of the CTC output. Thus the joint CTC + DLM model is like an extended encoder with downsampling in between to get to text-level representation, then another text-level encoder on top of that, which utilized the text data, and a decoder on top of that, as shown in Figure 1 . It was shown in [ 12 ] that the decoder can be very small in relation to the encoder, which is what we also argue here. Note that the TTS data utilized the text data on first part of the encoder and also the DLM. These results show that a larger encoder with a smaller decoder is just as good or even better than a smaller encoder with a larger decoder. With a larger encoder, it is crucial that the text data is utilized in the encoder as well, not only in the decoder. Here we explore different methods that avoid the TTS approach, which is computationally expensive and also requires a separate TTS model to be trained. The CTC + AED training on TTS data, while very simple and leading to very good results, is very expensive to train compared to the CTC on the paired audio-text data only and separate LM training on the text-only data, since the TTS representation of the text data is much larger than the text representation of the text data. Thus, our goals: • We want a simple training and recognition approach that can utilize the text-only data effectively. Thus, we want to avoid relying on a TTS model. The DLM approach also requires the TTS. • We want to improve the efficiency of training while utilizing the text-only data. While the CTC + AED training on TTS data is simple (despite the TTS), it is very expensive to train. • We want to have models that are efficient in recognition, which means encoder-dominated models with a small decoder. Our contributions are as follows: • We point out that an encoder-dominated model (larger encoder with a smaller decoder) is just as good or even better than a decoder-dominated model (smaller encoder with a larger decoder) when utilizing the text-only data ( Table 1 ). • For encoder-dominated models, it is crucial to utilize the text-only data in the encoder as well ( Table 3 ). • We study different dynamic downsampling variants in the middle of the encoder to get to text-level representation, to allow for more efficient training, and easier utilization of the text-only data in the upper text-level part of the encoder. • We compare various approaches to utilize the text-only data in the encoder and various variants of a pseudo-speech-encoder. This provides a unified comparison of existing text-utilization strategies such as modality matching (similar to MAESTRO [ 5 ] , CJST [ 23 ] ) and direct encoder training (similar to JOIST [ 16 ] , SpeechUT [ 22 ] , SpeechT5 [ 2 ] ) under a single, controlled framework. 2 Related Work A lot of work is on utilizing a pre-trained language model (LM) in the decoder, or even a large LM (LLM). The (L)LM is extended to integrate the encoder, usually by prepending the encoder output adapted into the LM dimension, known as prefix-LM [ 21 ] , and then finetuned on paired audio-text data. Text-utilization in encoder has also been explored in various ways, such as modality matching [ 6 , 5 , 20 , 23 , 10 ] , direct utilization of the text data as input to the encoder [ 4 , 3 , 18 , 19 , 2 , 22 , 16 , 17 ] or TTS [ 15 , 7 , 11 , 12 ] . Downsampling to text-level representation in the middle of the encoder via CTC compression has also been explored [ 9 , 1 , 24 ] . 3 Model The model architecture is shown in Figure 2 . Speech-only encoder. We perform log-mel feature extraction on the input audio, then a convolutional front-end to downsample the features by a factor of 6 in time, and then our speech encoder which is a stack of Conformer blocks. Downsampling to text-level. After the speech encoder, we optionally perform dynamic downsampling via a variant of CTC compression [ 8 , 23 ] , utilizing an intermediate CTC output to determine which frames to merge together, to get a text-level-like representation. \displaystyle\hat{y}_{t} \displaystyle=\operatorname{argmax}_{y}p_{\textrm{iCTC}}(y\mid h_{t}), \displaystyle m_{t} \displaystyle=(\hat{y}_{t}=\hat{y}_{t-1}) (merge mask) \displaystyle\quad\wedge(p_{\textrm{iCTC}}(\hat{y}_{t}\mid h_{t})\geq\tau) \displaystyle\quad\wedge(p_{\textrm{iCTC}}(\hat{y}_{t-1}\mid h_{t-1})\geq\tau), \displaystyle i_{t} \displaystyle=\begin{cases}1&\text{if

Read the complete text