From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition

Tianduo Wang , Lu Xu , Wei Lu , Shanbo Cheng · 2025

Read From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition by Tianduo Wang , Lu Xu , Wei Lu , Shanbo Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16972.txt.

Introduction

Tianduo Wang, Lu Xu, Wei Lu, and Shanbo Cheng present Speech Back-Translation, a way to grow speech data for many languages. Their twenty twenty-five paper starts from a gap: speech recognition needs huge labeled corpora, but most languages have little. Text, by contrast, is plentiful.

Their pipeline turns text into synthetic speech with text-to-speech models. A few tens of hours of real speech can train those models well enough to make hundreds of times more synthetic audio. The team built a test for synthetic quality and clear cutoffs for its use. They made over half a million hours in ten languages and cut errors of a top model by over thirty percent.

Catalog record

CREATORTianduo Wang , Lu Xu , Wei Lu , Shanbo Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16972

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition Tianduo Wang , Lu Xu , Wei Lu , Shanbo Cheng Abstract Recent advances in Automatic Speech Recognition (ASR) have been largely fueled by massive speech corpora. However, extending coverage to diverse languages with limited resources remains a formidable challenge. This paper introduces Speech Back-Translation, a scalable pipeline that improves multilingual ASR models by converting large-scale text corpora into synthetic speech via off-the-shelf text-to-speech (TTS) models. We demonstrate that just tens of hours of real transcribed speech can effectively train TTS models to generate synthetic speech at hundreds of times the original volume while maintaining high quality. To evaluate synthetic speech quality, we develop an intelligibility-based assessment framework and establish clear thresholds for when synthetic data benefits ASR training. Using Speech Back-Translation, we generate more than 500,000 hours of synthetic speech in ten languages and continue pre-training Whisper-large-v3, achieving average transcription error reductions of over 30%. These results highlight the scalability and effectiveness of Speech Back-Translation for enhancing multilingual ASR systems. 1 Introduction Automatic Speech Recognition (ASR) technology has become increasingly important in making digital services accessible across languages and modalities Baevski et al. (2020) ; Zhang et al. (2021) ; Radford et al. (2022) . While recent transformer-based architectures have achieved impressive results for high-resource languages, e.g., English and Chinese, many of the world’s languages still lack sufficient transcribed speech for training robust ASR models Pratap et al. (2020a) ; Babu et al. (2021) ; Chen et al. (2024) . This data scarcity creates a significant barrier to developing effective multilingual speech technologies, particularly affecting communities where manual data collection is resource-intensive or logistically challenging Costa-jussà et al. (2022) ; Communication et al. (2023) ; Pratap et al. (2023) . A natural way to mitigate the data scarcity issue is to leverage high-quality generative models. Recent work has demonstrated successful applications of these models for data augmentation in computer vision Fan et al. (2023) ; Azizi et al. (2023) , natural language processing Gunasekar et al. (2023) ; Li et al. (2024) , and speech recognition Yang et al. (2024) . Despite their demonstrated potential, the role of generative models in overcoming data scarcity presents a paradox. These models themselves typically demand vast amounts of labeled data to attain their remarkable capabilities. For instance, Stable Diffusion Rombach et al. (2022) , a leading text-to-image model frequently used for data augmentation Tian et al. (2023) ; Trabucco et al. (2023) , was trained on millions of labeled images. This reliance prompts a fundamental question: do synthetic data truly alleviate data scarcity in downstream tasks, or do they simply shift the burden of data collection to the pre-training stage of generative models? Figure 1: Pipeline of Speech Back-Translation . The main objective is to augment limited training data ( \leq 100 hours) for low-resource languages by synthesizing extensive amounts of speech ( > 10,000 hours). Starting from a multilingual TTS model pre-trained with high-resource languages, we fine-tune it on a small set of seed data, then generate synthetic speech by conditioning the fine-tuned model on a large textual corpus and diverse audio prompts. Our work investigates whether an off-the-shelf text-to-speech (TTS) model can be trained with limited real transcribed speech data—just tens of hours—to generate synthetic data that enhances multilingual ASR models. To address this challenge, we propose Speech Back-Translation (see Figure 1 ), a scalable method that build large-scale synthetic transcribed speech from text corpora with TTS models. Our results demonstrate that synthetic data, generated by TTS models trained on just tens of hours of labeled audio, can effectively expand small human-labeled datasets to tens of thousands of hours. To assess the quality of this back-translated synthetic dataset, we propose a novel intelligibility-based metric and use it to establish thresholds indicating when synthetic speech reliably enhances ASR performance. Finally, we scale Speech Back-Translation to 500K hours across ten languages and continue pre-training Whisper-large-v3 1 1 1 https://hf.co/openai/whisper-large-v3 , one of the state-of-the-art multilingual ASR models. As a result, we observe consistent improvements across all languages, achieving an average reduction of over 30% in transcription error rates. To summarize, our main contributions are listed as follows: 1. We demonstrate that just tens of hours of real transcribed speech can effectively train TTS models to generate tens of thousands of high-quality synthetic speech, achieving a scaling factor of several hundred. 2. We introduce an intelligibility-based evaluation framework for synthetic speech and establish thresholds to determine when synthetic data reliably benefits ASR performance. 3. We build the largest synthetic speech dataset to date—500K hours spanning ten languages—and use it to further pre-train Whisper-large-v3. This yields an average 30% reduction on transcription error rates, highlighting the scalability of our approach. 2 Background 2.1 Back-Translation Back-translation is a data augmentation technique originally used in machine translation to expand training data Sennrich et al. (2016a) ; Edunov et al. (2018) . In a typical setup, a model trained to translate from the target language back into the source language (i.e., a “ reverse ” model) is used to generate synthetic source sentences from real target-language data. These newly created source-target pairs can then be used to train a forward translation model, effectively increasing its exposure to a broader range of textual content. For speech recognition, back-translation offers a mechanism to supplement scarce or imbalanced datasets by leveraging an abundance of target-side text. Here, the “ reverse ” model is typically a text-to-speech model that generates synthetic speech from textual corpora. Integrating this synthetic speech with existing training data allows the model to handle a wider range of speech variability, enhancing recognition performance despite resource constraints. 2.2 Zero-shot Text-to-Speech Model Zero-shot Text-to-Speech (TTS) models Wang et al. (2023) ; Casanova et al. (2024) represent a milestone in speech synthesis, enabling the generation of high-quality speech for previously unseen speakers without additional fine-tuning. These models typically contain the following components: • Audio Tokenizer : Encodes raw acoustic inputs (e.g., mel-spectrograms) into discrete audio tokens, forming the basis for synthesis. • Speaker Embeddings : Contain speaker-specific acoustic features, which are normally extracted from audio clips, enabling zero-shot adaptation to new voices. • Decoder-only Transformer : Processes speaker embeddings alongside textual tokens to generate sequences of audio tokens. The Transformer model is trained in an auto-regressive manner. • Vocoder : Converts the generated audio tokens into waveform audio, producing the final synthesized output. The synergy of these components allows zero-shot TTS models to generalize effectively to speakers not encountered during training, maintaining high voice similarity and naturalness. 3 Approach: Speech Back-Translation In this section, we introduce the proposed Speech Back-Translation (see Figure 1 ). we first detail how we extend existing TTS models to support new low-resource languages with fine-tuning (Section 3.1 ). We then describe how we generate large-scale synthetic speech dataset (Section 3.2 ). 3.1 Fine-tuning with Low-resource Languages Obtaining high-quality transcribed speech for low-resource languages poses a significant challenge for multilingual ASR training. To address this, we extend existing multilingual TTS models—originally trained on high-resource languages—to new, low-resource languages via targeted fine-tuning with limited data. Vocabulary Expansion Before fine-tuning, we expand the vocabulary of pre-trained TTS models to accommodate words not encountered during the

Read the complete text