Whisper-UT: A Unified Translation Framework for Speech and Text
Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur · 2025
Read Whisper-UT: A Unified Translation Framework for Speech and Text by Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16375.txt.
Introduction
One model that can translate speech, text, or both at once would simplify language tools a great deal. Cihan Xiao and nine co-authors adapt the Whisper model with small add-on parts to form Whisper-UT, a unified translation frame. It can take speech plus source text together, using transcripts as prompts, and it decodes in two stages to lift speech translation quality.
Fine-tuning across modes and tasks improved results without needing costly three-way matched data. The adapters are light, so the frame stays efficient. Though shown on Whisper, the approach should carry to like multitask models.
Catalog record
| CREATOR | Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16375
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Whisper-UT: A Unified Translation Framework for Speech and Text Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur Abstract Encoder-decoder models have achieved remarkable success in speech and text tasks, yet efficiently adapting these models to diverse uni/multi-modal scenarios remains an open challenge. In this paper, we propose Whisper-UT, a unified and efficient framework that leverages lightweight adapters to enable seamless adaptation across tasks, including a multi-modal machine translation (MMT) task that explicitly conditions translation on both speech and source language text inputs. By incorporating ASR hypotheses or ground-truth transcripts as prompts, this approach not only enables the system to process both modalities simultaneously but also enhances speech translation (ST) performance through a 2-stage decoding strategy. We demonstrate our methods using the Whisper model, though in principle they are general and could be applied to similar multitask models. We highlight the effectiveness of cross-modal and cross-task fine-tuning, which improves performance without requiring 3-way parallel data. Our results underscore the flexibility, efficiency, and general applicability of the proposed framework for multi-modal translation. 1 Introduction The task of speech-to-text translation (ST) encompasses converting spoken content from one language to another, aiming to overcome language barriers to communication. Traditionally, the task involves an automatic speech recognition (ASR) module to transcribe spoken words, followed by a machine translation (MT) module to convert the transcribed text into the target language in a cascaded manner Ney 1999 . The recent development of end-to-end neural architectures and large pre-trained models have substantially propelled advancements in downstream speech tasks, via either self-supervised learning (SSL) Baevski et al. 2020 ; Hsu et al. 2021 ; Chen et al. 2022 or fully supervised learning. Among the pre-trained acoustic models, Whisper Radford et al. 2022 , a transformer-based encoder-decoder multi-task model trained with large-scale data in a supervised manner, has exhibited good performance on various ST corpora. However, in real-world scenarios, input modalities and data conditions vary widely. In offline settings, for instance, translating conversational or dialectal speech—characterized by disfluencies, code-switching, and noisy acoustic environments—poses significant challenges to end-to-end models, often resulting in degraded performance. Conversely, scenarios like business meetings or translated media archives frequently provide both source-language speech and (manual or ASR-generated) transcripts. Yet existing systems fail to exploit this multi-modal synergy. To address this, we systematically investigate how multi-task encoder-decoder models—using Whisper as a representative case study—can be efficiently adapted to these heterogeneous scenarios. First, we examine fine-tuning strategies for conventional ST (using 3-way parallel speech-transcript-translation data), speech-to-text tasks (ASR-only data), and MT, while also methods for multi-modal translation where both speech and transcripts are available. Our analysis reveals two key insights: • Cross-task training induces synergistic benefits —fine-tuning on in-domain ASR data improves ST performance, while ST training conversely enhances ASR accuracy, suggesting mutual reinforcement between the ASR and ST tasks even without 3-way parallel data; • Multi-modal inputs (speech + text) consistently enhance translation quality when fused , even with imperfect ASR transcripts. Building on these findings, we propose Whisper for U nified T ranslation 1 1 1 We open source our code at https://github.com/BorrisonXiao/Whisper-UT . , or Whisper-UT , a framework that transforms Whisper’s decoder into a unified conditional generation model, capable of dynamically conditioning on speech, text, or both modalities. The framework repurposes Whisper’s encoder-decoder architecture as a versatile multi-modal interface through two innovations: 1 A multi-task learning paradigm with a stochastic task-selection mechanism to adapt the system across ASR, MT, ST, and multimodal translation tasks using a single set of LoRA parameters; 2 A two-stage decoding strategy , where the decoder first generates an ASR transcript from speech, then reuses it as context for translation, perhaps emulating human thought processes, even when a transcript is not provided. Crucially, Whisper-UT requires no architectural modifications—only fine-tuning—ensuring compatibility with any encoder-decoder model. Experiments on CoVoST2’s Wang et al. 2020b French-English ( fr-en ) and German-English ( de-en ) subsets demonstrate strong performance. Extended evaluations on conversational telephone speech (CTS) corpora—Fisher-CallHome Spanish Post et al. 2013 , and BBN Mandarin-English Wotherspoon et al. 2024 further confirm the robustness of our approach across diverse domains. Notably, Whisper-UT outperforms the 1.3B-parameter NLLB model in multi-modal settings (speech + ground-truth text) and achieves superior speech-only translation via hypothesis prompting. Our work highlights the untapped potential of multi-task models in adaptive translation systems. By unifying modality handling and enabling efficient task specialization, Whisper-UT bridges the gap between rigid single-modality systems and the dynamic needs of real-world applications. 2 Related Work 2.1 Whisper Whisper is an end-to-end multi-task speech model that adopts a transformer-like encoder-decoder architecture. Its large-v2 version is pre-trained on 680,000 hours of speech data with multiple supervision. As with the original transformer model Vaswani et al. 2023 , the loss function Whisper used at its pre-training time is the cross-entropy objective for all tasks. Whisper’s decoder supports a prompting mechanism, originally designed for better capturing long-range dependencies of the transcripts/translations to resolve local audio ambiguities. Particularly, long utterances are segmented into chunks and the decoder generates its hypothesis for the current segment conditioning on the previous segment’s transcripts. Inspired by the effectiveness of GPT-like decoder-only models in machine translation, we hypothesize that Whisper’s decoder, which may be viewed as an audio-conditional language model, is also capable of performing audio-augmented text generation conditioning on both inputs . Our work extends recent work showing that the Whisper can be adapted via fine-tuning to perform a number of novel tasks including, audio-visual speech recognition Rouditchenko et al. 2024 , target-speaker ASR Guo et al. 2024 ; Polok et al. 2024 ; Ma et al. 2024a , translation to non-English languages Peng et al. 2023 , by showing that Whisper can be extended to enable multi-modal translation, i.e., using either only text or both text and speech inputs simultaneously. 2.2 Multi-modal/-task Speech Systems Recent developments in multi-modal and multi-task systems, e.g., Tang et al. 2021 , are exploring new ways to combine audio and text to improve various language-related tasks. mSLAM Bapna et al. 2022 , a multilingual speech and language model, has emerged as a pioneering approach. It aims to construct a shared representation space for both speech and text through joint pre-training on both self-supervised and supervised tasks with various loss objectives, including translation language modeling (TLM) loss for ST. SeamlessM4T Communication et al. 2023 is another innovative model that further refines the integration of multi-modal inputs for speech and text translation tasks. As a single model designed for ASR, T2T translation, T2S translation, S2T translation and S2S translation, it consists of multiple building blocks to leverage uni-modal data, including a w2v-BERT Chung et al. 2021 as the speech encoder, a 1.3B NLLB model Team et al. 2022 as the text encoder and decoder, a transformer-based text-to-unit encoder-decoder model for speech, with a vocoder for converting the unit-sequences to waveforms. These systems, along with most existing methods, primarily seek to simply align the representations of the text and speech modalities, limiting the model to still accept only one input modality at a time during inference, which prevents exploitation of cross-modal cues . More recently, speech - centric large language models such as QWen - Audio Chu et al. 2024 have shown