Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MT
Zeeshan Ahmed, Frank Seide, Niko Moritz, Ju Lin, Ruiming Xie, Simone Merello, Zhe Liu and Christian Fuegen · 2025
Read Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MT by Zeeshan Ahmed, Frank Seide, Niko Moritz, Ju Lin, Ruiming Xie, Simone Merello, Zhe Liu and Christian Fuegen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13358.txt.
Introduction
Zeeshan Ahmed and co-authors attacked the lag problem in phone based speech translation in 2025, where a device must hear one language and speak another almost at once. Splitting the job into a listener and a translator is natural, but naive handoffs pile up delay.
Their streaming translator starts rendering before the speaker finishes, keeping pace by aligning partial text. Cues from the speech recognizer help the translator hold context, and search shortcuts such as timeouts keep the whole chain real time. On a two language conversation setup on device, the system beat baselines on both speed and quality. It nearly matched systems allowed to hear the whole sentence first.
Catalog record
| CREATOR | Zeeshan Ahmed, Frank Seide, Niko Moritz, Ju Lin, Ruiming Xie, Simone Merello, Zhe Liu and Christian Fuegen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Overcoming Latency Bottlenecks in On-Device Speech Translation: A Cascaded Approach with Alignment-Based Streaming MT Zeeshan Ahmed, Frank Seide, Niko Moritz, Ju Lin, Ruiming Xie, Simone Merello, Zhe Liu and Christian Fuegen Abstract This paper tackles several challenges that arise when integrating Automatic Speech Recognition (ASR) and Machine Translation (MT) for real-time, on-device streaming speech translation. Although state-of-the-art ASR systems based on Recurrent Neural Network Transducers (RNN-T) can perform real-time transcription, achieving streaming translation in real-time remains a significant challenge. To address this issue, we propose a simultaneous translation approach that effectively balances translation quality and latency. We also investigate efficient integration of ASR and MT, leveraging linguistic cues generated by the ASR system to manage context and utilizing efficient beam-search pruning techniques such as time-out and forced finalization to maintain system’s real-time factor. We apply our approach to an on-device bilingual conversational speech translation and demonstrate that our techniques outperform baselines in terms of latency and quality. Notably, our technique narrows the quality gap with non-streaming translation systems, paving the way for more accurate and efficient real-time speech translation. I Introduction The traditional approach to speech translation involves a pipeline that combines an Automatic Speech Recognition (ASR) system with a Machine Translation (MT) system. However, current state-of-the-art cascade-based Speech Translation (ST) systems are for offline use, where the entire audio input is available in advance. In contrast, streaming setups require real-time processing of unbounded audio inputs, which poses significant challenges for cascade-based ST systems [ 1 , 2 ] . Conventional MT systems are the bottleneck for the real-time streaming translation set-up, as they are typically trained on sentence-aligned pairs of text. This creates a mismatch between the training data, which consists of short sentences with only a few hundred tokens, and the inference conditions (in a streaming environment), where thousands of tokens may be processed in live sessions. To address this issue, segmentation models are often used to divide the incoming text stream into manageable chunks, or segments, that can be translated independently by the MT system. However, translating the segment in isolation poses a significant risk to translation quality [ 3 , 4 , 5 ] . Simultaneous Machine Translation (SMT) [ 6 , 7 , 8 , 9 , 10 ] , on the other hand, is better suited for streaming speech translation scenarios. The SMT initiates the translation process while still receiving the input. Conventional MT relies on an encoder-decoder architecture. The decoder uses an attention module that attends to all encoder embeddings. For SMT, both the encoder and attention module must be adaptive, allowing the encoder to operate incrementally and the cross-attention in the decoder to handle variable-length encoder output sequences. In this setup, the SMT model is fed one token at a time, and at each decoding step, it decides whether it wants to wait for another token or generate a translation. The SMT systems face a trade-off between quality and latency. Prioritizing higher quality increases latency (with conventional non-streaming MT systems still representing the upper bound in quality), while reducing latency can impact translation quality. Therefore, SMT systems must carefully balance this trade-off for practical applications. Another challenge for SMT is that, despite its ability to translate input in real-time, it still requires frequent context flushing to effectively manage context and resources, particularly in on-device scenarios. In this paper, we propose a couple of advances in cascade-based speech translation. 1) We present a novel approach to SMT called ALIgnment BAsed STReaming Machine Translation (AliBaStr-MT) which efficiently balances the quality/latency trade-off in on-device streaming speech translation. 2) We develop a robust multi-channel multi-talker ASR model that can recognize multiple languages, distinguish between different speakers and suppress side-talk/background noise. 3) Despite operating in a streaming fashion, the SMT architecture needs to keep all the previous encoder/decoder states saved in memory (kv-cached) for translation of future input. When used in an on-device setting, the memory is limited and the context needs frequent flushing. We propose incorporating segmentation cues in ASR to detect the segment boundaries and flush the states based on detected segments. II Background Fig. 1: On-device Streaming Speech Translation Architecture using Cascade of ASR and MT. The AliBaStr-MT system processes input segments of varying sizes and dynamically decides whether to generate a translation or await additional input, guided by its read/write policy module. Cascade-based and end-to-end are two major paradigms for speech translation. Cascade-based systems use separate ASR and MT models [ 1 , 2 ] , while end-to-end systems translate directly from source speech to target text using a single model [ 11 ] . End-to-end systems offer advantages in terms of latency and model size, while cascade-based systems still demonstrate superior translation accuracy in many cases with well trained ASR and MT models. Furthermore, the decoupling of ASR and MT modules also allows independent training and optimization that favors low resource language pairs [ 12 ] for which bilingual end-to-end audio training data are scarce. Incorporating streaming capabilities into these systems is a challenging task that has received increasing attention in recent years. Although end-to-end models [ 13 , 14 , 15 , 16 ] are more favorable for streaming speech translation, cascade-based models [ 17 ] face considerable challenges. For instance, accurately identifying the correct segment of the ASR transcript for translation is a challenge. ASR systems often segment audio based on acoustic features such as pauses, which may not correspond to natural sentence boundaries [ 18 ] . This mismatch can lead to segments that are too long (spanning multiple sentences) or too short (containing only fragments), negatively affecting the accuracy of subsequent MT. To overcome this problem, researchers have explored re-segmenting ASR transcripts using punctuation restoration and language models to improve sentence boundaries and enhance MT performance [ 18 , 19 ] . In this work, we incorporate the prediction of punctuation directly into the RNN-T model and employ punctuation for segment boundary detection (mostly sentence) when context flushing is needed. Another challenge arises when attempting to translate these segments in real-time. Conventional MT systems are designed to translate complete sentences, and are not well-suited for translating smaller segments, making them impractical for real-time streaming translation. Simultaneous MT is particularly beneficial for the real-time speech translation scenario. SMT systems can be broadly classified into fixed policy and adaptive policy systems. In the fixed policy system, the model uses the predefined heuristics and rules to generate the translation. These heuristics remain constant throughout the translation. Fixed policy systems such as wait- k or prefix-to-prefix translation systems by [ 20 , 21 ] are simple and yet effective methods for streaming translation. However, the value of k in wait- k is not a learnable parameter of the model and is set beforehand, making it difficult to adjust based on the input and so far generated translation. The k is also language dependent for different translation directions, making it difficult to build a multilingual streaming translation model with wait- k . [ 22 ] on the other hand, tries to fix the quality gap in simultaneous translation models by introducing a look-ahead feature and the curriculum learning during training. The adaptive methods, on the other hand, dynamically adjust the read/write decision based