REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation
Nameer Hirschkind, Joseph Liu, Xiao Yu, Mahesh Kumar Nandwana · 2025
Read REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation by Nameer Hirschkind, Joseph Liu, Xiao Yu, Mahesh Kumar Nandwana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04946.txt.
Introduction
Live speech translation must decide, second by second, whether to speak now or wait for more words. Nameer Hirschkind and co-authors train that judgment with a new loss in this 2025 paper, guided by a simple rule: wait only when waiting brings new information. Their method, REINA, measures how much a model's uncertainty drops as audio arrives.
REINA reshapes training of a streaming system built on a standard translation model. Models trained this way set new best marks for their size on French, Spanish, and German speech, in both directions with English, using only open or synthetic data. A new efficiency score shows the method improves the speed-quality trade-off by up to 21 percent over earlier work.
Catalog record
| CREATOR | Nameer Hirschkind, Joseph Liu, Xiao Yu, Mahesh Kumar Nandwana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04946
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
REINA: Regularized Entropy Information-Based Loss for Efficient Simultaneous Speech Translation Nameer Hirschkind, Joseph Liu, Xiao Yu, Mahesh Kumar Nandwana Abstract Simultaneous Speech Translation (SimulST) systems stream in audio while simultaneously emitting translated text or speech. Such systems face the significant challenge of balancing translation quality and latency. We introduce a strategy to optimize this tradeoff: wait for more input only if you gain information by doing so. Based on this strategy, we present Regularized Entropy INformation Adaptation (REINA), a novel loss to train an adaptive policy using an existing non-streaming translation model. We derive REINA from information theory principles and show that REINA helps push the reported Pareto frontier of the latency/quality tradeoff over prior works. Utilizing REINA, we train a SimulST model on French, Spanish and German, both from and into English. Training on only open source or synthetically generated data, we achieve state-of-the-art (SOTA) streaming results for models of comparable size. We also introduce a metric for streaming efficiency, quantitatively showing REINA improves the latency/quality trade-off by as much as 21% compared to prior approaches, normalized against non-streaming baseline BLEU scores. 1 Introduction Simultaneous Speech Translation (SimulST) involves real-time translation of speech in one language into text in another. This extends the simpler speech-to-text-translation (S2TT) task, which involves translation with the full context of an entire speech clip. While S2TT allows for offline applications, conversational environments such as voice or video chat necessitate SimulST models to facilitate real-time communication across language barriers. Recently, End-to-end (E2E) S2TT models have largely superseded traditional cascaded approaches, which link separate Automatic Speech Recognition (ASR) and Machine Translation (MT) systems. E2E models mitigate error propagation and reduce latency by directly mapping source speech to target text ( Communication et al. 2023 ; Peng et al. 2024 ; Radford et al. 2023 ; Puvvada et al. 2024 ) . Rather than training SimulST models from scratch, most works take advantage of advances in S2TT research by adapting a non-streaming S2TT model into a SimulST model ( Communication et al. 2023 ; Chen et al. 2024 ; Zhang et al. 2024 ; Papi et al. 2024 ) . To transition from non-streaming S2TT to SimulST introduces the challenge of balancing translation quality and latency. This requires a policy to decide whether to wait for more input (READ) or generate output (WRITE) ( Gu et al. 2017 ) . This problem is particularly difficult in the translation setting because different languages can have different word orderings, requiring differing amounts of context before a suitable translation can occur. Many approaches have been formulated to determine this READ/WRITE policy, from baking the policy into the model architecture itself via monotonic attention mechanisms ( Arivazhagan et al. 2019 ; Ma et al. 2020b ; Communication et al. 2023 ) , or having a separate module to dictate the policy ( Chen et al. 2024 ) . However, these existing approaches suffer from issues including poor translation quality compared to non-streaming models and expensive, numerically unstable training ( Communication et al. 2023 ; Chen et al. 2024 ) . In this paper, we address the problem of efficiently training high quality SimulST models. The major contributions of this paper can be summarized as follows: New Policy Training Technique. We propose Regularized Entropy INformation Adaptation (REINA), a new technique for policy training that can efficiently convert non-streaming Speech-to-Text Translation (S2TT) models into simultaneous S2TT (SimulST) models. REINA is guided by an approximation of mutual information derived from the S2TT model’s log probabilities on partial versus full audio and is shown to produce higher quality policies than existing methods. Trained with open-source data. We train an E2E S2TT model with REINA on 130k hours of open-source data. Based on empirical studies, this model achieves SOTA streaming translation performance. Streaming efficiency evaluation metric. We propose a new evaluation metric to better compare SimulST models. This metric normalizes the streaming translation quality against the performance of the underlying non-streaming model, allowing for a fairer assessment of the capabilities of the streaming policy itself. 2 Related Work While learning adaptive policies for SimulST is a fairly new research area, it builds on a rich body of non-streaming S2TT work. In this section, we outline the S2TT foundation for SimulST and then move into a discussion of SimulST policies. Training Speech to Text Translation Models The literature around S2TT contains many large-scale, powerful models including Whisper ( Radford et al. 2023 ) , SeamlessM4T ( Communication et al. 2023 ) , Canary ( Puvvada et al. 2024 ) , and the Open Whisper-style Speech Model (OWSM) ( Peng et al. 2024 ) . These models vary in architecture (e.g., Whisper’s Transformer, OWSMv3.1’s E-Branchformer) and training data scale, ranging from Canary’s 86k hours (leveraging pseudo-labels) and OWSM’s 180k hours of public data to Whisper’s 680k hours of web data and SeamlessM4T’s 600k hours of synthetically aligned data. Due to the relative scarcity of parallel ST data compared to ASR or MT corpora, multi-task learning (MTL) is widely adopted ( Ye, Wang, and Li 2022 ; Chen et al. 2024 ; Communication et al. 2023 ) . Auxiliary tasks like ASR and neural machine translation (NMT) are jointly trained with S2TT to improve representations and leverage abundant text or speech data. Furthermore, contrastive learning techniques, such as in ConST ( Ye, Wang, and Li 2022 ) , are used to explicitly bridge the modality gap between speech and text representations by encouraging similarity between corresponding speech segments and their transcriptions. There is a notable gap in the literature between industry work leveraging massive proprietary datasets and less resourced research making heavy use of MTL to get the most out of smaller data scales. As OWSM ( Peng et al. 2024 ) bridges this gap for the non-streaming setting, we aim to do the same for SimulST. We are one of the first SimulST works to leverage large-scale open source data that we train on with an MTL framework including MT and ASR tasks. Streaming Policy Learning Transitioning S2TT models to SimulST introduces the challenge of learning a READ/WRITE policy that balances translation quality and latency. Fixed policies like wait-k are simple to implement but are usually suboptimal due to the mismatch between the sampling rate of the input audio frames and the frequency of outputted words ( Ma, Pino, and Koehn 2020 ) . That being said, recent works like SimulS2S-LLM ( Deng et al. 2025a ) have used wait-k with some success. Adaptive polices based on heuristics such as attention matrix weights as seen in EdAtt ( Papi, Negri, and Turchi 2023 ) also been proposed, which are usually better. On the other hand, adaptive learnable policies dynamically adjust decisions based on context. Prior works have integrated the policy within the model architecture, such as Transducer models ( Graves 2012 ; Xue et al. 2022 ) , which inherently support streaming via monotonic alignment, or models using monotonic attention mechanisms like MMA ( Arivazhagan et al. 2019 ) or EMMA ( Communication et al. 2023 ) . Monotonic alignment methods afford greater expressivity, but they tend to be excessively expensive to compute at train time and suffer from both poor numerical stability and difficulty in converging. Our