Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025
Dominik Macháček, Peter Polák · 2025
Read Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025 by Dominik Macháček, Peter Polák (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17077.txt.
Introduction
Dominik Macháček and Peter Polák describe a live speech translation system in a 2025 conference paper. Their Charles University team entered a shared contest task on speech translation that happens as the speaker talks. Four language pairs were covered, some by direct speech-to-text translation and some by a two-step chain. A speech model built for offline work was adapted to run live with a strong timing policy.
Prompt tricks fed the system field terms and prior context to steady its output. A second model handled one pair with no length cap. Scores beat the organizers' baseline on every pair, with the largest gains on three of them. The team also proposes a sharper way to measure speech recognition delay.
Catalog record
| CREATOR | Dominik Macháček, Peter Polák |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025 Dominik Macháček, Peter Polák Abstract This paper describes Charles University submission to the Simultaneous Speech Translation Task of the IWSLT 2025. We cover all four language pairs with a direct or cascade approach. The backbone of our systems is the offline Whisper speech model, which we use for both translation and transcription in simultaneous mode with the state-of-the-art simultaneous policy AlignAtt. We further improve the performance by prompting to inject in-domain terminology, and we accommodate context. Our cascaded systems further use EuroLLM for unbounded simultaneous translation. Compared to the Organizers’ baseline, our systems improve by 2 BLEU points on Czech to English and 13-22 BLEU points on English to German, Chinese and Japanese on the development sets. Additionally, we also propose a new enhanced measure of speech recognition latency. 1 Introduction In this paper, we describe the submission of the Charles University (CUNI) system to IWSLT 2025 Simultaneous Speech Translation Task Abdulmumin et al. 2025 . Our system is built on top of Whisper Radford et al. 2022 with AlignAtt Papi et al. 2023 simultaneous policy. To achieve higher translation quality, we apply beam search and prompting for in-domain terminology. In our end-to-end system for the Czech-to-English translation, we also exploit previous translations as a context. For the translation into German, Chinese, and Japanese, we adopted a cascaded approach consisting of Whisper for English ASR and EuroLLM Martins et al. 2025 for translation. We validate our systems’ latency in computationally unaware simulation. Our Czech-to-English systems work both in 2-second and 4-second latency regimes required by IWSLT 2025 (“low” and “high”). The English-to-German, Chinese and Japanese systems are available only in the high-latency regime of 4-5 seconds. For an overview of our systems, see Table 1 . Our main goal in this submission is to create a robust and straightforward implementation that can be used in further research as well as in many realistic use cases. We name the implementation SimulStreaming and publish it at https://github.com/ufal/SimulStreaming . Among the strengths of our submitted system is very high quality, because of using high-performing foundation models, and very high multilinguality. Whisper allows direct translation from 99 speech source languages to English, and EuroLLM allows English translation into 35 languages. Our systems are also adaptable; the prompts and in-context learning allow injecting specific in-domain terminology. Moreover, although we primarily focus on computationally unaware latency, our system is practically usable in real time only with feasible hardware resources. It requires hosting Whisper large 1.6B parameters model and and the EuroLLM 9B parameters model. Our second goal is to evaluate the state-of-the-art methods in combination. Our results show improvements by 2 BLEU points on Czech to English over the organizers baseline, and 13-22 BLEU points on English to German, Chinese, and Japanese, which highlights the effectiveness of our systems. We conclude we have reached both goals. The original contributions of this work is the SimulStreaming implementation and evaluation, and also a new enhanced method for measuring ASR latency using Continuous Levenshtein Alignment (see Section 5.1 ). Cs-En En-{De,Zh,Ja} Speech-to-text model Whisper large-v3 Whisper large-v3 task translate to En transcribe En beam yes no prompt yes no context yes no simult. policy AlignAtt AlignAtt + Text-to-text model - EuroLLM-9B-Instruct prompt - yes context - yes simult. policy - LocalAgreement Latency regime 2 seconds (“low”) yes no 4-5 seconds (“high”) yes yes Table 1: Overview of CUNI systems submissions to IWSLT 2025 Simultaneous Speech Translation Task. 2 Background Whisper is among the top-performing ASR and speech translation models for 99 languages. It has the ability to use initial and context prompts, which makes it adaptable for in-domain terminology. Works such as Macháček et al. 2023b and Wang et al. 2024 show that Whisper is adaptable to simultaneous mode, although it is primarily designed for offline mode. Whisper is available in multiple model versions that differ in size and quality. We use the large-v3 model, which achieves the highest quality. AlignAtt Papi et al. 2023 is a simultaneous policy. Given an offline translation model, partial source and previous target, it detects where to stop generating the partial target, which is when the most attended source frame by the decoder is behind a threshold. Papi et al. 2023 shows that this policy outperforms all previously proposed policies. Wang et al. 2024 later showed that AlignAtt also works with Whisper. LocalAgreement Polák et al. 2022 ; Polák et al. 2023 is a simultaneous policy that considers the target prefixes of two subsequent updates, each processing a newly incoming source chunk. It emits their longest common prefix as confirmed and uses it in forced decoding of the latter chunks. Simul-Whisper Wang et al. 2024 is an implementation of the simultaneous mode with Whisper using AlignAtt. It is an extension of the original OpenAI Whisper inference using the Torch deep learning framework. Simul-Whisper supports only ASR of speech that is segmented into individual sentences, and computationally unaware simulation, while the IWSLT 2025 Simultaneous Task focuses on a more realistic case of unbounded speech Papi et al. 2025 without any explicit sentence boundaries. On the other hand, Whisper-Streaming Macháček et al. 2023b is our implementation of Whisper with the LocalAgreement simultaneous policy and reprocessing the audio buffer from the beginning with every incoming source chunk, which is less computationally effective than AlignAtt. On the other hand, Whisper-Streaming supports both computationally aware and unaware simulations, as well as unbounded speech. It integrates Silero VAD Team 2024 that incrementally detects silence and non-voice sounds vs. voice. EuroLLM Martins et al. 2025 is a recent large language model for text-to-text translation between 35 EU and non-EU languages, including English, German, Japanese, and Chinese. It is a decoder-only model of the LLaMA family. We use its 9B parameter version with instruction tuning. It supports a system prompt, which can be used to suggest the domain, and a maximum context of 4096 tokens, which spans over 10 minutes of English source and German target of ACL 6060 Salesky et al. 2023 dev set reference (see estimation in Appendix A ). We use EuroLLM with the fast inference framework CTranslate2. It enables fast computation and efficient memory usage; however, it currently does not provide access to attention weights. Therefore, we can not apply the AlignAtt policy to EuroLLM, so we use it with the LocalAgreement simultaneous policy, which is the best-performing policy that does not require attention weights. 3 Direct Simultaneous Speech-to-Text with Whisper and AlignAtt Let us describe the process of simultaneous speech-to-text processing, which we apply to direct Czech-to-English translation and to English ASR in the cascaded system for English to German, Chinese, and Japanese. Our system uses our implementation called SimulStreaming which merges the Simul-Whisper AlignAtt policy with the Whisper-Streaming interface to support unbounded speech processing. Moreover, we extend the original Simul-Whisper with the following enhancements. First, we added support for the Whisper large-v3 model. Second, in addition to transcription, we enabled translation. Then, to improve quality, we implement beam search decoding. Finally, we incorporate support for initial prompts and contextual information from the preceding audio buffers. Our simultaneous speech-to-text pipeline consists of the prototypical processing steps that are described in Papi et