A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026

Aziz Sharipov Ortega, Dominik Macháček · 2026

Read A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026 by Aziz Sharipov Ortega, Dominik Macháček (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03948.txt.

Introduction

Sharipov and Macháček built a small system for live speech translation. It formed the CUNI entry to the IWSLT 2026 shared task. The entry covered Czech to English and English to German and Italian. They add live skill to Canary, an offline speech-to-text model. The AlignAtt policy guides when to emit each translated piece.

Small size is the core appeal here. The model holds just one billion parameters. It still beat like-sized rivals in simulated tests, the authors report. That held in both low-latency and high-latency runs. The system is also broad in language scope. It supports 25 source languages and 25 target languages.

Catalog record

CREATORAziz Sharipov Ortega, Dominik Macháček
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03948

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026 Aziz Sharipov Ortega, Dominik Macháček Abstract We implement simultaneous translation capability with the offline direct speech-to-text translation model Canary, using the state-of-the-art policy AlignAtt, and submit it to IWSLT 2026 Simultaneous Speech Translation Shared task for Czech to English and English to German and Italian. The strengths of our system are: (1) high translation quality, outperforming similarly sized baselines both in low- and high-latency regimes in computationally unaware simulations; (2) low computational requirements, as the model has only 1B parameters; (3) multilinguality – support of 25 source and 25 target languages. 1 Introduction In this paper, we describe a submission of Charles University (CUNI) to the IWSLT 2026 ( 2 ) Simultaneous Speech Translation Task 3 . Our system is built on top of Canary-1B-v2 1 1 1 Throughout this paper, we use a simple term Canary to refer to the Canary-1B-v2 model, although technically Canary is a family of models. ( 24 ) with the state-of-the-art AlignAtt ( 18 ) simultaneous policy. Following SimulStreaming 13 , the top-performing system in IWSLT 2025 1 that used the same approach with Whisper model 23 , we also use Silero Voice Activity Detection (VAD, 27 ) to reduce noise and hallucination. We validate the latency of our systems in computationally unaware simulation, and also submit a version for computationally aware simulation. One of the strengths of our system is high multilinguality. The Canary model supports 25 source and 25 target languages for direct translation, which overlaps with three language pairs of IWSLT 2026 that we focus on: Czech to English, English to German, and English to Italian. Additionally, the model has only 1B parameters, making it a strong candidate for pocket-device deployment. Recent work has shown that ASR models of similar scale can be further quantized with minimal accuracy loss on CPU only inference 5 , suggesting the possibility of running the full simultaneous translation pipeline on constrained devices such as smartphones, eliminating a need in an external API and reducing the overall latency. We follow the approach of repurposing offline speech translation models for simultaneous mode because it has been shown as a simple, effective, and well-performing strategy, e.g. in 16 and in 13 . Offline models often offer high quality, multilinguality, and strong general robustness, which frequently outweigh their limitations such as hallucination on partial source sentences, lack of left-only context, or repeated encoding of the whole source buffer with every new incoming chunk. These limitations can be mitigated by dedicated simultaneous methods, but such methods typically require computationally expensive training, e.g. for stability on source prefixes 14 , for learning to wait or translate 11 , or for building specific simultaneous architectures 4 ; 12 . We observe a higher demand for offline speech translation than for simultaneous. New, high-quality offline models are becoming available, yet they are rarely evaluated or deployed in simultaneous mode using the state-of-the-art methods. Canary is one such model, the only previously existing simultaneous implementation 7 uses a sliding window approach that is outperformed by AlignAtt. Our primary goal is therefore to pioneer Canary usage in simultaneous mode with AlignAtt and evaluate it against state-of-the-art systems, to test whether it can serve as a practical system and as a strong baseline for future research. We conclude we have reached this goal. The results show competitive and in some cases even superior performance to state-of-the-art baselines while being much smaller in size. Our results show improvements by over 4-5 BLEU points over the organizers’ baseline on English to German and Italian and 5-8 BLEU points for Czech to English over the IWSLT 2025 best performing system ( 13 ) . Our implementation is integrated into the SimulStreaming project: github.com/ufal/SimulStreaming . The paper is structured as follows: in the Section 2 , we introduce the core model and policies and provide a relevant background. In Section 3 , we dive into implementation details of our submission, while Section 4 serves as a detailed description of the frameworks and evaluation metrics used, which we then follow by the evaluation result in Section 5 . Finally, we wrap up the paper with a conclusion and limitations regarding the implementation. 2 Background Canary-1B-v2 ( 24 ) is a strong multitask speech transcription and translation model with state-of-the-art performance. It outperforms whisper-large-v3 ( 23 ) and seamless-m4t-medium 6 in automatic speech recognition (ASR) and automatic speech translation (AST) tasks, and on some domains and language directions is even superior to s eamless-m4t-v2-large , which is twice as large in size. Additionally, it supports injecting context in the decoder prompt to bias the prediction with in-domain context, and provides word-level timestamps. Unlike Whisper, Canary does not need to work with fixed size audio inputs and was instead trained to support audios in 0 to 40 seconds range, which makes it a perfect candidate for simultaneous adaptation. AlignAtt ( 18 ) is a simultaneous policy, a method to process an offline AST or ASR model on output that is incrementally growing, one chunk at a time. The core idea AlignAtt is to use the cross-attention of the decoder to omit the suffix of hypothesis after the first token that attends to a predefined frame threshold. It has been shown to work well in long-form translation 15 , but has not been previously applied on Canary. Sliding window ( 25 ) is another simultaneous policy that we use as a contrastive baseline. Its core idea is to re-translate a window of audio input, use longest common subsequence with the previous translation to avoid repeating, and slide the window with newly available audio chunk. 3 Implementation Let us describe the details of the implementation of our systems and the end-to-end long-form speech-to-text processing pipeline. Simultaneous frameworks Evaluating simultaneous speech translation typically relies on frameworks that simulate real-time conditions on pre-recorded audio, enabling reproducible benchmarking. Two such frameworks support speech-to-text simultaneous systems and allow integration of new models: SimulStreaming ( 13 ) and Simulstream ( 7 ) . We use primarily SimulStreaming because it provides a robust implementation of SileroVAD 27 , a streaming voice activity detection that filters non-voice parts of the input audio and allows to save computational power on processing empty input, while also avoiding potential hallucinations. Since IWSLT 2026 task required Simulstream for computational aware evaluation, we finally transfer our Canary implementation to Simulstream to enable it. However, in case of differences, we consider our SimulStreaming implementation as the primary. Adapting Nemo Applying the AlignAtt policy to Canary requires decoder forced prefix injection 2 2 2 The approach is described in this thread https://github.com/openai/whisper/discussions/117#discussioncomment-3727051 for incremental output, which the NeMo API did not natively support. Hence, we highlight our contribution to the Nemo’s speech framework ( 9 ) that allows for this core incremental concept to be used with Canary. With our contribution, the prefix can be provided as an optional initial prompt to the decoder. Additionally, we fixed the bug in the cross-attention outputs for the beam decoder strategy, which allowed us to get deterministic dimension outputs and map the cross-attention scores to their respective output tokens. Processing loop Our audio processing pipeline follows the stages

Read the complete text