AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task

Quentin Fuxa, Dominik Macháček · 2026

Read AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task by Quentin Fuxa, Dominik Macháček (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03967.txt.

Introduction

Fuxa and Macháček describe a live speech translation system for three target languages. It turns English speech into German, Italian, and Chinese. The system formed their IWSLT 2026 entry. It pairs Qwen3-ASR with Gemma-4 for translation in a timed cascade. AlignAtt guides when each draft piece may be released.

AlignAtt first served encoder-decoder models, which decoder-only models lack. The pair recover the policy with four moves. They place a clear source span in the prompt. They pick translation heads offline and replay key attention fast. Runtime capture of queries and keys keeps outputs unchanged, bit for bit. On the dev set, the system beat the given baselines for German and Italian. Results for Chinese were mixed, the authors note.

Catalog record

CREATORQuentin Fuxa, Dominik Macháček
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03967

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task Quentin Fuxa, Dominik Macháček Abstract We describe AlignAtt4LLM, an IWSLT 2026 simultaneous speech translation system for English to German, Italian, and Chinese. The system is a synchronous cascade: Qwen3-ASR with forced alignment produces an incrementally updated source transcript, and Gemma-4 E4B-it translates that prefix under an MT-side AlignAtt policy. To our knowledge, this is the first application of AlignAtt to a decoder-only LLM, where the encoder-decoder cross-attention used by earlier AlignAtt systems is absent. We recover a usable policy by proposing (1) an explicit source span in the prompt, (2) offline selection of translation-specific alignment heads, (3) selective qk-fast replay of the draft-to-source attention block, and (4) runtime query/key capture that preserves model outputs bit-identically. On the IWSLT 2026 development set, AlignAtt4LLM outperforms the supplied baselines for the European target languages, English to German and English to Italian, in both the low-latency regime around 2 seconds and the high-latency regime below 4 seconds CU-LongYAAL. Results for English to Chinese are more mixed, but the method is not tied to Gemma-4: because AlignAtt4LLM only requires a deterministic prompt layout, calibrated attention heads, and query/key capture, the same policy can be reapplied to stronger translation-focused decoder-only MT backbones for non-European target languages. 1 Introduction This paper describes the AlignAtt4LLM IWSLT 2026 submission for English to German, Italian, and Chinese simultaneous speech translation ( 1 ) . The system is a synchronous cascade: Qwen3-ASR with the Qwen3 forced aligner ( 19 ) produces an incrementally updated source transcript with word end times, and Gemma-4 E4B-it ( 3 ) translates the current source text under an MT-side AlignAtt policy. The central contribution is to make an offline decoder-only LLM usable in simultaneous mode. Standard AlignAtt reads encoder-decoder cross-attention, but decoder-only LLMs have no such cross-attention. We instead expose the source transcript as a known prompt span, select a small set of translation-specific self-attention heads offline, and accept only draft prefixes whose reconstructed attention signal remains within the currently available source frontier. The second contribution is making the implementation fast enough for computational-aware (CA) evaluation. Both ASR and MT are served through vLLM ( 7 ) , which gives the cascade one high-throughput inference stack and hot model reuse across chunks. On the MT side, this also means the policy cannot inspect a Python-visible attention matrix: self-attention is hidden inside fused kernels. We therefore capture the exact query/key tensors consumed by the deployed Gemma attention module and replay only the draft-to-source block needed by AlignAtt. Our implementation is available at https://github.com/QuentinFuxa/AlignAtt4LLM . Section 2 describes the context of the task and prior work, Section 3 describes the cascade, Section 4 details the MT-side AlignAtt realization, and Section 5 reports the evaluation. 2 Background AlignAtt ( 12 ; 11 ) is a simultaneous policy that lets an offline sequence-to-sequence model operate in simultaneous mode. At each generation step, the policy derives a source position for the next target token from attention and stops generation when that position crosses the accessible-source frontier. Earlier AlignAtt systems obtained this signal from encoder-decoder cross-attention. AlignAtt is currently recognized as state of the art, it has been used in e.g. by the top-performing IWSLT 2025 system ( 9 ) . Offline models in simultaneous mode. Repurposing offline models for simultaneous translation is a highly promising approach because it allows reusing high-quality, multilingual, instruction-following models without training dedicated simultaneous models for every language direction. However, the disadvantage is that an offline model is not trained to decide when a partial source prefix is sufficient, and naive re-translation can flicker or hallucinate on incomplete input. Simultaneous policies such as AlignAtt provide the missing commit decision while preserving the quality and flexibility of the offline model ( 12 ; 9 ) . Decoder-only LLMs have recently become central in high-quality text-to-text MT systems ( 6 ; 5 ) . Prior simultaneous systems have already used instruction-tuned decoder-only LLMs by prompting them with the translation direction, current source prefix, and already emitted target prefix ( 9 ) . Long context, instruction following, and in-context examples make this family attractive for robust MT, but previous decoder-only simultaneous implementations used LocalAgreement ( 15 ; 16 ) rather than AlignAtt. LocalAgreement is a strong simultaneous policy, but it does not take advantage of model-internal attention and generally introduces more latency than AlignAtt ( 9 ) . Translation-specific attention heads. 8 show that multilingual decoder-only LLMs contain a sparse set of heads whose attention argmaxes track source-target word alignments. This suggests a decoder-only AlignAtt path, but only if two runtime conditions are met: the source span must be identifiable in the prompt, and the policy-visible attention rows must be reconstructed from the same tensors used by the deployed inference engine. Our proposed method supplies both conditions. 2.1 IWSLT 2026 Simultaneous Task Simulation environment. The IWSLT 2026 Simultaneous Task proposes the Simulstream toolkit ( 2 ) for simulating simultaneous translation sessions. We follow the long-form, unsegmented task setup, which allows re-translation at the evaluation interface, but AlignAtt4LLM itself emits append-only incremental output, the mode preferred in real-world deployments where re-translation flicker could disrupt readers following the live text. Development data. We use the official IWSLT 2026 MCIF development set. We report translation quality with BLEU ( 13 ) , chrF ( 17 ) , and XCOMET-XL ( 4 ) . Translation latency is reported with LongYAAL ( 14 ) in both computational-unaware (CU) and computational-aware (CA) variants. 3 AlignAtt4LLM System Overview ASR. The speech component of the cascade uses two upstream Qwen components: Qwen3-ASR-1.7B transcribes the live audio tail, and Qwen3-ForcedAligner-0.6B assigns word-level start and end times to that transcript ( 19 ) . At every chunk boundary, the cascade re-transcribes the current live utterance tail and invokes the forced aligner with timestamp output enabled; the word times are therefore produced online inside our simulation run, not added later as an offline post-processing step. Adjacent ASR hypotheses are stabilized by a longest-common-prefix commit rule up to sentence-final punctuation, while the remaining live tail is allowed to change on the next chunk. We retain this dedicated aligner because it performed better than all the alternative candidates summarized in Appendix A . MT. The translation component uses Gemma-4 E4B-it ( 3 ) served through vLLM ( 7 ) . With every source update, Gemma receives the current transcript prefix, the already accepted target prefix, and a fixed translation instruction, then greedily generates a draft continuation of at most 16 new tokens. AlignAtt accepts the longest draft prefix whose alignment signal stays on the currently accessible side of the source frontier. Synchronization. The cascade is synchronous and chunk-based. Each chunk first updates the ASR transcript prefix and then launches one MT request. Source words are considered accessible only once their end time is older than a hold-back margin; in the official IWSLT runs, this margin is 0 ms, so a word becomes accessible as soon as its aligned end time is observed (see Section 5.3 for a reliability analysis suggesting a conservative 250 ms tail hold-back). Both regimes also delay the first MT emission until 2 seconds of source audio have accumulated. This conservative synchronous regime cleanly separates policy behavior

Read the complete text