SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation
Zeyu Yang, Lai Wei, Roman Koshkin, Xi Chen, Satoshi Nakamura · 2025
Read SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation by Zeyu Yang, Lai Wei, Roman Koshkin, Xi Chen, Satoshi Nakamura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07781.txt.
Introduction
Zeyu Yang and co-authors build SASST, a live speech translation system that cuts the audio at grammar boundaries, in a paper from 2025. Simultaneous translation must decide, mid-sentence, when enough meaning has arrived to speak. Cutting at random points scrambles sense; cutting at phrase boundaries keeps it whole.
Their chunker parses dependency relations and punctuation to form complete units. A frozen Whisper encoder feeds a language model that either emits translated words or a WAIT token, balancing timing against content. Target-side reordering handles languages that stack words differently. On the CoVoST2 benchmark, translating English speech into German, Chinese, and Japanese, the system beat prior quality marks. The authors credit the syntax-aware cuts.
Catalog record
| CREATOR | Zeyu Yang, Lai Wei, Roman Koshkin, Xi Chen, Satoshi Nakamura |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07781
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SASST: Leveraging Syntax-Aware Chunking and LLMs for Simultaneous Speech Translation Zeyu Yang, Lai Wei, Roman Koshkin, Xi Chen, Satoshi Nakamura Abstract This work proposes a grammar-based chunking strategy that segments input streams into semantically complete units by parsing dependency relations (e.g., noun phrase boundaries, verb-object structures) and punctuation features. The method ensures chunk coherence and minimizes semantic fragmentation. Building on this mechanism, we present SASST (Syntax-Aware Simultaneous Speech Translation), an end-to-end framework integrating frozen Whisper encoder and decoder-only LLM. The unified architecture dynamically outputs translation tokens or <WAIT> symbols to jointly optimize translation timing and content, with target-side reordering addressing word-order divergence. Experiments on CoVoST2 multilingual corpus (En → \rightarrow De, Zh, Ja) demonstrate significant translation quality improvements across languages and validate the effectiveness of syntactic structures in LLM-driven SimulST systems. Introduction Simultaneous speech translation (SimulST) aims to generate target-language translations in real time while listening to ongoing source speech. Unlike offline translation, where the entire input is available before translation begins, SimulST must operate under streaming constraints and make decisions dynamically, balancing three often competing goals: translation quality, latency, and output coherence. Traditional SimulST pipelines typically consist of multiple independent modules, such as automatic speech recognition (ASR), segmentation, and neural machine translation (NMT) ( Ma et al. 2018 ; Zeng, Li, and Liu 2021 ) . While modular designs provide flexibility, they also suffer from error propagation, latency accumulation, and a mismatch between training and inference. In particular, segmentation and triggering decisions are often based on heuristics or shallow models, lacking deep contextual reasoning and limiting adaptability to varying speech patterns. Recent progress in large language models (LLMs) has revealed strong abilities in language generation, contextual reasoning, and task generalization ( Brown et al. 2020 ; Chowdhery et al. 2022 ; Achiam et al. 2023 ) . This has motivated research into LLM-based SimulST, where powerful sequence modeling capabilities are leveraged for low-latency translation. However, existing approaches often retain external policy modules or handcrafted segmentation strategies ( Zhang and Feng 2023 ) , separating “when to translate” from “what to translate” and thereby limiting interpretability and joint optimization. In this work, we propose a linguistically motivated, data-driven framework that internalizes read/write decision-making into an instruction-tuned LLM , unifying segmentation and translation within a single model. Instead of applying predefined segmentation rules during inference, we generate chunk-aligned supervision based on syntactic and semantic boundaries and use it in a two-stage training strategy to teach the model to predict explicit <WAIT> tokens alongside translation tokens. This enables the LLM to autonomously learn when and what to translate, guided by linguistic structure but without relying on external alignment tools or policy heads. Inspired by human interpreters, who naturally pause at syntactic or semantic boundaries, our approach yields translations that are more coherent and interpretable under streaming constraints. To further improve output fluency for language pairs with divergent word orders, we incorporate a chunk-aware reordering mechanism that rearranges translated segments into the natural target-language order. Our framework is model-agnostic and can be instantiated with different decoder-only LLM backbones. In this work, we evaluate two representative backbones, LLaMA3-8B ( Meta AI 2024 ) and Qwen3-8B ( Yang et al. 2025 ) , paired with a frozen Whisper encoder ( Cao et al. 2012 ) for speech feature extraction, and operate under causal constraints with all segmentation, alignment, and translation decisions unified within the model itself. Contributions Our main contributions are: • We present a unified, end-to-end SimulST system that integrates translation generation and read/write decision-making into a single LLM. • We propose a linguistically motivated chunk supervision method and a two-stage training strategy that enables the model to autonomously learn translation triggering through explicit <WAIT> token prediction, removing the need for external decision modules or handcrafted rules. • We design a chunk-aware reordering mechanism to improve translation coherence for language pairs with divergent word orders. Related Work Simultaneous speech translation (SimulST) aims to deliver translations while speech input is still ongoing, requiring models to balance translation fidelity and latency. Cascaded Systems Early SimulST systems predominantly adopted a cascaded architecture consisting of automatic speech recognition (ASR) followed by machine translation (MT) ( Oda et al. 2014 ; Le, Lecouteux, and Besacier 2017 ) . While effective, these pipelines suffered from error propagation and increased latency due to module coupling. Recent cascaded approaches have leveraged powerful pre-trained models to improve translation quality and latency control. For example, BeaverTalk ( Raffel, Agostinelli, and Chen 2025a ) combines Whisper ASR with an LLM-based translation module. Although these systems reduce latency and improve quality, their multi-module design still inherently couples recognition and translation processes, limiting joint optimization. End-to-End SimulST To overcome the limitations of cascaded architectures, end-to-end (E2E) SimulST models directly map input speech to translations within a single unified neural network, avoiding explicit intermediate transcriptions and module coupling ( Berard et al. 2016 ; Weiss et al. 2017 ; Bansal et al. 2018 ; Ren et al. 2020 ) . By integrating acoustic modeling, language modeling, and translation into a single optimization objective, these systems jointly balance latency and translation quality. Early approaches adopted encoder–decoder frameworks with streaming encoders and monotonic attention to handle partial speech input, enabling low-latency generation without waiting for utterance completion. Subsequent work further leveraged pretraining, adaptive alignment mechanismss, and multi-task objectives (e.g., simultaneous ASR + translation) to improve robustness and reduce lag. Despite their success, these models often require carefully designed read/write policies or specialized attention modules to handle the streaming nature of speech, and their decision-making process for when to emit translations remains either fixed or dependent on external heuristics. This limitation has motivated the recent shift towards incorporating large language models (LLMs) into SimulST, aiming to exploit their strong reasoning and generative capabilities while reducing reliance on handcrafted decision modules. LLM-based SimulST Recently, large language models (LLMs) have been introduced into SimulST to exploit their strong reasoning and generation capabilities. TransLLaMA ( Koshkin, Sudoh, and Nakamura 2024 ) is one of the earliest works to use LLMs for integrated read/write policy learning, showing that translation triggering decisions can be learned jointly with content generation. SimulS2S-LLM ( Deng et al. 2025 ) is the first to extend speech LLMs for simultaneous speech-to-speech translation (Simul-S2ST), leveraging boundary-aware speech prompts and a test-time wait-k policy to unlock streaming capability for offline-trained LLMs. StreamUni ( Guo et al. 2025 ) further explores unifying segmentation, translation, and generation within a single model using multi-stage reasoning steps. These approaches demonstrate the potential of LLM-based architectures for streaming translation but often rely on explicit decision policies or intermediate reasoning stages to determine translation triggers, instead of fully integrating decision-making into the translation process itself. Read/Write Policies and Segmentation Strategies A key challenge in SimulST is deciding when to emit translation tokens, commonly referred to as the read/write policy . Fixed strategies such as wait-k ( Ma et al. 2018 ) and fixed-length chunking ( Ma et al. 2021 ) offer predictable latency but lack adaptability. Adaptive approaches learn context-dependent policies through attention analysis ( Papi, Negri, and Turchi 2022 ; Papi, Turchi, and Negri 2023 ) , information-flow estimation ( Zhang and Feng 2022 ) , or segmentation-based decision making ( Zhang et al. 2022 ; Dong et