SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation

Chenyang Le, Bing Han, Jinshun Li, Songyong Chen, Yanmin Qian, Shanghai Jiao Tong University, China · 2025

Read SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation by Chenyang Le, Bing Han, Jinshun Li, Songyong Chen, Yanmin Qian, Shanghai Jiao Tong University, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01200.txt.

Introduction

Translating speech as it is spoken demands a constant trade between speed, accuracy, and smooth meaning. Chenyang Le, with four co-authors, presents SimulMEGA, a system for simultaneous speech translation that learns when to listen and when to write. Past systems struggled most when many languages were involved and their reading and writing rules diverged.

SimulMEGA joins prefix-based training with a mixture-of-experts refiner, a set of specialist parts guided by a router, to learn those timing choices without labels. It adds no cost at run time and needs only small changes to standard transformers. Across six language pairs, its speech-to-text model beat a leading baseline, losing little quality at low lag. The team also extended it to streaming speech output.

Catalog record

CREATORChenyang Le, Bing Han, Jinshun Li, Songyong Chen, Yanmin Qian, Shanghai Jiao Tong University, China
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01200

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation Chenyang Le, Bing Han, Jinshun Li, Songyong Chen, Yanmin Qian, Shanghai Jiao Tong University, China Abstract Simultaneous Speech Translation (SimulST) enables real-time cross-lingual communication by jointly optimizing speech recognition and machine translation under strict latency constraints. Existing systems struggle to balance translation quality, latency, and semantic coherence, particularly in multilingual many-to-many scenarios where divergent read/write policies hinder unified strategy learning. In this paper, we present SimulMEGA ( Simul taneous Generation by M ixture-of- E xperts GA ting), an unsupervised policy learning framework that combines prefix-based training with a Mixture-of-Experts refiner to learn effective read/write decisions in an implicit manner, without adding inference-time overhead. Our design requires only minimal modifications to standard transformer architectures and generalizes across both speech-to-text and text-to-speech streaming tasks. Through comprehensive evaluation on six language pairs, our 500 M-parameter speech-to-text model outperforms the Seamless baseline, achieving under 7% BLEU degradation at 1.5 s average lag and under 3% at 3 s. We further demonstrate SimulMEGA’s versatility by extending it to streaming TTS with a unidirectional backbone, yielding superior latency–quality trade-offs. 1 Introduction Simultaneous Speech Translation (SimulST) addresses the critical need for real-time cross-lingual communication by jointly optimizing speech recognition and machine translation under strict latency constraints. Unlike conventional offline systems that process complete utterances, SimulST operates on streaming audio input, incrementally generating translations while simultaneously decoding ongoing speech - a capability mirroring human interpreters’ cognitive processing. This technology enables transformative applications in international diplomacy, live media localization, and low-latency dialogue systems. However, it faces fundamental challenges in reconciling three competing objectives: translation quality, computational latency, and semantic coherence, exacerbated by fragmented acoustic patterns and partial contextual dependencies. Recent progress combines adaptive segmentation strategies [ 1 , 2 , 3 ] with streaming architectures like RNN-T transducers [ 4 , 5 , 6 ] and Monotonic Attention mechanisms [ 7 , 8 ] , alongside policy-based decision frameworks [ 9 , 10 , 11 , 12 , 13 , 14 ] . Despite these advances, current systems struggle to achieve human-like efficiency-accuracy trade-offs across diverse linguistic contexts and acoustic conditions. A particularly underexplored challenge lies in multilingual many-to-many translation, where divergent read/write policies across language pairs complicate the learning of unified operational strategies. This paper introduces SimulMEGA, an unsupervised policy learning framework that synergizes prefix-based training with Mixture-of-Experts (MoE) mechanisms to address these limitations. Our approach enables high-performance multilingual SimulST through three key innovations: (1) An auxiliary MoE refiner module enabling implicit policy learning without inference-time overhead, (2) Minimal architectural modifications to standard transformers for broad applicability, and (3) A unified framework supporting both speech-to-text and text-to-speech streaming tasks. Experimental results demonstrate state-of-the-art performance across six languages, with our 500M-parameter speech-to-text model outperforming SeamlessM4T [ 15 ] by achieving <7% BLEU degradation at 1.5s average lagging (AL) and <3% at 3s AL. What’s more, we extended our methods to streaming text-to-speech (TTS) task via CosyVoice2’s unidirectional backbone [ 16 ] , the system achieves superior latency control compared to conventional interleaved streaming approaches. Our principal contributions are as follows. • Introduction of SimulMEGA : An unsupervised policy learning framework that combines prefix-based training with a Mixture-of-Experts (MoE) refiner module, enabling efficient and effective simultaneous speech translation without adding inference-time overhead. • Multilingual Advancement : SimulMEGA achieves SOTA performance for quality-latency tradeoffs in many-to-many translation, demonstrating cross-lingual robustness through comprehensive 6-language evaluation. • Universal Streaming Framework : SimulMEGA supports both speech-to-text and text-to-speech streaming tasks within a single framework, making it broadly applicable and easily integrable into existing models for multilingual SimulST tasks. • General Conversational System : The integrated solution combining low-latency speech-to-text and text-to-speech conversion, validated through real-world dialog interpretation scenarios. 2 Related Works Figure 1 : Overview of the training and inference paradigm of SimulMEGA. SimulMEGA is composed of a Streaming Speech Encoder, a text decoder, an MoE routing Gate(Router), and an MoE refiner. In the first stage, the model is pre-trained on \mathcal{L}^{\mathrm{offline}} . In the second stage, the model is trained on a combination of \mathcal{L}_{\mathrm{offline}} , \mathcal{L}^{\mathrm{prefix}} and \mathcal{L}^{\mathrm{refiner}} . SSE denotes Streaming Speech Encoder. SimulST Early works in SimulST focus on finding the input segmentation boundary and then applying a variant of the wait-k policy [ 1 , 17 , 2 , 3 ] . Later work seeks to find a more sophisticated dynamic policy. [ 18 ] designs a learnable matrix between source and target for an integrate-and-fire(IF) policy, and [ 19 , 20 ] use attention score for a similar approach. [ 11 ] proposes a policy based on divergence between the prefix and full input. [ 12 ] employs an extra CTC [ 21 ] head for policy control. Another approach includes using an inherently streaming architecture like monotonic attention [ 7 , 22 , 8 , 23 ] , transducer [ 4 , 5 , 6 ] and Hidden Markov [ 24 ] . [ 25 ] explores non-autoregresive generation for SimulST. Streaming TTS Many works on TTS claim to be streamable [ 26 , 27 ] , yet they are not optimized for fine-grained(chunk or token level) streaming and suffer from degraded quality due to a lack of future information. [ 28 ] adapts a streaming model from an offline model via restricted attention. [ 29 ] uses a fixed number of text tokens for look-ahead. [ 30 ] processes each text chunk in a separate AR loop with look-ahead. [ 16 ] interleaves text and speech tokens at a fixed ratio, which potentially incurs larger latency. 3 SimulMEGA: Simultaneous Translation via Mixture-of-Experts Routing In this section, we introduce SimulMEGA ( Simul taneous Generation by M ixture-of- E xperts GA ting), an innovative framework that effectively converts an offline autoregressive model into a simultaneous system with minimal computational overhead and negligible performance degradation. The core principle of simultaneous training lies in enabling the model to autonomously determine whether the currently available input suffices for generating the next output token. Previous approaches typically rely on either artificial policy signals, which constrain the model’s self-learning capacity, or architectural modifications that compromise generation quality. In contrast, our framework leverages a Mixture-of-Experts (MoE) architecture to achieve unsupervised policy learning while maintaining an identical structure to the offline model, thereby preserving translation performance. Our approach employs a prefix-based training strategy that closely mimics real-world streaming inference scenarios. 3.1 Architecture Overview Figure 1 illustrates the overall architecture of our SimulMEGA framework, which consists of four key components: (1) a streaming speech encoder, (2) a text decoder, (3) a global routing gate, and (4) a Mixture-of-Experts (MoE) refiner module. The streaming speech encoder and text decoder adopt the Transformer architecture, maintaining compatibility with conventional speech-to-text translation (S2TT) systems. As illustrated in Figure 2 (a), the encoder combines chunk-wise autoregressive (Chunk-AR) blocks with non-autoregressive (NAR) blocks in a hybrid design. The Chunk-AR blocks optimize inference efficiency through a cached key-value mechanism, while the NAR blocks preserve translation quality by capturing global context. To handle streaming inputs lacking an explicit end-of-sequence (EOS) token, we prepend a learnable end-of-stream (EoSt) flag (a binary embedding) to the NAR blocks’ input, signaling whether the current chunk terminates the audio stream. The text decoder follows a standard autoregressive Transformer

Read the complete text