MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts

Yuxuan Lou, Kai Yang, Yang You · 2026

Read MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts by Yuxuan Lou, Kai Yang, Yang You (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10272.txt.

Introduction

Yuxuan Lou, Kai Yang, and Yang You build a model that takes in both speech and text. Most mixed models use the same weights for all inputs. But speech and text code facts in unlike ways. One set of weights may serve both poor.

The team made MoST, with a mix of expert nets at its core. Each input is routed to experts fit for its type. Speech goes to speech experts, text to text experts. The route is named modality aware. It lets one model hold both skills in one frame. The aim is smooth work across speech and text.

Catalog record

CREATORYuxuan Lou, Kai Yang, Yang You
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10272

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts Yuxuan Lou, Kai Yang, Yang You Abstract We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters—disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. 1 1 1 We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST 1 Introduction The seamless integration of speech and text modalities represents a cornerstone objective in the pursuit of versatile and natural human-computer interaction. Foundation models, particularly Large Language Models (LLMs), have demonstrated remarkable capabilities in text understanding and generation ( 32 ; 9 ) , motivating efforts to extend their prowess to encompass spoken language. However, unifying inherently different modalities like continuous, high-dimensional speech waveforms and discrete, symbolic text sequences within a single model presents significant architectural and training challenges. A prevalent approach in multimodal learning involves mapping different modalities into a shared representational space, often processed by a common set of transformer parameters ( 31 ; 10 ) . While conceptually simple, forcing diverse modality representations through identical parameters disregards their inherent representational differences. Specifically for speech and text, this uniform treatment risks representational interference, where the distinct statistical properties and structural nuances of each modality are diluted. Furthermore, it can hinder the development of highly specialized processing pathways necessary to capture the fine-grained details unique to each domain—such as phonetic variations in speech or complex semantic relations in text. Mixture of Experts (MoE) architectures have emerged as a powerful technique for scaling model capacity while maintaining computational efficiency ( 33 ; 24 ; 14 ) . By sparsely activating only a subset of parameters (experts) for each input token, MoE models can significantly increase their parameter count without a proportional increase in inference cost. While promising for handling complex data, standard MoE routing mechanisms are typically modality-agnostic; tokens are routed based on learned gating functions over their representations, without explicit consideration of their originating modality. This leaves untapped the potential to dedicate specialized expert capacity based on known input characteristics. In this work, we propose MoST (Mixture of Speech and Text), a novel speech-text LLM built upon a Modality-Aware Mixture of Experts (MAMoE) architecture. MAMoE directly addresses the limitations of uniform parameter processing by incorporating two principal components: modality-specific expert groups (e.g., text-only, audio-only) governed by a modality-aware routing mechanism, and shared experts accessible to all modalities, which facilitate cross-modal interaction. The modality-aware routing explicitly leverages token modality information to direct inputs towards appropriate modality-specific experts. Text tokens are primarily routed to text and shared experts, while audio tokens target audio and shared experts. This design allows for: (1) Specialized Learning: Modality-specific experts can capture the intricate patterns unique to either speech or text without interference. (2) Efficient Cross-Modal Interaction: Shared experts act as a bridge, facilitating knowledge transfer and cross-modal alignment crucial for tasks like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) synthesis. To effectively realize MoST, we develop an efficient LLM to Speech-Text Transformation Pipeline. Starting from a pretrained MoE LLM, we first conduct targeted post-training on large ASR and TTS datasets. This stage primes the modality-specific experts and adapts the shared experts for cross-modal tasks. Subsequently, we perform instruction fine-tuning using a carefully curated dataset comprising diverse speech and text instructions, enhancing the model’s versatility and controllability. A key aspect of our approach is the exclusive use of fully accessible, open-source training data to achieve strong performance. This strategic choice distinguishes MoST from some prominent speech-text models, such as SpiritLM ( 31 ) and Moshi ( 10 ) , whose development incorporated datasets that are not publicly accessible. Comprehensive evaluations across a range of speech-text benchmarks, including Automatic Speech Recognition (ASR), Text-to-Speech (TTS) synthesis, audio language modeling, and spoken question answering, demonstrate that MoST achieves state-of-the-art or competitive performance compared to existing models with similar parameter counts. Our ablation studies rigorously validate the effectiveness of the MAMoE architecture, confirming that the modality-aware routing mechanism is a key contributor to performance gains across both modality-specific and cross-modal tasks. To the best of our knowledge, MoST is the first fully open-source speech-text LLM leveraging a Mixture of Experts architecture with modality-specific routing. Our main contributions are: • Modality-Aware Mixture of Experts (MAMoE) : A novel multimodal MoE architecture that directs tokens to modality-specific expert groups and shared experts, enhancing specialized learning and cross-modal interaction. • Efficient Text-Speech Transformation Pipeline : A data-efficient pipeline for adapting a pretrained LLM into a speech-text LLM via targeted post-training and instruction tuning. • Open Release of MoST Model : We release the MoST model weights, training and inference code, and training dataset to facilitate future research and development in multimodal AI. 2 Related Work End-to-end Speech-Text Modeling . Existing end-to-end speech-text models like Qwen2-Audio ( 7 ) (text output only) and AudioLM ( 5 ) (no direct text integration) have specific limitations. Others such as SpiritLM ( 31 ) and Moshi ( 10 ) employ dense transformers, which process all tokens uniformly, unlike MoST’s Modality-Aware Mixture of Experts (MAMoE), which uses specialized and shared experts for modality-specific processing and cross-modal understanding. Mixture of Experts . Standard Mixture of Experts (MoE) architectures ( 24 ; 14 ; 11 ) scale models efficiently but typically feature modality-agnostic routing, limiting their multimodal effectiveness. While some research adapts MoE for general multimodal tasks (e.g., image-text) by grouping experts ( 25 ; 34 ) , MoST is the first to introduce MAMoE specifically for speech-text. Another key difference is that MoST’s MAMoE employs shared experts for cross-modal interaction, which is novel. Multimodal Adaptation from LLMs . LLMs are often adapted for multimodality via fine-tuning ( 37 ) , instruction tuning ( 26 ) , or speech-specific methods ( 13 ) . MoST presents a unique transformation pipeline: starting from a pretrained MoE LLM, it undergoes targeted ASR/TTS post-training, followed by fine-tuning with a curated speech-text instruction dataset. This approach offers significant data efficiency over conventional adaptation strategies. Figure 1: MoST overall architecture. MoST supports interleaved text and audio as input and can generate both text and audio. MoST blocks are transformer blocks

Read the complete text