MLMA: Towards M ulti L ingual ASR with MA mba-Based Architectures

Mohamed Nabih Ali, Daniele Falavigna, Alessio Brutti · 2025

Read MLMA: Towards M ulti L ingual ASR with MA mba-Based Architectures by Mohamed Nabih Ali, Daniele Falavigna, Alessio Brutti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18684.txt.

Introduction

Mohamed Nabih Ali works with Daniele Falavigna and Alessio Brutti. They build a speech recognition system for many languages. The system rests on a newer sequence model type. Speech across languages is a hard task. Some languages have little training data at all. Their system is called MLMA. It swaps the transformer backbone for Mamba. Mamba is a state-space design built for long sequences.

Mamba absorbs language-aware conditioning on its own. It also builds shared representations across languages. No separate machinery per language is needed. Standard multilingual benchmarks tested the system. MLMA performed on par with transformer-based systems. Transformers have long dominated this field. The authors present the result as clear evidence. State-space models can back speech recognition well. They are scalable, efficient, and accurate at once. That holds even for languages with scarce resources.

Catalog record

CREATORMohamed Nabih Ali, Daniele Falavigna, Alessio Brutti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MLMA: Towards M ulti L ingual ASR with MA mba-Based Architectures Mohamed Nabih Ali, Daniele Falavigna, Alessio Brutti Abstract Multilingual automatic speech recognition (ASR) remains a challenging task, especially when balancing performance across high- and low-resource languages. Recent advances in sequence modeling suggest that architectures beyond Transformers may offer better scalability and efficiency. In this work, we introduce MLMA (Multilingual Language Modeling with Mamba for ASR), a new approach that leverages the Mamba architecture—an efficient state-space model optimized for long-context sequence processing—for multilingual ASR. Using Mamba, MLMA implicitly incorporates language-aware conditioning and shared representations to support robust recognition across diverse languages. Experiments on standard multilingual benchmarks show that MLMA achieves competitive performance compared to Transformer-based architectures. These results highlight Mamba’s potential as a strong backbone for scalable, efficient, and accurate multilingual speech recognition. † † address: Center for Augmented Intelligence, Fondazione Bruno Kessler, Trento, Italy 1 Introduction Automatic Speech Recognition (ASR) has become a cornerstone of modern computing, supporting applications such as voice assistants, transcription services, and real-time speech translation. Driven by large-scale datasets and deep learning advances, ASR systems have reached near-human performance in high-resource languages like English and Mandarin [ 1 , 2 ] . However, most existing systems are language-specific, which limits scalability and exacerbates the performance gap for low-resource languages with limited annotated data [ 3 ] . Multilingual ASR has emerged as a promising alternative by training a single model across multiple languages [ 4 , 5 ] . Such models exploit shared phonetic and acoustic representations, enabling cross-lingual transfer from high-resource to under-represented languages. Despite this potential, achieving robust multilingual performance remains challenging. Transformer-based architectures, now dominant in ASR [ 6 , 7 ] , provide strong sequence modeling capabilities but with high computational and memory costs. These inefficiencies are especially problematic in multilingual scenarios, where diverse speech rates, prosodic patterns, and phenomena such as code-switching demand processing of long and variable-length utterances. Recently, Mamba architecture [ 8 ] has been proposed to handle variable-length input sequences and temporal irregularities, common in multilingual speech data. Therefore, it can generalize across languages with different rhythmic and phonetic structures. Mamba also supports streaming ASR with mechanisms like lookahead and unimodal aggregation (UMA), which help it adapt to real-time multilingual input [ 9 ] . These features are particularly beneficial for languages characterized by rapid speech transitions or tonal variations, where conventional models often struggle to maintain recognition accuracy and latency. Integrating Mamba into multilingual ASR offers several advantages: its memory-efficient design lowers training and inference costs [ 10 ] , its sequential inductive bias can better capture cross-lingual phonetic structures—benefiting code-switching and low-resource languages—and its scalability enables adding languages without proportional computational overhead. In this work, we investigate the application of Mamba-based architectures to multilingual ASR. ❶ We conducted experiments across a diverse set of European languages. ❷ Analyzing their ability to learn shared linguistic representations by comparing performance against Transformer-based baselines, and their robustness to multilingual challenges. Our goal is to bridge the gap between recent advances in efficient sequence modeling and the development of inclusive, scalable ASR systems that can serve a truly global user base. Our MLMA model, trained on almost 12K hours covering 6 languages, is the first multilingual ASR system based on Mamba . MLMA code and weights are publicly available under the most permissive license. 2 Related Works With the advent of deep learning, multilingual ASR systems, capable of recognizing multiple languages, has grown in demand for cross-lingual use [ 11 ] . Recent work in multilingual ASR has drastically increased language coverage to support hundreds and even thousands of languages. This includes approaches based on labeled training data such as Whisper [ 12 ] , USM [ 13 ] , Seamless [ 14 ] and MMS [ 15 ] , Ml-superb 2.0 [ 16 ] , FAMA [ 17 ] as well as zero-shot work [ 18 ] . While these transformer-based approaches are highly effective for modeling long-range dependencies, Transformers have notable drawbacks: their quadratic complexity makes long-sequence processing costly, they require vast amounts of labeled or weakly labeled data that are scarce for low-resource languages, and their large size limits their deployment in resource-constrained settings [ 19 ] . 2.1 Mamba for ASR Mamba has been applied to various speech tasks, for example, separation and enhancement [ 20 , 21 , 22 ] , leveraging its property of linear-time complexity to model the long sequence while maintaining low computational cost. Motivated by this, numerous studies have been conducted to evaluate Mamba’s performance in ASR tasks. Table 1 summarizes the most recent research papers leveraging Mamba for ASR tasks, highlighting the main contribution of each work. Table 1: The table summaries recent works exploring Mamba for ASR. Ref. Dataset (hours) Multilingual Language Contribution [ 20 ] LibriSpeech EN ConMamba for monolingual ASR [ 9 ] AISHELL-1&2 Mandarin Efficiency of Mamba for streaming ASR [ 23 ] LibriSpeech, AN4, SEAME, ASRU EN & EN-Mandarin BiMamba [ 24 ] LibriSpeech, GigaSpeech, SPGISpeech EN Samba-ASR: Mamba as Encoder and Decoder [ 25 ] LibriSpeech-100 EN Mamba-based HuBERT model for ASR, [ 26 ] LibriSpeech, GigaSpeech, TEDLIUM2, AISHELL, CSJ, VoxVorge EN, Mandarin, Japanese, IT Mamba performance against Transformer architectures [ 27 ] TEDLIUM3 EN Mamba-based HuBERT against Transformer-based SSL [ 28 ] LibriSpeech-100 EN Mamba for long-context ASR [ 29 ] LibriSpeech-100 EN Augmented ConMamba Encoder MLMA (ours) LibriSpeech, CommonVoice, Voxpopuli, Multilingual LibriSpeech EN, IT, FR, ES, DE, NL MLMA: a European Multilingual ASR based on Mamba Based on the literature review in Table 1 , current Mamba-based ASR research exhibits significant limitations. Existing studies mainly investigate architectural replacements within Transformer backbones, but are restricted to monolingual or at most bilingual settings on small datasets like LibriSpeech-100 [ 30 ] . These works operate under matched conditions and lack the scale to test Mamba’s multilingual effectiveness. In contrast, our proposed MLMA model explores Mamba in a large-scale multilingual setting with nearly 12,000 hours of training across six European languages, representing the first multilingual ASR system based on Mamba and demonstrating its viability beyond constrained setups. 3 Proposed Mamba architecture for multilingual ASR 3.1 Overview of Mamba Mamba is a Structured State Space Model (SSM) defined in discrete time as: h_{t}=\bar{A}h_{t-1}+\bar{B}x_{t},\hskip 9.24994pty_{t}=Ch_{t} (1) where h_{t} is the state, \bar{A} the transition matrix, \bar{B} the input-state interaction, and C the output map. Since \bar{A} and \bar{B} derive from continuous-time parameters, they are not learned directly but approximated via Zero-Order Hold (ZOH): \bar{A}=\exp(\Delta A),\hskip 9.24994pt\bar{B}=(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot\Delta B (2) with A,B the continuous forms and \Delta the discretization step. Training is done on A,B , which are converted to \bar{A},\bar{B} at each forward pass, enabling efficient discrete-time modeling while preserving long-range dependencies. ZOH ensures that the temporal structure of the continuous-time model is retained after discretization, allowing it to track dependencies across long sequences. To increase adaptability, [ 31 ] introduced a selection mechanism: B=f_{B}(x),\hskip 9.24994ptC=f_{C}(x),\hskip 9.24994pt\Delta=\text{Broadcast}_{D}(f_{\Delta}(x)) (3) that is, instead of using fixed matrices B,C,\Delta , the model learns functions f_{B},f_{C},f_{\Delta} , that generate these parameters based on the input x , allowing the model to flexibly adapt its state transition and output mapping according to the current input. Mamba [ 8 ] extends

Read the complete text