Cross-Lingual Interleaving for Speech Language Models
Adel Moumen, Guangzhi Sun, Philip C. Woodland · 2025
Read Cross-Lingual Interleaving for Speech Language Models by Adel Moumen, Guangzhi Sun, Philip C. Woodland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01865.txt.
Introduction
Adel Moumen, Guangzhi Sun, and Philip Woodland interleave languages to lift speech language models. Speech models learn language straight from audio units. That widens access for tongues with little written text. Progress has stayed English-centric due to scarce data and tests elsewhere.
Cross-lingual interleaving mixes speech units across languages in training. Shared structure can then transfer to low-resource tongues. The team tests whether that transfer holds in practice. Gains would bring voice tools to many communities now left out of language technology. Interleaving needs no parallel corpora, a key gain for tongues where aligned speech pairs across languages simply do not exist yet.
Catalog record
| CREATOR | Adel Moumen, Guangzhi Sun, Philip C. Woodland |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01865
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Lingual Interleaving for Speech Language Models Adel Moumen, Guangzhi Sun, Philip C. Woodland Abstract Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English - centric due to scarce spoken evaluation benchmarks and training data, making cross - lingual learning difficult. We present a cross - lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN – FR training dataset, TinyStories ( ∼ 42 \sim\!42 k hours), together with EN – FR spoken StoryCloze and TopicCloze benchmarks for cross - lingual semantic evaluation, both synthetically generated using GPT - 4. On 360 360 M and 1 1 B SLMs under matched training - token budgets, interleaving improves monolingual semantic accuracy, enables robust cross - lingual continuation, and strengthens cross - lingual hidden - state alignment. Taken together, these results indicate that cross - lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility. † † address: Department of Engineering, University of Cambridge, UK {am3303,gs534,pcw117}@cam.ac.uk 1 Introduction Spoken Language Models (SLMs) aim to learn language directly from speech, without relying on textual supervision. This line of work emerged from ‘textless’ NLP [ 19 ] , which combines discrete speech representations [ 23 ] and neural language modelling [ 3 , 28 ] to capture both acoustic and linguistic regularities from audio alone. Within this framework, Generative Spoken Language Modelling (GSLM) formalises the objective by training an autoregressive LM on sequences of discrete speech units and reconstructing waveforms with a unit vocoder, with the motivation of broadening access to NLP in settings where written resources are scarce. Despite rapid progress, the cross-lingual setting, which aims to train a single SLM over multiple languages, remains comparatively under-explored [ 25 , 36 , 13 ] . Two practical bottlenecks persist: (i) evaluations are predominantly English-centric, hindering measurement of cross-lingual semantic competence; and (ii) training methods that encourage sharing across languages often depend on text , for example via speech–text interleaving in Text–Speech LMs (TSLMs) [ 5 , 25 , 7 , 35 ] , which is not directly compatible with ‘textless’ SLMs. As a result, the original promise of ‘textless’ NLP learning from audio alone at scale and across languages has not yet been fully realised. We address these limitations by introducing a cross-lingual interleaving scheme that mixes speech tokens from different languages within the same training sequence, using sentence-level alignments but no text tokens. Exposure to interleaved multilingual contexts encourages a shared representational subspace and promotes transfer, while remaining compatible with pure ‘textless’ pipelines. To make this practical and measurable, we curate bilingual, sentence-aligned spoken resources and new spoken semantic benchmarks. Our contributions are as follows: (i) We propose a cross-lingual interleaving strategy that concatenates sentence - aligned speech token sequences across languages, promoting a shared representational subspace. (ii) We release a French–English, sentence-aligned spoken corpus derived from TinyStories [ 12 ] (approximately 42 k hours in total) together with bilingual spoken StoryCloze and TopicCloze benchmarks [ 22 ] for semantic evaluation synthetically generated using GPT-4 [ 1 ] . (iii) We show on 360 M and 1 B parameter SLMs that the method yields positive transfer to monolingual semantic tasks and strong cross-lingual performance, under a matched training-token budget. Additionally, representation analyses indicate stronger cross-lingual alignment in the hidden states. Section 2 situates our approach with respect to SLMs, speech–text interleaving, and multilingual training. Section 3 formalises the modelling setup and the cross-lingual interleaving scheme; Sections 4 – 6 present datasets, experiments, and analyses. 2 Related Work 2.1 Speech Language Models Early SLMs instantiate the ‘textless’ pipeline by modelling discrete speech units and decoding back to waveforms. GSLM [ 19 ] trains a decoder-only LM over units and synthesises speech with a unit-based vocoder; pGSLM [ 17 ] augments units with prosodic tokens to improve expressiveness; and dGSLM [ 24 ] introduces a dual-tower variant for two-channel dialogues. While these systems can produce locally fluent speech, long-form coherence is challenging. SpeechSSM [ 26 ] explores state-space models [ 14 ] for long-form generation, TWIST [ 15 ] shows that initialising from a pre-trained text Large Language Model (LLM) can boost semantics, and AudioLM [ 2 ] cascades semantic and acoustic token streams. More recently, Slamming [ 21 ] provides a competitive compute-efficient recipe, and Align-SLM [ 20 ] fine-tunes SLMs with direct preference optimisation. Our work stays within the speech-only paradigm but targets cross-lingual sharing through a new learning method. 2.2 Joint Speech–Text Interleaving TSLMs extend SLMs with text to encourage cross-modal transfer. VoxtLM/SUTLM [ 5 ] jointly trains on ASR, TTS, and continuation, and identifies interleaving (switching between text and speech inside a sequence) as particularly effective for building a shared space. Spirit-LM [ 25 ] scales this recipe and emphasises interleaving as a key quality driver; SmolTolk [ 7 ] proposes architecture refinements for vocabulary expansion; and GLM4-Voice [ 36 ] leverages a text-to-token model [ 35 ] to generate audio tokens on-the-fly, sidestepping explicit aligners [ 29 , 27 ] . However, these approaches depend on text and interleave within a single language, limiting direct applicability to textless, cross-lingual SLM training. We instead interleave speech tokens across languages without introducing text. 2.3 Cross-Lingual Training Multilingual training for speech LMs is gaining traction but remains under-evaluated on spoken semantic understanding beyond English. Spirit-LM [ 25 ] includes multiple languages at scale but reports only English spoken evaluations; GLM4-Voice [ 36 ] trains on Chinese and English yet evaluates English semantics; and scheduled interleaving for speech-to-speech translation [ 13 ] interleaves monolingually with no cross-language interaction during continuation. In text, mixing languages can induce positive transfer [ 32 ] . Our method brings this intuition to the ‘textless’ speech regime by interleaving sentence-aligned speech sequences across languages, enabling both cross-lingual abilities and improved monolingual semantics under a matched training-token budget. 3 Spoken Language Modelling The task of modelling spoken language from raw audio typically comprises three stages: (i) quantising a waveform \mathbf{x}\in\mathbb{R}^{T} into a sequence of L discrete speech units with L\ll T ; (ii) training an SLM under a next - token objective on the quantised sequence; and (iii) synthesising a waveform from the discrete units via a neural decoder. We first describe speech tokenisation, then the SLM objective, and finally our cross - lingual interleaving scheme. 3.1 Speech Language Models Speech tokenisers. Speech tokenisers transform raw waveforms into discrete representations. Let \mathcal{X} denote the domain of audio samples and let a waveform be \mathbf{x}=(x_{1},\dots,x_{T})\in\mathcal{X}^{T} . An encoder f:\mathbb{R}^{T}\!\rightarrow\!\mathbb{R}^{T^{\prime}\times d_{e}} produces a sequence of continuous features f(\mathbf{x})=(\mathbf{v}_{1},\dots,\mathbf{v}_{T^{\prime}}) , sampled at a lower frame rate T^{\prime}\ll T . A quantiser q maps these to discrete units \,\mathbf{s}=(s_{1},\dots,s_{T^{\prime}}) with s_{i}\in\{1,\dots,K\} , where K is the codebook size. Common choices include k - means and residual vector quantisation (RVQ). In this work we adopt Mimi [ 9 ] , a neural audio codec [ 33 , 8 ] implemented as a convolutional auto - encoder with RVQ in the latent space. Following prior evidence that