Efficient Interleaved Speech Modeling through Knowledge Distillation

Mohammadmahdi Nouriborji, Morteza Rohanian · 2025

Read Efficient Interleaved Speech Modeling through Knowledge Distillation by Mohammadmahdi Nouriborji, Morteza Rohanian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23670.txt.

Introduction

Mohammadmahdi Nouriborji and Morteza Rohanian build compact speech generation models by distilling a large one, in this 2025 paper. Current speech language models are too big and slow for many settings. Their TinyWave family holds two billion parameters and was trained on fifty thousand hours of public audio. Distillation matched the teacher's inner states, attention maps, and softened outputs, layer by layer.

The shrink cut size threefold with small quality loss. TinyWave handles speech-only output and mixed speech-text streams. On a public test set it landed close to its teacher in perplexity. On spoken story and sound tasks it kept most of the teacher's accuracy and beat same-size rivals. Models, code, and test scripts are released for others to build on.

Catalog record

CREATORMohammadmahdi Nouriborji, Morteza Rohanian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23670

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Interleaved Speech Modeling through Knowledge Distillation Mohammadmahdi Nouriborji, Morteza Rohanian Abstract Current speech language models exceed the size and latency constraints of many deployment environments. We build compact, expressive speech generation models through layer-aligned distillation, matching hidden states, attention maps, and softened logits to compress large multimodal transformers by 3× with minimal loss in performance. We introduce TinyWave, a family of 2B-parameter models for speech-to-speech and interleaved speech–text generation, trained on 50k hours of public audio. TinyWave supports (i) speech-only generation using phonetic or expressive tokens and (ii) mixed speech–text continuations. Evaluation on Libri-Light shows TinyWave within 1.4 normalized-perplexity points of its teacher. Accuracy on spoken StoryCloze and SALMon reaches 93–97% of the teacher’s performance, outperforming size-matched baselines. These models are optimized for deployment on commodity hardware, enabling applications in real-time conversational agents, assistive technologies, and low-resource environments. We release models, training code, and evaluation scripts to facilitate reproducible research on compact, expressive speech generation. 1 1 1 Generation samples are available at: https://mohammadmahdinoori.github.io/tinywave-landing/ ; models are available on Hugging Face: https://huggingface.co/tinywave ; training and inference code is available at: https://github.com/mohammadmahdinoori/TinyWave . 1 Introduction Expressive speech models aim to generate speech that captures not only the correct words, but also rich prosody, speaker-specific traits, and emotionally appropriate delivery. These features are essential for natural and effective human–computer interaction, particularly in storytelling, conversation, and assistive settings. While large language models (LLMs) excel at reasoning and linguistic generalization, text-only systems overlook acoustic and paralinguistic information critical to spoken communication. Speech models often struggle with semantic understanding due to the complexity of mapping discrete audio tokens to high-level linguistic concepts, a challenge compounded by limited paired speech-text data. Recent research has explored multimodal models that process both speech and text, as well as interleaved inputs where audio and text alternate within a single context. These models better reflect real-world communication patterns and offer more flexible interaction. However, they typically require extensive training data and compute, making them unsuitable for edge deployment and resource-constrained environments. A more tractable but limited alternative assembles an automatic speech recognition (ASR) frontend, a text-only LLM, and a text-to-speech (TTS) backend. While effective for basic language tasks, this pipeline discards much of the expressive signal—prosody, speaker identity, and acoustic nuance—leading to unnatural outputs. Recent speech models address this by modeling audio end-to-end using discrete units. However, most of these models are trained from scratch at large scales, and model providers often maintain separate checkpoints for each deployment size. This practice is resource-intensive and inefficient, especially when targeting multiple operating points. Figure 1: Distillation framework: The student aligns with the teacher’s logits, intermediate states, and ground-truth labels, guided by a multi-part loss. We introduce TinyWave, a family of 2 billion-parameter speech-to-speech transformers distilled from the larger SpiritLM Nguyen et al. 2025 . TinyWave models preserve expressive qualities of speech and support both audio-only and interleaved speech–text inputs. Instead of training new models from scratch, we use layer-aligned distillation to transfer knowledge from a high-capacity teacher. We match hidden states, attention maps, and softened logits between teacher and student, encouraging the student to learn the teacher’s internal mechanisms. We also fine-tune the teacher on target-domain data to reduce distribution mismatch before distillation. TinyWave performs strongly across several benchmarks. On StoryCloze and the SALMon style suite, it retains 93–97% of the teacher’s accuracy. On Libri-Light continuations, it remains within 1.4 Normalised Perplexity Score (NPS) points and sometimes surpasses the teacher on style metrics. It also outperforms several larger models from prior work, showing that well-designed, compact models can rival or exceed larger systems under constrained resources. All experiments use the publicly available Libri-Light dataset, which represents only a fraction of the data used to train the teacher model Nguyen et al. 2025 . Our results show that high-quality expressive speech modeling is possible without access to private datasets or large-scale compute. In summary, our work contributes: 1. A layer-aligned distillation framework for expressive end-to-end speech generation, enhanced by teacher fine-tuning to reduce domain shift. 2. Three efficient models: Base-Speech , Expressive-Speech , and Expressive-Interleaved , covering audio-only and multimodal speech–text inputs. 3. Open-source training and evaluation tools that enable reproducibility, extension, and deployment on commodity hardware. 2 Methodology We present the TinyWave family: compact 2-billion-parameter speech-to-speech models distilled from 7B-parameter SPIRIT LM teachers Nguyen et al. 2025 . The SPIRIT LM models extend LLaMA-2 with speech capabilities by training on continuous streams of HuBERT-based audio tokens and interleaved text. TinyWave models inherit this cross-modal foundation through distillation. We release two speech-only variants and one interleaved model that handles mixed audio-text sequences. Our training pipeline combines model initialization, teacher correction, and a layer-to-layer distillation framework. 2.1 Model Initialization We initialize the TinyWave models to retain the core architectural properties of their SPIRIT LM teachers. For the speech-only models, Base-Speech and Expressive-Speech , we start from SpiritLM-Base-7B and SpiritLM-Expressive-7B, respectively. We prune every third transformer block, reducing the depth to 10 layers while keeping the original embedding layers, rotary position encoding (RoPE) settings, and language modeling head. This pruning yields models with approximately 2 billion parameters. SPIRIT LM uses HuBERT-based discrete audio tokens, grouped into two variants: (i) a base phonetic tokenizer with 100 clusters and (ii) an expressive tokenizer that adds pitch and style tokens on top of the phonetic stream. The base speech variant adopts the base tokenizer, while expressive speech uses the expressive tokenizer to support richer acoustic modeling, including prosody and speaker variation. For the interleaved model, Expressive-Interleaved , we initialize from the already distilled expressive-speech student rather than the 7B teacher. This transfer speeds up training and leverages the expressive speech priors learned during earlier distillation. Teacher correction involves fine-tuning SpiritLM on a 10,000-hour subset of Libri-Light using LoRA adapters. This step aligns the teacher’s distribution with the target domain, reducing mismatch during distillation and improving supervision quality. 2.2 Speech-Only Models For the base speech variant, we fine-tune the SpiritLM-Base-7B teacher on a 20% subset of Libri-Light. This teacher correction step ensures the supervision matches the target domain and avoids distributional mismatch. We skip this step for the expressive-speech model, since its teacher is already trained to encode prosodic and expressive cues relevant to the target task. We tokenize audio into sequences of HuBERT-based discrete units using the appropriate tokenizer. We train the student with an autoregressive language modeling loss, predicting the next token in the sequence. We apply our distillation framework to align the student’s internal representations and output distributions with the teacher’s, as described below. 2.3 The Interleaved Model The expressive interleaved model extends speech modeling to mixed audio-text inputs. Following SPIRIT LM, we create interleaved training data by combining Libri-Light speech with pseudo-transcriptions generated by Whisper-v3 Large. We construct interleaved input sequences by inserting text spans into audio token streams at word boundaries. This preserves semantic continuity and minimizes alignment errors between modalities. We sample from five interleaving patterns during training: [Speech] , [Text] , [Speech][Text] , [Text][Speech] , and [Speech][Text][Speech] , each with a fixed probability. This sampling encourages diversity and teaches the model to switch between modalities mid-sequence. We fine-tune the 7B expressive teacher on this interleaved corpus for a short duration (10k steps), then distill its knowledge into the student using our

Read the complete text