TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
Hainan Wang, Mehdi Hosseinzadeh, Reza Rawassizadeh · 2025
Read TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization by Hainan Wang, Mehdi Hosseinzadeh, Reza Rawassizadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00914.txt.
Introduction
Hainan Wang, with two co-authors, presents TinyMusician for on-device music generation. Transformer models set new marks for music generation quality. But their many parameters demand huge compute and time. That makes them infeasible on edge devices like phones and wearables. TinyMusician is a lightweight model distilled from MusicGen.
Distillation trains a small student to mimic a larger teacher. The model adds two key innovations. One is a stage-mixed form of KL-divergence training. The other is adaptive mixed-precision quantization, which stores numbers at varied precision. Tests show TinyMusician keeps most of MusicGen-Small's performance. It does so with fifty-five percent less model size. It is the first mobile music model with no cloud dependency.
Catalog record
| CREATOR | Hainan Wang, Mehdi Hosseinzadeh, Reza Rawassizadeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00914
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization Hainan Wang, Mehdi Hosseinzadeh, Reza Rawassizadeh Abstract The success of the generative model has gained unprecedented attention in the music generation area. Transformer-based architectures have set new benchmarks for model performance. However, their practical adoption is hindered by some critical challenges: the demand for massive computational resources and inference time, due to their large number of parameters. These obstacles make them infeasible to deploy on edge devices, such as smartphones and wearables, with limited computational resources. In this work, we present TinyMusician , a lightweight music generation model distilled from MusicGen (a State-of-the-art music generation model). TinyMusician integrates two innovations: (i) Stage-mixed Bidirectional and Skewed KL-Divergence and (ii) Adaptive Mixed-Precision Quantization. The experimental results demonstrate that TinyMusician retains 93% of the MusicGen-Small performance with 55% less model size. TinyMusician is the first mobile-deployable music generation model that eliminates cloud dependency while maintaining high audio fidelity and efficient resource usage. 1 1 1 https://github.com/maxW2000/tinyMusician 1 Introduction Music, reflecting culture, social classes, ethnic identities, and historical eras, has woven itself into humanity’s shared heritage through centuries of evolution ( Toynbee 2012 ) . Today, artificial intelligence (AI) has demonstrated remarkable breakthroughs across multimodal domains, including image generation ( Liu 2023 ) , inpainting, outpainting ( Silva & Oliveira 2024 ) , short video production ( Sun 2024 ) , etc. Large-Language Models (LLMs) ( Liu et al. 2023 ; Bi et al. 2024 ) showed excellent modeling capabilities in obtaining complex relationships in long-term contexts, which the music genre inherited. In view of this, MusicLMs ( Agostinelli et al. 2023 ) and many subsequent works ( Copet et al. 2023 ; Lam et al. 2023 ; Suno-Ai 2023 ) successfully applied LLMs in music generation, capitalizing on their ability to capture intricate patterns in musical sequences. However, the pursuit of higher-quality AI music generation, driven by two technical imperatives, Scaling Law ( Kaplan et al. 2020 ) and Emergent Capability ( Berti et al. 2025 ) , has led to a surge in model parameters, creating critical challenges in both computation and deployment. The frequently discussed text-to-music models, for example, MusicGen-Large ( Copet et al. 2023 ) and YuE-7B ( Yuan et al. 2025 ) , having undergone training on large-scale datasets, exhibited excellent capabilities in synthesizing music ( Austin et al. 2021 ) . The generated music was characterized by high fidelity ( Yao et al. 2025 ) , a high level of accuracy and detail in its sound quality, and a strong coherence with the provided text prompts. Yet, this parameter escalation introduces a trilemma between musical fidelity, computational cost, and deployment feasibility, especially for on-device deployment, where there are limited computational resources, such as smartphones and extended reality glasses. The dependency on a cloud, with high computational overheads, hinders the proliferation of AI-generated products into real-world applications, such as games, and keeps these models server-dependent ( Rawassizadeh et al. 2018 ; Huang et al. 2024 ; Rawassizadeh & Rong 2023 ) . There are several efforts to reduce the model size, especially in Transformer architecture, such as efficient self-attention mechanisms ( Tian et al. 2025 ) . Furthermore, several approaches such as Mixture of Experts (MoE) ( Jacobs et al. 1991 ) , and Low-Ranked Adaptation (LoRA) Hu et al. 2021 are used to reduce the computational cost of feed-forward layers of transformer architecture. On the other hand, several approaches focus on enabling the neural network models’ deployment on consumer electronics, such as Federated Learning ( Li et al. 2020 ) and model compression techniques. Three common approaches focus on model compression and reducing the size of a neural network, including Knowledge Distillation ( Gou et al. 2021 ) , pruning ( Zhang et al. 2022 ) , and quantization ( Wei et al. 2024 ) . In short, knowledge distillation transfers knowledge from a teacher model (baseline model) to a student model (smaller model), which ensures fidelity of results while having a smaller number of model parameters. This technique has demonstrated its efficacy across multiple AI domains. For instance, Mullapudi et al. 2019 proposed JITNet, employing MRCNN ( Tian et al. 2019 ) as the teacher model, and reduced the number of parameters from 300 million to 7 million. Sun et al. 2019a introduced Patient Knowledge Distillation, which distills the 12-layer BERT-original ( Devlin et al. 2019 ) into a 6-layer BERT while preserving 97% of its performance. In addition to the knowledge distillation, neural network quantization has emerged as another paradigm of critical model compression ( Gou et al. 2021 ; Wei et al. 2024 ) . By converting high-precision model weights into compact low-bit representations without altering the network architecture ( Gholami et al. 2022 ) , this technique has been proven effective in domains such as audio processing ( Derrien et al. 2006 ) , Deep Reinforcement Learning ( Lu et al. 2024 ) and image processing ( Rokh et al. 2023 ) . The third common approach is pruning ( Zhu & Gupta 2017 ) , which includes removing neurons that are not contributing much to the final output. Pruning could be determined based on the weight, activation value, or even the entire neuron and its connections ( Rawassizadeh 2025 ) . While these techniques have been extensively explored in different fields, especially image recognition ( Rokh et al. 2023 ) and natural language processing ( Sun et al. 2019a ) , their application to music generation remains underexplored. In this work, we present TinyMusician , a novel lightweight model for mobile, on-device music generation, distilled from the state-of-the-art MusicGen-Small ( Copet et al. 2023 ) architecture, integrating Stage-mixed Bidirectional KL-divergence in Knowledge Distillation with temperature annealing strategy ( Zhang et al. 2024 ) to enhance knowledge transfer fidelity between teacher and student models. To further enhance inference efficiency, we also implement adaptive mixed-precision quantization ( Chauhan et al. 2023 ) and achieve 55% reduction in model size compared to the original MusicGen with 9.5% sacrificing melodic or harmonic fidelity. Figure 1 presents the architecture of TinyMusician. We have integrated TinyMusician into the iOS mobile platform through ONNX runtime conversion and platform-specific optimization for iOS and Android. To our knowledge, this is the first on-device music generation model that can run on smartphones independently of the cloud or other large computational resources. Figure 1: The architecture of TinyMusician with respect to its teacher model, i.e., MusicGen small. 2 Related Works Deploying high-fidelity music generation on edge devices faces two co-dependent barriers: (1) Resource-Intensive Models: Transformer-based music models (Table 1 ) achieve remarkable quality but require strong GPU resources, which are incompatible with edge devices’ constraints; (2) Compression Limitations: Existing compression techniques lack specialized mechanisms to preserve musical fidelity, risking perceptual degradation. Therefore, our related work is composed of two sections: synthetic music generation and model compression. 2.1 Synthetic Music Generation Previous to neural network advances, the Markov chain ( Hassani & Wuryandari 2016 ; Shapiro & Huber 2021 ) , rule-based models ( Hastuti et al. 2017 ; Sneyers & De Schreye 2010 ) , and evolutionary algorithms ( Loughran & O’Neill 2020 ; Kaliakatsos-Papakostas et al. 2020 ) are three main groups