BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction
Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu, Shinji Watanabe · 2025
Read BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction by Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu, Shinji Watanabe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06150.txt.
Introduction
Haoran Wang and co-authors build a neural audio codec that treats speech, music, and sound with one system without slighting any of them. Codecs compress audio into compact codes. Speech energy hugs narrow bands near pitch harmonics, while music and sound need the full spectrum, above all the high frequencies that carry timbre.
Speech-tuned codecs degrade on other audio, and full-band designs spread capacity evenly despite uneven needs. Their BSCodec splits the spectrum into bands and compresses each on its own. Trained on a combined set of speech, music, and sound, it beat baselines on sound and music reconstruction while staying competitive on speech. Downstream task tests backed its promise as a general audio tokenizer.
Catalog record
| CREATOR | Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu, Shinji Watanabe |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu, Shinji Watanabe Abstract Neural audio codecs have recently enabled high-fidelity reconstruction at high compression rates, especially for speech. However, speech and non-speech audio exhibit fundamentally different spectral characteristics: speech energy concentrates in narrow bands around pitch harmonics (80-400 Hz), while non-speech audio requires faithful reproduction across the full spectrum, particularly preserving higher frequencies that define timbre and texture. This poses a challenge—speech-optimized neural codecs suffer degradation on music or sound. Treating the full spectrum holistically is suboptimal: frequency bands have vastly different information density and perceptual importance by content type, yet full-band approaches apply uniform capacity across frequencies without accounting for these acoustic structures. To address this gap, we propose BSCodec (Band-Split Codec), a novel neural audio codec architecture that splits the spectral dimension into separate bands and compresses each band independently. Experimental results demonstrate that BSCodec achieves superior reconstruction over baselines across sound and music, while maintaining competitive quality in the speech domain, when trained on the same combined dataset of speech, music and sound. Downstream benchmark tasks further confirm that BSCodec shows strong potential for use in downstream applications. 1 1 1 https://github.com/whr-a/espnet/tree/bscodec 1 Introduction Neural audio codecs (NACs) Mousavi et al. 2025 ; Guo et al. 2025 ; Shi et al. 2024b ; Défossez et al. 2022 ; Kumar et al. 2023 ; Zeghidour et al. 2021 have revolutionized audio compression and achieved high-fidelity reconstruction. These codecs generate discrete tokens that not only enable efficient transmission but also serve as representations for downstream audio understanding tasks Wang et al. 2023 ; Tian et al. 2025 . To enhance task-specific performance, recent work Zhang et al. 2023 ; Ye et al. 2025 has incorporated external semantic information into codec design, demonstrating improved results on speech-related benchmarks. Additionally, single-codebook approaches Ji et al. 2024 ; Xin et al. 2024 ; Jiang et al. 2025 have achieved excellent reconstruction quality and downstream performance on clean speech through carefully designed codebook structures, encoder-decoder architectures, and other components. Figure 1: Frequency domain energy distribution of speech, sound and music (extraction method in Appendix D ). The distributions exhibit significant structural differences across these three domains, demonstrating distinct spectral characteristics. However, the real acoustic world would contain components beyond speech, such as sound and music, all with vastly different acoustic characteristics. As illustrated in Figure 1 , speech, music, and sound exhibit substantially different average energy distributions across the frequency spectrum. While the aforementioned task-oriented codecs excel on clean speech, their performance degrades significantly Mousavi et al. 2025 when applied to music and sound domains. Given these spectral differences shown in Figure 1 , it is reasonable that speech codec designs struggle to generalize to music and sound. Current RVQ-based generalized codecs like DAC Kumar et al. 2023 still exhibit advantages Mousavi et al. 2025 in reconstruction quality for universal audio. While RVQ-based codecs demonstrate reasonable multi-domain performance, their design could be enhanced. The residual quantization hierarchy lacks explicit acoustic structure, with each layer quantizing residuals that do not correspond to interpretable acoustic attributes. This poses challenges for multi-domain compression: as different audio types have distinct energy distributions (Figure 1 ), a codec handling multiple domains faces a signal source with substantially higher entropy, demanding increased bitrate. Also RVQ applies uniform residual encoding across all domains, requiring the encoder to learn entangled representations for diverse content. Alternative quantization schemes like FSQ Mentzer et al. 2023 and GroupVQ Yang et al. 2023 partition along dimensional axes, but these divisions similarly lack grounding in the physical properties of audio signals. To effectively handle diverse audio domains, we need to account for their structural differences. As shown in Figure 1 , different domains exhibit substantially different energy distribution across frequency bands. Therefore, band-splitting offers a natural way to decouple these domain-specific structures by processing each frequency band independently. This approach has proven effective in other audio tasks Luo and Yu 2023 ; Yang et al. 2021 , and a RVQ-based multi-band codec has been successfully built for the speech domain Ng et al. 2025 , which motivate us to adopt similar decomposition principles for universal audio codec design. Motivated by these insights, we propose a band-split codec that decomposes input audio into separate frequency bands, each of which is processed independently and in parallel through dedicated encoder, quantizer, and decoder modules. While band-splitting provides a physically grounded design philosophy, its effective application to codec design requires careful consideration of band configuration and quantization allocation. We conduct extensive experiments to systematically investigate the impact of band count, frequency boundaries, and codebook design on multi-domain codec performance. Through principled design of band partitioning and quantization structure, our approach achieves strong performance across all three domains. Specifically, under the same training conditions, our 2.55 kbps and 3.83 kbps models achieve comparable overall performance to DAC at 4.5 kbps and 6 kbps respectively. Our contributions can be summarized as follows: • We propose BSCodec, a band-split codec that independently processes time-domain signals from separated frequency bands through parallel encoder-quantizer-decoder modules, achieving strong multi-domain performance through carefully designed band configuration and quantization allocation. • Comprehensive experiments on reconstruction quality and downstream tasks demonstrate that BSCodec achieves both perceptual superiority and enhanced effectiveness for audio understanding. 2 Related Work Neural Audio Codecs . The advent of deep learning has enabled significant advances in neural audio compression. They typically adopt the core architecture of RVQ-based discretization paired with GAN Goodfellow et al. 2014 -driven reconstruction, and further improve performance via targeted architectural enhancements or optimized training strategies Zeghidour et al. 2021 ; Défossez et al. 2022 ; Kumar et al. 2023 . Using a unified model remains challenging due to the distinct energy distribution of universal audio. Recent efforts have pursued universal codecs capable of handling diverse audio types. However, achieving strong multi-domain performance remains challenging due to the distinct acoustic characteristics across domains—speech exhibits narrow-band harmonic structure, music requires wide-band spectral richness, and sound encompasses diverse sound textures Scharf 1970 . Existing multi-domain codecs address this heterogeneity through various specialized mechanisms. DAC Kumar et al. 2023 employs balanced sampling strategies, ensuring each training batch contains data from all three domains (speech, music, sound) to prevent domain bias. Further works Jiang et al. 2025 ; Liu et al. 2024 ; Yang et al. 2025 introduce domain-specific codebooks via Mixture-of-Experts layers Shazeer et al. 2017 , semantic priors and MAE-derived representations He et al. 2022 ; Huang et al. 2022 , and more fine-grained supervision to further enhance the language modeling capability of universal audio codecs. In contrast, our work adopts a simpler design philosophy. We employ an architecture and training framework closely aligned with DAC Kumar et al. 2023 , yet achieve comparable performance for both reconstruction and downstream tasks at half the bitrate on the same dataset. This demonstrates that band-split decomposition offers a simple yet effective alternative for multi-domain codec design, obviating the need for complex domain-specific mechanisms while maintaining strong generalization across audio types. Figure 2: BSCodec architecture with band split, multi-band parallel generators and discriminators. Band-Splitting for Audio Processing . Band-splitting decomposes audio signals into separate frequency bands for