SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland, University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University, Shanghai Jiao Tong University · 2025

Read SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations by Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland, University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University, Shanghai Jiao Tong University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25955.txt.

Introduction

Xiaoyu Yang leads seven co-authors in a self supervised framework that learns both speech and general audio in one model. Self supervised learning trains on raw data without labels, using parts of the signal as targets. Most prior models specialize in either speech or sound events, which leaves a gap between the two domains.

Their system, SPEAR, distills knowledge from a speech focused teacher and a general audio teacher into one student model. Continuous teacher outputs are turned into discrete tokens through multi codebook quantization, which splits the code into several parts. The student predicts both token sets from masked input, with a token mixing step for complex scenes. Results set a new best on the SUPERB speech benchmark in their tests.

Catalog record

CREATORXiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland, University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University, Shanghai Jiao Tong University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland, University of Cambridge, Shanghai Artificial Intelligence Laboratory, Tsinghua University, Shanghai Jiao Tong University Abstract Self-Supervised Learning (SSL) excels at learning generic representations of acoustic signals, yet prevailing methods remain domain-specific, tailored to either speech or general audio, hindering the development of a unified representation model with a comprehensive capability over both domains. To address this, we present SPEAR (SPEech and Audio Representations), the first SSL framework to successfully learn unified speech and audio representations from a mixture of speech and audio data. SPEAR proposes a unified pre-training objective based on masked prediction of fine-grained discrete tokens for both speech and general audio. These tokens are derived from continuous speech and audio representations using a Multi-codebook Vector Quantisation (MVQ) method, retaining rich acoustic detail essential for modelling both speech and complex audio events. SPEAR is applied to pre-train both single-domain and unified speech-and-audio SSL models. Our speech-domain model establishes a new state-of-the-art on the SUPERB benchmark, a speech processing benchmark for SSL models, matching or surpassing the highly competitive WavLM Large on 12 out of 15 tasks with the same pre-training corpora and a similar model size. Crucially, our unified model learns complementary features and demonstrates comprehensive capabilities across two major benchmarks, SUPERB and HEAR, for evaluating audio representations. By further scaling up the model size and pre-training data, we present a unified model with 600M parameters that excels in both domains, establishing it as one of the most powerful and versatile open-source SSL models for auditory understanding. The inference code and pre-trained models will be made publicly available. 1 Introduction Auditory signals, encompassing speech and general audio, are a fundamental medium for human perception and interaction ( Bregman 1994 ) . While speech conveys linguistic and paralinguistic information for communication, general audio provides the environmental context and sound events crucial for situational awareness. Developing a unified encoder with comprehensive capabilities across both domains is therefore a significant goal for modern AI systems ( Bommasani 2021 ) . Self-supervised learning (SSL) has emerged as an effective paradigm for learning generic representations in the field of speech and audio processing ( Baevski et al. 2020 ; Baevski et al. 2022 ; Huang et al. 2022 ; Dinkel et al. 2024 ) . By leveraging a large amount of unlabelled data during pre-training, SSL models can achieve very high performance on tasks with limited supervised data ( Chen et al. 2020b ; Baevski et al. 2020 ) . In speech processing, one dominant approach is masked token prediction ( Hsu et al. 2021 ; Chung et al. 2021 ; Chiu et al. 2022 ) . This approach involves quantisation techniques, such as k-means clustering ( Hsu et al. 2021 ) or random projection quantisation ( Chiu et al. 2022 ) , to generate coarse-grained discrete tokens from the raw speech signal or intermediate SSL representations. Although the quantisation process discards some acoustic details, such as prosody and paralinguistic information ( Xin et al. 2024 ) , these tokens still effectively capture the phonetic content of speech ( Baevski et al. 2020 ; Hsu et al. 2021 ) . This makes the pre-training strategy highly suitable for the speech domain and has enabled models to achieve excellent performance on a wide range of downstream speech processing tasks ( Chen et al. 2022 ; Yang et al. 2021 ) . However, such coarse-grained tokens are inadequate for general audio. In contrast to speech, general audio exhibits more irregular and complex spectro-temporal patterns ( Attias & Schreiner 1996 ) that are difficult to capture with coarse-grained discrete units produced by methods like k-means. While BEATs ( Chen et al. 2023 ) adapts discrete token prediction for audio, it requires a complex iterative process to train a dedicated audio tokeniser, substantially increasing training complexity. Consequently, most SSL models for general audio forgo token prediction in favour of other objectives, such as masked autoencoder (MAE) ( Huang et al. 2022 ; Dinkel et al. 2024 ) or bootstrapping-based methods ( Li et al. 2024a ; Chen et al. 2024 ) . This divergence in pre-training objectives has prevented the emergence of a unified SSL approach for both the speech and general audio domains. To address this lack of unification, we propose SPEAR (SPEech and Audio Representations), a unified SSL framework based on masked prediction of fine-grained discrete tokens for both speech and general audio. Our approach applies multi-codebook vector quantisation (MVQ) ( Guo et al. 2023 ) to the intermediate representations of existing speech and audio SSL models to obtain fine-grained discrete tokens, which then serve as the targets for a masked prediction objective. MVQ decomposes the representation space into multiple subspaces spanned by parallel codebooks. This multi-codebook design enables MVQ tokens to retain far more detail than coarse-grained discrete units, making SPEAR suitable for both speech and general audio. The pre-training objective can be viewed as performing multiple masked language modelling ( Devlin et al. 2019 ) tasks simultaneously, one for each MVQ codebook. A key aspect of SPEAR is the joint pre-training on speech and audio, where the model learns to predict two sets of MVQ tokens derived from separate expert models from two domains. Together with a specially designed asymmetrical pre-training strategy, the dual-target objective enables SPEAR to learn a single, unified representation space bridging both domains. Finally, to enhance the model’s versatility for tasks requiring different temporal granularities ( Shi et al. 2024 ) , we integrate a multi-temporal resolution encoder ( Yao et al. 2024 ) , allowing the model to process the input signal at variable frame rates in intermediate layers. Our contributions can be highlighted as follows: • We propose SPEAR, the first unified SSL framework for both speech and general audio that successfully learns high-quality unified representations for both domains. • We conduct extensive experiments and validate the effectiveness of SPEAR in both single-domain and unified settings. Notably, our speech-domain model achieves the same or better performance than the competitive WavLM Large ( Chen et al. 2022 ) on 12 out of 15 SUPERB ( Yang et al. 2021 ; Tsai et al. 2022 ) tasks under a fair comparison. Our audio model approaches the best-performing SSL models on the HEAR ( Turian et al. 2022 ) benchmark, and even outperforms them on environment-related tasks while using far less data. • We demonstrate that SPEAR unifies joint speech and audio pre-training, resulting in a model with a comprehensive capability over both domains. Furthermore, we show that this synergy is enhanced when scaling up model parameters and pre-training data. Inference code and pre-trained models will be made open-source to facilitate future research. 2 Related Work SSL for Speech and Audio As introduced in Section 1 , SSL has been widely adopted for learning generic representations in both the speech domain ( Baevski et al. 2020 ; Hsu et al. 2021 ; Chen et al. 2022 ) and audio domain ( Huang et al. 2022 ; Dinkel et al. 2024 ) . Yet, existing SSL methods are typically domain‑specific, and a unified framework for both

Read the complete text