BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition

Liuyuan Jiang, Xiaodong Cui, Brian Kingsbury, Tianyi Chen, Lisha Chen · 2025

Read BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition by Liuyuan Jiang, Xiaodong Cui, Brian Kingsbury, Tianyi Chen, Lisha Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15430.txt.

Introduction

Liuyuan Jiang leads a team that teaches speech models with less labeled data. Paired speech and text are costly to make, so models often learn from raw sound first. The catch is making good practice labels. Strong labels need extra models and long pipelines, while cheap labels are weak. The team wanted both speed and quality in one method.

Their method, BiRQ, reuses part of the model itself to make refined labels. A random mapping turns inner states into labels, while plain labels from the raw sound keep training steady. The whole task is set up as a two level problem that trains end to end. Tests show steady gains over the cheap method it builds on, at low cost.

Catalog record

CREATORLiuyuan Jiang, Xiaodong Cui, Brian Kingsbury, Tianyi Chen, Lisha Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15430

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition Liuyuan Jiang, Xiaodong Cui, Brian Kingsbury, Tianyi Chen, Lisha Chen Abstract Speech is a rich signal, and labeled audio–text pairs are costly to obtain, making self-supervised learning (SSL) essential for scalable representation learning. A core challenge in speech SSL is generating pseudo-labels that are both informative and efficient: strong labels, such as those used in HuBERT [ 1 ] , improve downstream performance but rely on external encoders and multi-stage pipelines, while efficient methods like BEST-RQ [ 2 ] achieve simplicity at the cost of weaker labels. We propose BiRQ, a bilevel SSL framework that combines the efficiency of BEST-RQ with the refinement benefits of HuBERT-style label enhancement. The key idea is to reuse part of the model itself as a pseudo-label generator: intermediate representations are discretized by a random-projection quantizer to produce enhanced labels, while anchoring labels derived directly from the raw input stabilize training and prevent collapse. Training is formulated as an efficient first-order bilevel optimization problem, solved end-to-end with differentiable Gumbel-softmax selection. This design eliminates the need for external label encoders, reduces memory cost, and enables iterative label refinement in an end-to-end fashion. BiRQ consistently improves over BEST-RQ while maintaining low complexity and computational efficiency. We validate our method on various datasets, including 960-hour LibriSpeech, 150-hour AMI meetings and 5,000-hour YODAS, demonstrating consistent gains over BEST-RQ. † † address: ⋆ University of Rochester † IBM Research ‡ Cornell University 1 Introduction Large language models (LLMs) have shown remarkable effectiveness, enabled by abundant discrete text data [ 3 ] . In contrast, speech is continuous, and labeled audio–text pairs are scarce, making supervised auto-speech recognition (ASR) training challenging. Self-supervised learning (SSL) addresses this by discretizing signals into pseudo-labels (PLs) and training models to capture hierarchical information from low-level acoustics to higher-level lexical and semantic knowledge [ 4 , 1 , 2 ] . These pretrained models can then be fine-tuned on downstream tasks like ASR with limited data, and their discrete representations can be aligned with text tokens via lightweight modules such as Q-formers for integration with LLMs [ 5 ] . Figure 1: Overview of BiRQ: a bilevel SSL framework combining anchoring labels (lower-level) for stability and enhanced labels (upper-level) for improved representation learning. The encoder, parameterized by \theta , is assumed to be a generic multi-layer network (e.g., Transformer, Conformer). The green arrows show the forward pass taking input x . The red arrows show the pipeline to generate enhanced labels y^{(k)}(\theta;x) , with intermediate representations taken from the first k layers and quantized via a random-projection quantizer. The blue arrows show a parallel process in which random quantization on the raw input x generates anchoring labels y(x) . Existing SSL approaches for speech typically involve two key components: (1) generation of PLs and (2) representation learning with a masked prediction paradigm [ 3 ] . For (1), the method used strongly affects performance, but often comes with efficiency tradeoffs. For instance, wav2vec 2.0 [ 4 ] uses a multi-layer convolutional encoder and a product quantizer to discretize latent representations. HuBERT [ 1 ] applies a separate acoustic unit discovery module (e.g., k -means clustering) to iteratively refine pseudo-labels from learned representations. More recent work in computer vision with joint embedding methods [ 6 ] similarly depends on complex label encoders. For (2), the choice of model architecture for representation learning is crucial for downstream task performance. Recent advances in ASR encoders, such as the Conformer [ 7 ] , have demonstrated strong recognition accuracy, yet they are not inherently equipped with modules for pseudo-label generation, making them less directly compatible with methods like wav2vec 2.0 [ 4 ] and HuBERT [ 1 ] . A more recent work, BEST-RQ [ 2 ] , demonstrates strong performance on ASR benchmarks. It achieves efficiency in pseudo-label generation via a random quantizer and combines well with strong encoder architectures such as the Conformer [ 7 ] , which underpins many state-of-the-art ASR systems. However, insights from other SSL baselines show that the quality of pseudo-labels significantly impacts performance. For instance, HuBERT [ 1 ] demonstrates that using a powerful label encoder and a multi-stage training process, which refines targets from the model’s learned representations, can lead to substantial performance improvements. Building on these insights, we improve upon BEST-RQ [ 2 ] by integrating an iterative target refinement strategy [ 8 , 1 ] , resulting in higher-quality pseudo-labels while preserving end-to-end learning as well as computational and memory efficiency . This leads to our proposed Bi level self-labeling R andom Q uantization (BiRQ) method for speech SSL, as illustrated in Figure 1 . Given a multi-layer acoustic encoder of general architecture and an unmasked speech input sample, BiRQ uses part of the encoder itself as a label encoder to generate an intermediate representation, which is then discretized with a random-projection quantizer to form the enhanced label . The encoder is then trained, under the mask-and-predict paradigm, to reconstruct randomly masked regions of input using these labels. To avoid degenerate solutions, an additional random quantizer generates an anchoring label directly from the raw input. Training thus follows a Bi-Level Optimization (BLO) principle: the encoder learns from enhanced labels (upper-level) while staying consistent with anchoring labels (lower-level). See Section 3 for details. Given trainable multi-layer encoder \theta and input unlabeled data x , this BLO framework allows the encoder to leverage its own intermediate representations as pseudo labels y^{(k)}(\theta;x) for supervision, while the raw-input labels y(x) , produced without any external clustering or auxiliary encoder, anchor training, and guard against collapse. To make the training procedure time-efficient and practical for large-scale problems, we develop a fully single-loop first-order optimization algorithm. As a by-product, since BiRQ reuses part of the model (e.g., layers 1 - k ) as the label encoder, it does not need a separate pseudo-label encoder and is thus memory-efficient. Finally, the random-projection quantizer keeps label generation lightweight (no costly k-means or additional encoder) and, together with differentiable Gumbel-softmax selection, enables end-to-end training that propagates useful learning signals from the mask-and-predict objective while maintaining stability through the anchoring constraint. We evaluate BiRQ using various Conformer configurations and sizes on multiple datasets and find that BiRQ consistently outperforms BEST-RQ at similar computational cost and low complexity. 2 Related Work Self-supervised speech recognition. The earliest approaches to speech SSL used generative models trained with likelihood maximization [ 9 ] . Prediction-based self-supervised learning has recently gained more attention. Early methods exploited the sequential nature of speech, predicting future frames based on past observations [ 10 , 11 ] . Recently, random mask-and-predict training, which encourages the model to reconstruct masked portions of the input, has demonstrated strong performance and gained widespread interest [ 12 , 4 , 1 , 2 ] . One line of work generates pseudo labels on learned continuous representations, requiring either a model architecture with a representation-learning module or external label encoders [ 12 , 11 , 4 , 1 , 13 ] . However, such approaches often introduce additional computational and memory overhead, since training requires either maintaining a separate representation-learning module or an external label encoder, making them less compatible with recent ASR architectures such

Read the complete text