Enhancing BEST-RQ Pseudo-Label Quality through Online Refinement for Automatic Speech Recognition
Jingjing, Zijian, Mohammad, Eugen, Ralf, Hermann · 2026
Read Enhancing BEST-RQ Pseudo-Label Quality through Online Refinement for Automatic Speech Recognition by Jingjing, Zijian, Mohammad, Eugen, Ralf, Hermann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30671.txt.
Description
Jingjing Xu and five colleagues improve the training labels behind a self-supervised speech method called BEST-RQ. Speech recognisers can learn from unlabelled audio by predicting pseudo-labels, targets the model generates itself. BEST-RQ creates its labels with a fixed quantiser, which is simple but gives weaker supervision than rivals that refine labels over rounds. The team wanted refinement's benefits without losing simplicity.
Three changes do the work. Principal component analysis replaces the quantiser's linear projection. That method finds the main axes of variation in data. An iterative scheme refines the codebook, the label inventory, during training. A second codebook is updated through distillation from the first. Pretraining used the Librispeech corpus, with one hundred supervised hours for fine-tuning. The full setup then cut word error rate on the test-other set by twelve percent. Error fell from about ten percent to under nine percent.
Catalog record
| CREATOR | Jingjing, Zijian, Mohammad, Eugen, Ralf, Hermann |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30671