UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

Yuxuan, Peize, Haoyuan, Junzi · 2026

Read UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation by Yuxuan, Peize, Haoyuan, Junzi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21772.txt.

Description

A universal audio representation should capture fine speech cues and high-level meaning for environmental sounds and music in one encoder, yet existing encoders excel in one domain and degrade in others. Yuxuan Chen and three co-authors propose UniWhisper, an efficient continual multi-task training framework that casts varied audio tasks into a unified instruction-and-answer format. That format allows standard next-token training with no task-specific heads or losses.

Trained on 38 thousand hours of public audio, UniWhisper was judged with shallow MLP probes and k-nearest neighbors across 20 tasks spanning speech, environmental sound, and music. It beats Whisper on both probe types while retaining strong speech performance.

Catalog record

CREATORYuxuan, Peize, Haoyuan, Junzi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-21772

Browse the full library · About the Tower of Babel