VibeVoice-ASR Technical Report
Zhiliang Peng, Jianwei Yu , Yaoyao Chang , Zilong Wang , Li Dong Yingbo Hao, Yujie Tu, Chenyu Yang, Wenhui Wang, Songchen Xu, Yutao Sun Hangbo Bao, Weijiang Xu, Yi Zhu, Zehua Wang, Ting Song, Yan Xia, Zewen Chi, Shaohan Huang, Liang Wang, Chuang Ding, Shuai Wang, Xie Chen, Furu WeiMicrosoft Research · 2026
Read VibeVoice-ASR Technical Report by Zhiliang Peng, Jianwei Yu , Yaoyao Chang , Zilong Wang , Li Dong Yingbo Hao, Yujie Tu, Chenyu Yang, Wenhui Wang, Songchen Xu, Yutao Sun Hangbo Bao, Weijiang Xu, Yi Zhu, Zehua Wang, Ting Song, Yan Xia, Zewen Chi, Shaohan Huang, Liang Wang, Chuang Ding, Shuai Wang, Xie Chen, Furu WeiMicrosoft Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18184.txt.
Introduction
Zhiliang Peng and a large team present VibeVoice-ASR, a speech system built for long recordings. Their 2026 preprint is a technical report. Short-clip speech recognition has improved fast. Long audio, like meetings and podcasts, still trips systems up. Context breaks into pieces across a long file, and many speakers trade turns. The framework builds on the team's earlier VibeVoice work to handle those strains.
A general-purpose design aims to serve many kinds of long audio at once. Keeping track of who said what, and what was said an hour earlier, is the hard part. Progress here serves transcription, search, and meeting tools. The report lays out the system for other builders.
Catalog record
| CREATOR | Zhiliang Peng, Jianwei Yu , Yaoyao Chang , Zilong Wang , Li Dong Yingbo Hao, Yujie Tu, Chenyu Yang, Wenhui Wang, Songchen Xu, Yutao Sun Hangbo Bao, Weijiang Xu, Yi Zhu, Zehua Wang, Ting Song, Yan Xia, Zewen Chi, Shaohan Huang, Liang Wang, Chuang Ding, Shuai Wang, Xie Chen, Furu WeiMicrosoft Research |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18184
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VibeVoice-ASR Technical Report Zhiliang Peng, Jianwei Yu , Yaoyao Chang , Zilong Wang , Li Dong Yingbo Hao, Yujie Tu, Chenyu Yang, Wenhui Wang, Songchen Xu, Yutao Sun Hangbo Bao, Weijiang Xu, Yi Zhu, Zehua Wang, Ting Song, Yan Xia, Zewen Chi, Shaohan Huang, Liang Wang, Chuang Ding, Shuai Wang, Xie Chen, Furu WeiMicrosoft Research https://aka.ms/GeneralAI Abstract This report presents VibeVoice-ASR , a general-purpose speech understanding framework built upon VibeVoice [ 13 ] , designed to address the persistent challenges of context fragmentation and multi-speaker complexity in long-form audio (e.g., meetings, podcasts) that remain despite recent advancements in short-form speech recognition. Unlike traditional pipelined approaches that rely on audio chunking, VibeVoice-ASR supports single-pass processing for up to 60 minutes of audio. It unifies Automatic Speech Recognition, Speaker Diarization, and Timestamping into a single end-to-end generation task. In addition, VibeVoice-ASR supports over 50 languages, requires no explicit language setting, and natively handles code-switching within and across utterances. Furthermore, we introduce a prompt-based context injection mechanism that allows users to supply customized conetxt, significantly improving accuracy on domain-specific terminology and polyphonic character disambiguation. Code : github.com/microsoft/VibeVoice Hugging Face : microsoft/VibeVoice Demo : aka.ms/VibeVoice-ASR Figure 1 : VibeVoice-ASR sets a new state-of-the-art for long-form speech understanding, consistently outperforming strong closed-source multimodal models (Gemini-2.5/3-Pro) across five public benchmarks. The results demonstrate superior accuracy in both speaker attribution (DER) and time-aligned transcription (tcpWER), particularly in complex multi-speaker environments. 1 Introduction Recent years have witnessed a paradigm shift in speech processing, driven by the integration of Large Language Models (LLMs) with acoustic encoders [ 5 ] . While these large audio models have achieved remarkable success in short-form speech recognition, transcribing and analyzing long-form audio—such as hour-long meetings, podcasts, and academic lectures—remains a formidable challenge. The prevailing approach to long-form audio involves cascaded pipelines that segment continuous speech into short clips (typically <30 seconds) for independent processing [ 8 , 1 , 2 ] . While practical, this "divide-and-conquer" strategy suffers from two fundamental limitations: Context Fragmentation and Pipeline Complexity. First, independently processing segments severs global semantic dependencies, causing the model to lose track of cross-sentence context, which is fatal for disambiguating homophones or resolving coreferences in extended dialogue. Second, traditional systems treat Automatic Speech Recognition (ASR), Speaker Diarization, and Timestamping as separate tasks managed by disjoint models. Reconciling their outputs often requires complex heuristics, leading to error propagation where a failure in segmentation or diarization corrupts the final transcript. To bridge this gap, we introduce VibeVoice-ASR , a unified, general-purpose framework designed for high-fidelity long-form speech understanding. Built upon the VibeVoice architecture [ 13 ] , our system fundamentally abandons the sliding-window paradigm in favor of a single-pass approach. By leveraging an ultra-low frame rate tokenizer ( 7.5 Hz), VibeVoice-ASR compresses an hour of audio into a sequence length that fits comfortably within the context window of modern LLMs. This allows the model to attend to the entire global context of a 60-minute session simultaneously, ensuring semantic coherence and consistent speaker tracking without the need for external clustering algorithms. Concurrent with the development of VibeVoice-ASR , a number of related research efforts have emerged [ 9 , 21 , 15 , 22 ] . Nevertheless, the majority of these works have not made their models publicly available. VibeVoice-ASR reformulates long-form transcription as an end-to-end generation task, as shown in Figure 2 . Instead of outputting plain text, it generates a structured Rich Transcription stream that explicitly interleaves speaker identities (“Who"), precise timestamps (“When"), and speech content (“What"). Furthermore, acknowledging the diverse needs of real-world applications, we introduce a prompt-based context injection mechanism. This allows users to supply customized context—ranging from hotword lists to background descriptions—significantly enhancing the model’s ability to recognize domain-specific terminology and handle complex code-switching scenarios. Figure 2 : The architectural overview of VibeVoice-ASR . VibeVoice-ASR processes 60-minute long-form audio in a single pass by ingesting continuous latents from dual-tokenizers alongside optional user-provided context. The output is a generated stream of Rich Transcription, explicitly interleaving Speaker ID (Who), Timestamps (When), and Content (What) 2 Method 2.1 Overview Figure 2 presents the architectural overview of VibeVoice-ASR . We formulate long-form speech understanding as a language modeling task. The model takes a sequence of continuous audio embeddings, encoded from from the pre-trained Acoustic and Semantic encoders, as its primary input. To enable context-aware capabilities, optional text prompts (e.g., hotwords or background information) can be prepended to the audio sequence. These inputs are processed by a decoder-only Large Language Model backbone (e.g., Qwen 2.5 [ 20 ] ) to autoregressively generate the target sequence. Distinct from conventional ASR models that output plain text, VibeVoice-ASR is designed to produce a Rich Transcription . As illustrated in the output stream of Figure 2 , the model generates a structured sequence that explicitly interleaves speaker identity (“Who”), temporal boundaries (“When”), and speech content (“What”), enabling simultaneous recognition, diarization, and timestamping in a single pass. 2.2 Speech Tokenizer In this work, we directly employ the pre-trained dual-tokenizers from VibeVoice [ 13 ] , which integrates an Acoustic Tokenizer for spectral fidelity and a Semantic Tokenizer for linguistic alignment. The Acoustic tokenizer, inspired by \sigma -VAE [ 17 ] , applies a hierarchical design with a cumulative 3200\times downsampling rate to the 24 kHz input, yielding an extremely compact representation of approximately 7.5 tokens per second. Meanwhile, the Semantic module extracts deterministic content features aligned with textual semantics. Note we only use tokenizer encoders here. This ultra-low frame rate is pivotal, as a one-hour continuous audio session translates to: 3600\text{ seconds}\times 7.5\text{ tokens/sec}=27,000\text{ tokens}, (1) which fits comfortably within the single-pass context window of modern LLMs. 2.3 VibeVoice-ASR 2.3.1 Pre-training We use the data processing pipeline proposed in VibeVoice [ 13 , 24 ] to obtain the initial data corpus. The pre-training data distribution can be found in Figure 3 . The pipeline consists of three stages: segmentation and transcription, diarization, and quality filtering. Long recordings are first segmented using Silero voice activity detection (VAD) into clips of up to 30 seconds, followed by transcription with Whisper-large-v3-turbo [ 14 ] to obtain punctuated text and word-level timestamps; segment boundaries are further refined by splitting at punctuation end timestamps (e.g., [.?!] ) to better align with speaker turns. Speech diarization is then performed using the vblinkp model from the WeSpeaker toolkit [ 18 ] , where speaker embeddings are extracted from overlapping frames (1.5 s window, 0.75 s hop), clustered with HDBSCAN [ 3 ] , and refined by merging clusters whose centroids have a cosine similarity greater than 0.67, yielding final speaker turn annotations. Finally, to ensure annotation reliability, segments are re-transcribed using a secondary ASR model [ 19 ] , and recordings are discarded if more than 30% of segments have a WER exceeding 20%, if speech accounts for less than 60% of the total duration. To ensure the effectiveness of the data processing pipeline, we conducted a comparative study between our pipeline and two widely adopted audio processing pipelines, WhisperX [ 1 ] and Emilia [ 8 ] . The evaluation is performed on three commonly used public multi-speaker meeting datasets—AMI [ 4 ] , AliMeeting [ 23 ] , and AISHELL-4 [ 7 ] —and reports both diarization