UniVerse-1: Uni fied Audio- V ideo Gen er ation via S titching of E xperts
Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, Gang Yu, StepFun, The Hong Kong University of Science, Technology(GuangZhou), The Hong Kong University of Science, Technology Tsinghua University · 2025
Read UniVerse-1: Uni fied Audio- V ideo Gen er ation via S titching of E xperts by Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, Gang Yu, StepFun, The Hong Kong University of Science, Technology(GuangZhou), The Hong Kong University of Science, Technology Tsinghua University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06155.txt.
Introduction
One model that generates video and its soundtrack together is introduced in a 2025 paper led by Duomin Wang with nine co-authors. UniVerse-1 targets coordinated ambient sound and speech, in the style of leading closed systems. Instead of training from scratch, the team stitches together pretrained video and music generation experts, fusing corresponding blocks deeply.
An online annotation pipeline labels training data during training itself, keeping sound and picture aligned in time and avoiding the damage stale text labels cause. After fine-tuning on thousands of hours of audio-video material, the model produces well-coordinated ambient sound and strongly aligned speech. A new benchmark, Verse-Bench, supports systematic evaluation, and the model and code are released publicly.
Catalog record
| CREATOR | Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, Gang Yu, StepFun, The Hong Kong University of Science, Technology(GuangZhou), The Hong Kong University of Science, Technology Tsinghua University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06155
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UniVerse-1: Uni fied Audio- V ideo Gen er ation via S titching of E xperts Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, Gang Yu, StepFun, The Hong Kong University of Science, Technology(GuangZhou), The Hong Kong University of Science, Technology Tsinghua University Abstract We introduce UniVerse-1 , a unified, Veo-3-like model capable of simultaneously generating coordinated audio and video. To enhance training efficiency, we bypass training from scratch and instead employ a stitching of experts (SoE) technique. This approach deeply fuses the corresponding blocks of pre-trained video and music generation experts models, thereby fully leveraging their foundational capabilities. To ensure accurate annotations and temporal alignment for both ambient sounds and speech with video content, we developed an online annotation pipeline that processes the required training data and generates labels during training process. This strategy circumvents the performance degradation often caused by misalignment text-based annotations. Through the synergy of these techniques, our model, after being finetuned on approximately 7,600 7,600 hours of audio-video data, produces results with well-coordinated audio-visuals for ambient sounds generation and strong alignment for speech generation. To systematically evaluate our proposed method, we introduce Verse-Bench , a new benchmark dataset. In an effort to advance research in audio-video generation and to close the performance gap with state-of-the-art models such as Veo3, we make our model and code publicly available. We hope this contribution will benefit the broader research community. Project page: https://dorniwang.github.io/UniVerse-1/ . 1 Introduction The era of diffusion models song2020score ; song2020improved ; lipman2022flow has culminated in the rise of Diffusion Transformer (DiT) architectures peebles2023scalable , exemplified by landmark models like Sora openaisora2024 and its open-source counterparts kong2024hunyuanvideo ; yang2024cogvideox ; wan2025wan . These models, leveraging unprecedented scales of data and computation, have achieved remarkable quality and prompt alignment in video generation. This success is catalyzing a profound transformation across creative industries and has spurred a wave of research into downstream applications such as talking head synthesis wang2023progressive ; yu2023talking ; wang2025fantasytalking ; luo2025dreamactor and human animation tan2024animate ; chen2025hunyuanvideo ; lin2025omnihuman , which now offer viable real-world solutions. However, this rapid progress has been almost exclusively confined to the visual domain, treating video as a silent movie. This unimodal focus represents a fundamental bottleneck, as it ignores the inherently multimodal nature of video. Post-hoc video-to-audio models shan2025hunyuanvideo serve as a superficial fix, but they are inherently limited; while capable of adapting audio to existing visual content, they fail to enforce temporal alignment in the reverse direction. This makes critical tasks, such as synchronizing lip movements with speech, impossible. While closed-source systems like Google’s Veo3 veo32025 have demonstrated synchronous audio-video generation, the lack of publicly available technical details leaves a critical gap in open research. To bridge this gap between closed-source systems and open research, we introduce UniVerse-1 : a unified, fully open-source, Veo-3-like model capable of simultaneously generating coordinated audio and video. Our work is underpinned by several key technical contributions designed to address the unique challenges of bimodal generation. Instead of the costly process of training a new model from scratch, we propose a novel and efficient stitching of experts (SoE) paradigm. This methodology effectively fuses a state-of-the-art video generation model, WAN2.1 wan2025wan , with a music generation model, Ace-step gong2025ace . The core of this fusion lies in lightweight, cross-modal MLP connectors introduced within corresponding blocks of each model. These connectors facilitate bidirectional interaction between modalities, and we found this strategy to significantly accelerate training convergence by leveraging the powerful priors of the pre-trained experts. Furthermore, we tackle the critical challenge of data alignment in bimodal training. We argue that static, pre-processed annotations are a flawed paradigm for tasks requiring precise temporal consistency. To address this, we developed an online annotation pipeline that generates labels dynamically during training. This approach ensures strict temporal and semantic alignment between audio-video data and their textual descriptions, mitigating the performance degradation caused by static misalignment. During our investigation, we also uncovered a crucial, yet overlooked, factor in bimodal diffusion modeling: cross-modal noise correlation . We identified that the standard pseudo-random number generation process hamming1952mathematical can introduce spurious correlations between the noise vectors for video and audio, which subsequently degrades audio quality during inference. Our solution involves ensuring independent noise sampling for each modality. To support this work, we curated a high-quality dataset comprising approximately 7,600 hours of precisely aligned audio-video content. To systematically evaluate our method, we also propose Verse-Bench , a new benchmark featuring 600 image-text prompt pairs covering a diverse range of sound categories. Highlighting its versatility, Verse-Bench supports not only joint audio-video generation but also unidirectional tasks, including a specialized Verse-Ted subset designed for evaluating audio-to-video synthesis. In summary, our primary contributions are: • An Open-source Audio-Video Foundation Model: We present UniVerse-1, a novel, open-source model capable of producing highly coherent and well-aligned synchronous audio-visual content, closing a critical gap in the open-source community. • A Novel Methodology for Joint Audio-Video Generation: We propose a comprehensive methodology to enable efficient and high-quality joint audio-video synthesis. This is achieved through three key innovations: a stitching of experts (SoE) paradigm to accelerate convergence by fusing pre-trained models; an online data annotation pipeline to solve the critical static misalignment problem in training; and the identification and mitigation of the previously overlooked cross-modal noise correlation issue, a crucial factor for generation quality. • A Comprehensive Evaluation Benchmark: We propose Verse-Bench, a new benchmark designed to comprehensively evaluate joint audio-video generation models across a diverse set of tasks. 2 Related Works Video Diffusion Models The field of video generation was revolutionized by the introduction of diffusion models, with pioneering works like AnimateDiff guo2023animatediff and Video Diffusion Models ho2022video marking the beginning of this new era. This initial wave of research was further advanced by models such as Stable Video Diffusion blattmann2023stable , which first demonstrated that curating large-scale, high-quality datasets is critical for enhancing model performance. A common characteristic of these early models was their reliance on UNet architectures. To mitigate the challenges posed by the limited availability and quality of video data compared to images, these models were typically fine-tuned from pre-trained UNet image foundations. A significant paradigm shift occurred with the introduction of Sora openaisora2024 , which heralded a new age defined by the Diffusion Transformer (DiT) architecture peebles2023scalable and training on massive, high-quality video corpora. This breakthrough spurred a proliferation of subsequent research. CogVideox yang2024cogvideox was the first to release an open-source DiT-based model, providing a significant catalyst for community-driven innovation. This was followed by other notable open-source models such as HunyuanVideo kong2024hunyuanvideo , WAN2.1 wan2025wan , and Step-Video ma2025step , as well as high-performing closed-source systems including Kling kuaishou2024kling , SeeDance 1.0 gao2025seedance , Movie Gen polyak2024movie , and Veo2 veo22025 . Architecturally, these contemporary models converge on a common blueprint. They employ a 3D Variational Autoencoder (VAE) to achieve spatio-temporal compression of video into a latent space. The core generative process is then handled by a DiT, which learns to denoise these noisy latents. Across these state-of-the-art models, the quality and scale of the training data have been identified as paramount factors, making data curation and processing a central component of their development. Audio Diffusion