Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation
Kang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu, Arda Senocak, Joon Son Chung · 2025
Read Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation by Kang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu, Arda Senocak, Joon Son Chung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24103.txt.
Introduction
Kang Zhang and five co-authors present MGAudio, a system that makes sound for open domain video. The task is to create audio that fits what is seen, with good timing and realistic quality. Older methods lean on outside guidance during generation, which can limit how well sight and sound line up.
MGAudio uses a flow based transformer model that learns to guide itself through a special training goal. Its audio visual encoder both conditions the generation and aligns features across sight and sound. On the VGGSound benchmark, the system reached a top score, cutting the main error measure to a low value and beating prior best results by a wide margin.
Catalog record
| CREATOR | Kang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu, Arda Senocak, Joon Son Chung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation Kang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu, Arda Senocak, Joon Son Chung Abstract We present MGAudio , a novel flow-based framework for open-domain video-to-audio generation, which introduces model-guided dual-role alignment as a central design principle. Unlike prior approaches that rely on classifier-based or classifier-free guidance, MGAudio enables the generative model to guide itself through a dedicated training objective designed for video-conditioned audio generation. The framework integrates three main components: (1) a scalable flow-based Transformer model, (2) a dual-role alignment mechanism where the audio-visual encoder serves both as a conditioning module and as a feature aligner to improve generation quality, and (3) a model-guided objective that enhances cross-modal coherence and audio realism. MGAudio achieves state-of-the-art performance on VGGSound, reducing FAD to 0.40, substantially surpassing the best classifier-free guidance baselines, and consistently outperforms existing methods across FD, IS, and alignment metrics. It also generalizes well to the challenging UnAV-100 benchmark. These results highlight model-guided dual-role alignment as a powerful and scalable paradigm for conditional video-to-audio generation. Code is available at: https://github.com/pantheon5100/mgaudio Figure 1: V2A on VGGSound ( chen2020vggsound ) . MGAudio attains the best FAD among video-to-audio methods with full data and 1.1M iters, and remains competitive even with only 10% data and 300k iters, highlighting its strong data efficiency. 1 Introduction Vision-guided audio generation is gaining increasing attention due to its critical role in Foley sound synthesis for video and film production ( ament2021foley ) . In particular, video-to-audio (V2A) generation has emerged as a key task, not only for enriching silent videos produced by emerging text-to-video models ( blattmann2023align ; khachatryan2023text2video ; huang2024free ; ouyang2024flexifilm ) , but also for enhancing realism and immersion in professional video editing workflows. Realistic audio is essential for audio-visual coherence and immersive experience, but generating sound semantically aligned and temporally synchronized with video remains challenge. Prior works have explored GAN-based ( chen2020generating ) and autoregressive Transformer-based models ( iashin2021taming ) , which often struggle with synchronization and semantic consistency. More recent methods adopt diffusion models to improve generation quality. For example, Diff-Foley ( luo2023diff ) uses a contrastively trained video encoder to extract visual conditions, which are then used to guide a diffusion model for audio generation. Other approaches, such as See and Hear ( xing2024seeing ) and FoleyCrafter ( zhang2024foleycrafter ) , leverage large pretrained models to achieve high-fidelity results, but at the cost of massive model size (billions of parameters). In contrast, models like MDSGen ( pham2025mdsgen ) , FRIEREN ( wang2024frieren ) , and MMAudio ( cheng2025taming ) (131–159M parameters) explore more lightweight denoising or flow-matching ( lipman2023flow ) objectives, achieving competitive performance across fidelity and alignment metrics. Despite architectural differences, diffusion- and flow-matching–based methods typically rely on classifier-free guidance (CFG) to improve generation quality. This involves randomly dropping conditioning signals during training to simulate both conditional and unconditional objectives. While effective, this multi-task setup may dilute the model’s capacity and lead to mismatched sampling behavior at inference. To address the limitations of classifier-free guidance (CFG) in audio generation, we propose MGAudio , a novel model-guided framework that replaces CFG with direct model-based supervision during training. Our approach builds on the Model-Guided (MG) framework tang2025diffusion , originally introduced for class-conditional image generation on ImageNet, and effectively extends and adapts it to the video-to-audio domain. While MG is specified for discrete class labels, its application to conditional generation from continuous video inputs remains unexplored. To this end, MGAudio introduces a dual-role audio-visual encoder: one branch modulates the diffusion process via LayerNorm-based conditioning, while the other provides intermediate alignment signals to guide audio synthesis at multiple denoising steps. We show that MGAudio (131M) achieves state-of-the-art performance on VGGSound ( chen2020vggsound ) , with a FAD of 0.40. Notably, the same model generalizes robustly to the UnAV-100 benchmark ( geng2023dense ) without any fine-tuning, demonstrating strong cross-dataset transferability. Furthermore, when trained on just 10% of the VGGSound, it outperforms existing methods trained on the full dataset across key metrics such as FAD (Fig. 1 ). These results demonstrate that model-guided training not only enables more efficient learning but also offers a compelling alternative to CFG for scalable and effective audio generation. Our key contributions are: • We introduce MGAudio , a novel framework for video-to-audio generation that replaces the conventional classifier-free guidance (CFG) objective with a model-guidance (MG) objective, leading to more efficient training and improved generation quality. • We propose a dual-role alignment mechanism that uses a shared jointly trained visual and audio encoder for conditioning and intermediate representation alignment, enabling more effective video-audio feature integration. • MGAudio sets a new state-of-the-art on VGGSound, achieving an FAD of 0.4 with 131M parameters, and maintains strong performance with only 10% of the training data (FAD = 0.8), highlighting its efficiency and scalability. • Extensive experiments and analyses on VGGSound and UnAV-100 benchmarks demonstrate that the model-guidance objective, combined with dual-role alignment, significantly improves data efficiency and generalization over prior methods. 2 Related Works 2.1 Classifier-Free Guidance Learning for Video-Guided Sound Generation Early works such as SpecVQGAN ( iashin2021taming ) and Im2Wav ( sheffer2023hear ) employ autoregressive Transformers to generate audio tokens conditioned on video or CLIP features. However, their sequential decoding results in slow inference and weak cross-modal alignment, limiting their effectiveness in open-domain scenarios. Recent advances in generative modeling have led to powerful diffusion-based methods that show strong performance across diverse application domains ( pham2024cross ; niu2024acdmsr ) . Diff-Foley ( luo2023diff ) introduces a two-stage pipeline combining contrastive video-audio pretraining with latent diffusion for more efficient generation. This has inspired follow-up work ( xing2024seeing ; zhang2024foleycrafter ) : ( xing2024seeing ) focuses on leveraging pre-trained generative models rather than training from scratch, while ( zhang2024foleycrafter ) targets improvements in audio-visual synchronization. Despite their success, these approaches rely heavily on large U-Net backbones, which limits scalability and efficiency. To address this, MDSGen ( pham2025mdsgen ) incorporates recent innovations in generative modeling, such as Diffusion Transformers (DiT) ( peebles2023scalable ) and spatial masking ( gao2023masked ; pham2024crossview ) , adapting masked diffusion Transformers for video-guided audio generation and achieving improved performance and efficiency. The introduction of flow matching ( lipman2023flow ) has further expanded generative modeling abilities. Recent methods like FRIEREN ( wang2024frieren ) and MMAudio ( cheng2025taming ) adopt flow matching as an alternative to diffusion, using Transformer-based frameworks and reporting strong results on video-to-audio (V2A) tasks. Similarly, we adopt flow matching with a Transformer-based model. However, despite architectural differences, most state-of-the-art approaches, whether diffusion, masked diffusion, or flow-based, continue to rely on classifier-free guidance (CFG) ( ho2022classifier ) , where input conditions are randomly dropped during training (typically at a rate of \eta= 10%) to enable multi-task learning (unconditional and conditional). In contrast, we shift the focus from this paradigm by adopting the recent Model Guided approach ( tang2025diffusion ) , moving toward a more efficient audio model-guided (AMG) strategy that achieves stronger generation quality and faster convergence. 2.2 Model-Guidance Learning in Audio Generation Model-guided learning has recently been introduced in the vision domain as Vision Model-Guidance (VMG) ( tang2025diffusion ) which enables efficient single-pass inference and is complementary to CFG, offering improvements in both inference modes. Inspired by this, we propose