Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition
Juncheng Wang Chao Xu * Cheng Yu * Lei Shang Zhe Hu, Shujun Wang Liefeng Bo · 2025
Read Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition by Juncheng Wang Chao Xu * Cheng Yu * Lei Shang Zhe Hu, Shujun Wang Liefeng Bo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06984.txt.
Introduction
Juncheng Wang, Chao Xu, Cheng Yu, Lei Shang, Zhe Hu, Shujun Wang, and Liefeng Bo generate soundtracks for silent videos that land in sync with the action. Their route extracts control signals from the video and feeds them into a mature text-to-audio diffusion model. The hard part is representing the mel-spectrogram, the sound picture such models use, in a form that is complete yet simple enough to predict from video.
Their method splits the spectrogram into three kinds of signals, some stored as discrete codes and some as smooth values. A predictor reads the video and forecasts all three, which are then recombined and passed to a ControlNet with a textual inversion trick to steer the audio. Across eight measures of quality, synchronization, and meaning match, the system reached state-of-the-art results.
Catalog record
| CREATOR | Juncheng Wang Chao Xu * Cheng Yu * Lei Shang Zhe Hu, Shujun Wang Liefeng Bo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06984
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition Juncheng Wang Chao Xu * Cheng Yu * Lei Shang Zhe Hu, Shujun Wang Liefeng Bo Abstract Video-to-audio generation is essential for synthesizing realistic audio tracks that synchronize effectively with silent videos. Following the perspective of extracting essential signals from videos that can precisely control the mature text-to-audio generative diffusion models, this paper presents how to balance the representation of mel-spectrograms in terms of completeness and complexity through a new approach called Mel Quantization-Continuum Decomposition (Mel-QCD). We decompose the mel-spectrogram into three distinct types of signals, employing quantization or continuity to them, we can effectively predict them from video by a devised video-to-all (V2X) predictor. Then, the predicted signals are recomposed and fed into a ControlNet, along with a textual inversion design, to control the audio generation process. Our proposed Mel-QCD method demonstrates state-of-the-art performance across eight metrics, evaluating dimensions such as quality, synchronization, and semantic consistency. Our codes and demos will be released at https://wjc2830.github.io/MelQCD/ . 1 Introduction The relevance of multimedia content has positioned video-to-audio (V2A) [ 21 , 45 , 43 , 50 , 19 ] generation as a critical area of research, focusing on synthesizing realistic audio tracks synchronized with silent video footage. This task is vital for enhancing user experiences in applications such as video editing [ 4 , 28 ] , post-production [ 33 , 25 ] , and content creation [ 16 , 38 ] , while also improving accessibility in media consumption. Moreover, as AI increasingly influences video production—often resulting in silent outputs [ 3 , 30 , 32 ] —the development of effective V2A solutions has become essential. High-quality audio generation for AI-generated videos enhances storytelling and emotional impact, bridging the gap between visual and auditory information in various multimedia contexts. Recent studies have shown promising results in V2A tasks by fine-tuning approaches from mature text-to-audio (T2A) [ 46 , 26 , 18 , 27 , 17 ] generation, particularly diffusion models. While these methods are driven by various motivations, they can generally be viewed as extracting signals from videos to control the synthesis of audio represented in mel-spectrogram format, as illustrated in Figure 1 . For example, FoleyCrafter [ 50 ] extracts sound event onset signals from video, which are then used to control the temporal distribution within a text-to-audio generative diffusion model. Figure 1 : Task formulation of this paper (a). Previous mainstream approaches focus on extracting control signals from videos to govern audio generation (b). However, they struggle to balance between ease of prediction and precision of control. In response, our proposed Mel-QCD achieves a more effective trade-off (c). From this perspective, we hypothesize the primary challenge that the community faces is how to better represent an audio signal that can be easily predicted from video while also enabling precise control over audio generation? Intuitively, an easily predictable signal promises effective generalization across different videos. However, to ensure that the generated audio is highly aligned with the conditioned video in both semantic and temporal representations, the signal must encompass adequate information. However, existing methods that fine-tune T2A diffusion models often fall short in one of these two dimensions, limitations that will be further discussed in Section 2 . In this paper, we present Mel Quantization-Continuum Decomposition (Mel-QCD) Control, as a way of trade-off between simplicity and precision. Specifically, we approach the challenge by balancing the representation of the mel-spectrogram in terms of completeness and complexity . We begin by decomposing the mel-spectrogram into three components: semantic, energy, and standard deviation vectors. To facilitate signal prediction from video, we propose quantizing the semantic vectors into discrete codes and constructing a semantic vector quantization (SVQ) codebook, where the index in the codebook represents the semantic component. This quantization effectively reduces complexity , without losing too much information. Moreover, for the energy and standard deviation vectors, as low-dimension data, we analyze their strong dependency on maintaining continuous distributions to ensure completeness . Hence, we advocate for retaining their original representations. By decomposing the mel-spectrogram into these components, we develop a video-to-all (V2X) predictor that generates signals from a given video. These individually predicted signals are subsequently recomposed to form a mel-like representation, termed Mel-QCD. Following FoleyCrafter [ 50 ] and ReWas [ 21 ] , we employ a ControlNet [ 48 ] to manage the spatial distribution of the generated mel through a T2A diffusion model. Furthermore, recognizing that the predicted Mel-QCD inevitably has some deviations in certain time slots, we incorporate a textual inversion technique to mitigate the semantic shifts caused by these inaccuracies. This enhancement ensures a more consistent alignment of the generated audio with the video, thereby preserving the fundamental semantic integrity. A primary objective of this work is to explore a more efficient framework for generating high-quality audio conditioned by video. To evaluate our method, we compare its performance with existing mainstream pipelines on the widely recognized VGGSound benchmark across three perspectives: generation quality, synchronization, and semantic consistency, utilizing eight different metrics. Additionally, we conduct extensive analytical experiments to validate the proposed insights. 2 Related Work Text-to-Audio Generation The primary driver of success in audio generation tasks is the use of Text-to-Audio (TTA) models. Previous efforts adopt GANs [ 13 , 24 ] , normalizing flows [ 31 , 22 ] , and VAEs [ 39 ] in this task. Recently, diffusion models [ 15 ] have demonstrated significant generative potential in visual domains and have subsequently been extended to audio generation. DiffSound [ 47 ] employs a discrete diffusion model to generate sounds from text prompts. Make-An-Audio [ 18 ] and AudioLDM [ 26 ] excel in generation quality by leveraging advanced latent diffusion models (LDMs) [ 36 ] . The following works further incorporates several enhancing designs. Make-An-Audio2 [ 17 ] introduces structured text encoding to improve semantic alignment. AudioLDM2 [ 27 ] unifies any audio signal into a universal representation, thus supporting various audio generation. Tango [ 11 ] utilizes Flan-T5 [ 7 ] as the text encoder to precisely convey complex textual concepts. Auffusion [ 46 ] reduces the demand for data and computational resources while maintaining excellent text-audio alignment. In this paper, our method is built upon Auffusion, inheriting its excellent text-conditioned generation capabilities while extending it to understand visual semantics. Video-to-Audio Generation Another popular research, Video-to-Audio (VTA), aims to generate audio that is semantically aligned and temporally synchronized with the video content. Based on how they connect visual and audio, we categorize the current work into two types. The first category involves training video-to-audio models from scratch. SpecVQGAN [ 19 ] and Im2wav [ 37 ] design an autoregressive network to generate audio tokens based on the input visual tokens. DiffFoley [ 29 ] employs a diffusion model while introducing contrastive learning to unify the video-audio features. VTA-LDM [ 45 ] is based on the LDMs and directly combine the visual embeddings through cross attention. However, they require a large-scale high-quality visual-audio aligned data for training, hence another category of methods has emerged, resorting to the foundational TTA models. Seeing-and-hearing [ 43 ] and VTA-Mapper [ 41 ] both project audio to the text embedding space, which is then processed by a pretrained TTA