MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji · 2024

Read MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis by Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15322.txt.

Introduction

Ho Kei Cheng and co-authors present MMAudio in 2024, a system that synthesizes audio for video. Given a video and optional text, it aims to produce high-quality sound that matches both meaning and timing.

Instead of training on limited video data alone, the system jointly trains on larger text-audio collections to learn rich sound quality. A synchronization module aligns video frames with audio codes to tighten timing. Despite a small model size, it sets a new state of the art among public video-to-audio models and generates eight seconds of audio in about a second. It also performs well at text-to-audio.

Catalog record

CREATORHo Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15322

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji Abstract We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework ( MMAudio ). In contrast to single-modality training conditioned on (limited) video data only, MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples. Additionally, we improve audio-visual synchrony with a conditional synchronization module that aligns video conditions with audio latents at the frame level. Trained with a flow matching objective, MMAudio achieves new video-to-audio state-of-the-art among public models in terms of audio quality, semantic alignment, and audio-visual synchronization, while having a low inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio also achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance. Code, models, and demo are available at: hkchengrex.github.io/MMAudio . 1. Introduction We are interested in Foley , i.e . , for a given video we want to synthesize ambient sound ( e.g . , rain, river flow) and sound effects induced by visible events ( e.g . , a dog barks, a racket hits a tennis ball). Note, Foley does not focus on synthesizing background music or human speech, which is often added in post-processing. Importantly, Foley requires to synthesize convincing high-quality audio that is 1) semantically and 2) temporally aligned to an input video. For semantic alignment, methods need to understand scene contexts and their association with audio – the visual concept of rain should be associated with the sound of splashing raindrops. For temporal alignment, methods need to understand audio-visual synchrony as humans can perceive audio-visual misalignment as slight as 25 ms [ 51 ] . Inspired by the efficacy of training data scaling demonstrated by recent works [ 21 , 53 , 8 ] , we pursue a data-driven approach to synthesize high-quality audio that respects these two types of alignment. Figure 1 : In addition to training on audio-visual-(text) datasets, we perform multimodal joint training with high-quality, abundant audio-text data which enables effective data scaling. At inference, MMAudio generates conditions-aligned audio with video and/or text guidance. Current state-of-the-art video-to-audio methods either train only on audio-visual data [ 1 ] from scratch [ 67 ] or train new “control modules” [ 73 , 20 , 66 , 69 ] for pretrained text-to-audio models on audio-visual data. The former is limited by the amount of available training data: the most commonly used audio-visual dataset VGGSound [ 1 ] contains only around 550 hours of videos. Audio-visual data are expensive to curate at a large scale, as in-the-wild videos from the Internet 1) contain music and speech 1 1 1 In AudioSet [ 9 ] (collected from YouTube), 82.48% of videos contain either human speech or music according to the provided audio labels. , which has limited utility for training a general Foley model, and 2) contain non-diegetic [ 58 ] sounds such as background music or sound effects added in post-processing, again unsuitable for a Foley model. The latter, i.e . , finetuning pretrained text-to-audio models (with added control modules) on audio-visual data, enables models to benefit from audio-generation knowledge obtained from larger-scale audio-text data ( e.g . , WavCaps [ 46 ] at 7,600 hours). However, adding control modules to pretrained text-to-audio models complicates the network architecture and limits the design space. It is also unclear whether pretrained text-to-audio models have sufficient degrees of freedom to support all video-to-audio scenarios, compared to training from scratch. To avoid these limitations, we propose a multimodal joint training paradigm ( Figure 1 ) that jointly considers video, audio, and text in a single multimodal transformer network and masks missing modalities during training. This allows us to train from scratch on both audio-visual and audio-text datasets with a simple end-to-end framework. Jointly training on large multimodal datasets enables a unified semantic space and exposes the model to more data for learning the distribution of natural audio. Empirically, with joint training, we observe a significant relative improvement in audio quality (10% lower Fréchet Distance [ 23 ] and 15% higher Inception Score [ 56 ] ), semantic alignment (4% higher ImageBind [ 11 ] score), and temporal alignment (14% better synchronization score). To further improve temporal alignment, we introduce a conditional synchronization module that uses high frame-rate visual features (extracted from a self-supervised audio-visual desynchronization detector [ 19 ] ) and operates in the space of scales and biases of adaptive layer normalization (adaLN) layers [ 50 ] , leading to accurate synchronization (50% relative improvement in synchronization score). In summary, we first propose MMAudio, a multimodal joint training paradigm for video-to-audio. It enables accessible data scaling and cross-modal understanding, significantly improving audio quality and semantic alignment. We also propose a conditional synchronization module which enables more precise audio-visual synchrony. We train MMAudio on openly accessible datasets and scale it to two audio sampling rates (16kHz and 44.1kHz) and three model sizes (157M, 621M, 1.03B), with the smallest model already achieving new state-of-the-art performance in video-to-audio synthesis among public models. Surprisingly, our multimodal approach also achieves competitive performance in text-to-audio generation compared to dedicated text-to-audio methods, showing that joint training does not hinder single-modality performance. 2. Related Works Semantic alignment. Semantic alignment between audio and video is learned by training on paired audio-visual data with a generation objective [ 18 , 6 , 44 , 66 , 49 , 60 , 2 , 71 ] or a contrastive objective [ 42 ] . To further understand audio semantics, we additionally train on paired audio-text data. We argue that the semantic understanding learned from audio-text pairs can be transferred to video-text pairs, as joint training leads to a shared semantic space (similar to ImageBind [ 11 ] and LanguageBind [ 74 ] ) and enables the network to learn richer semantics from more diverse data. Temporal alignment. Besides learning temporal alignment directly from audio-visual pairs, some recent works first learn from videos a separate model to predict handcrafted proxy features such as audio onsets [ 73 , 54 ] , energy [ 20 , 17 ] , or root-mean-square of waveforms [ 34 , 4 ] . We deviate from these handcrafted features and directly learn alignment from the deep feature embeddings of a pre-trained self-supervised desynchronization detector Synchformer [ 19 ] , allowing a more nuanced interpretation of the input signal. A recent work V-AURA [ 65 ] also uses Synchformer [ 19 ] for synchronization in an autoregressive framework. However, [ 65 ] does not perform multimodal training on text and has a short context window (2.56s) while we produce longer-term (8-10s) temporally consistent generations. In terms of improving positional embeddings for temporal alignment, Mei et al. [45] concurrently propose to subsample high-frequency (audio) positional embeddings, while we propose to scale up the frequencies of low-frequency (visual) positional embeddings – the effects are identical when the higher frequency is an integer multiple of the lower one. Multimodal conditioning. The most common way to support multimodal conditioned generation is to add “control modules” that inject visual

Read the complete text