TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis

Tri Ton, Ji Woo Hong, Chang D. Yoo · 2025

Read TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis by Tri Ton, Ji Woo Hong, Chang D. Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05684.txt.

Introduction

Tri Ton, Ji Woo Hong, and Chang Yoo present TARO, a framework for generating sound that matches video. The system builds on flow based transformers, which train stably and transform signals smoothly. Two additions target fidelity and timing, the twin demands of video to audio work.

One addition adjusts how strongly internal representations align as noise levels change during generation. The other feeds in onset cues, sharp markers of moments where sound should strike, drawn from the visuals. On two standard data sets, TARO beat prior methods on sound quality measures and reached over ninety-seven percent alignment accuracy. The results show gains in both realism and synchronization.

Catalog record

CREATORTri Ton, Ji Woo Hong, Chang D. Yoo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis Tri Ton, Ji Woo Hong, Chang D. Yoo Abstract This paper introduces Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning (TARO), a novel framework for high-fidelity and temporally coherent video-to-audio synthesis. Built upon flow-based transformers, which offer stable training and continuous transformations for enhanced synchronization and audio quality, TARO introduces two key innovations: (1) Timestep-Adaptive Representation Alignment (TRA), which dynamically aligns latent representations by adjusting alignment strength based on the noise schedule, ensuring smooth evolution and improved fidelity, and (2) Onset-Aware Conditioning (OAC), which integrates onset cues that serve as sharp event-driven markers of audio-relevant visual moments to enhance synchronization with dynamic visual events. Extensive experiments on the VGGSound and Landscape datasets demonstrate that TARO outperforms prior methods, achieving relatively 53% lower Frechet Distance (FD), 29% lower Frechet Audio Distance (FAD), and a 97.19% Alignment Accuracy, highlighting its superior audio quality and synchronization precision. 1 Introduction Video-to-audio generation has garnered increasing attention due to its broad applications in content creation and film production [ 1 , 7 ] . The goal is to synthesize audio that is both high-quality and temporally aligned with video frames. However, achieving high-fidelity and precisely synchronized audio remains challenging, especially when dealing with rapidly changing visual content. Despite progress in video-to-audio generation, existing approaches struggle to balance fidelity, synchronization, and efficiency. GAN-based methods [ 4 ] improve audio fidelity but often fail to maintain temporal coherence. Transformer-based autoregressive methods [ 13 , 26 ] offer greater modeling expressiveness but suffer from slow inference and accumulating timing errors. Diffusion-based methods [ 21 , 31 , 34 , 24 ] improve fidelity through latent diffusion and cross-modal pretraining, yet their high computational costs hinder real-time deployment. In contrast, flow-based methods [ 9 , 32 ] provide stable training and continuous transformations, emerging as a promising alternative for efficient video-to-audio synthesis. Figure 1 : Our TARO leverages onset-aware conditioning to improve synchronization, aligning generated audio with event-driven cues in video. Existing video-to-audio approaches naively learn latent audio representation through reconstruction loss, which can lead to suboptimal quality. Recently, REPA [ 33 ] has attempted to leverage pretrained features [ 23 ] as a constraint to guide latent representations in image synthesis. Inspired by this, we explore applying a similar constraint-driven approach to latent audio representation learning. However, directly adapting this approach for video-to-audio synthesis is challenging, as imposing strict constraints on latent representations in early timesteps, when noise dominates, can hinder learning rather than improve alignment. Instead, a more adaptive strategy is required to gradually refine alignment as the latent representation evolves. Additionally, achieving precise synchronization at the event level remains an open challenge. Existing methods struggle to capture fine-grained auditory cues tied to specific visual events, often generating misaligned or unnatural sounds. This issue arises from the lack of explicit mechanisms to detect and condition on key auditory events such as footsteps, object interactions, or percussive impacts, which serve as strong temporal anchors for realistic audio synthesis. To address these challenges, we introduce Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning (TARO), a novel flow-based transformer framework that enhances both fidelity and synchronization in video-to-audio generation. TARO introduces two key innovations to improve latent audio representation and event-level alignment. First, Timestep-Adaptive Representation Alignment (TRA) dynamically adjusts alignment strength based on the noise schedule, ensuring progressive refinement of latent representations during training. Second, Onset Aware Conditioning (OAC) explicitly integrates event-driven cues, aligning generated audio with video motion by detecting key auditory events. These mechanisms enhance temporal precision, improving both perceptual quality and naturalness in video-to-audio synthesis. Extensive experiments on two public datasets, VGGSound [ 3 ] , and Landscape [ 15 ] datasets, demonstrate that TARO synthesizes high-quality, well-aligned audio, outperforms state-of-the-art (SOTA) methods across various comprehensive metrics as shown in Figure 2 . Figure 2 : Comparison of FD (y-axis) and inference speed (x-axis) across different models, with marker size representing parameter size. Our proposed TARO (red) achieves state-of-the-art performance with a relatively small parameter size and fast inference speed. In summary, our contributions are as follows: • We propose T imestep- A daptive R epresentation Alignment with O nset-Aware Conditioning (TARO), a novel flow-based transformer framework for video-to-audio generation. TARO integrates both visual and onset-based control signals to enhance the fidelity and synchronization of generated audio. • We introduce a novel Timestep-Adaptive Representation Alignment strategy that dynamically adjusts alignment strength based on the noise schedule, refining latent representations for high-quality, temporally coherent audio. To our knowledge, we are the first to apply representation alignment in video-to-audio synthesis. • We present an Onset-Aware Conditioning that captures key event-driven cues to refine synchronization, enabling precise audio alignment with dynamic visual events. 2 Related Works Video-to-audio generation. Video-to-audio generation aims to synthesize temporally synchronized and semantically relevant audio from video. Transformer-based autoregressive methods like SpecVQGAN [ 13 ] offer flexibility but suffer from slow inference and weak alignment. Im2Wav [ 26 ] extends this to images via CLIP [ 25 ] embeddings but struggles with dynamic video content. Diffusion-based models improve quality but require multiple steps and large parameters, slowing inference. Diff-Foley [ 21 ] enhances alignment via latent diffusion, while See & Hear [ 31 ] improves multimodal integration but lacks fine-grained temporal precision. FoleyCrafter [ 34 ] incorporates a temporal controller for high-quality Foley sound but remains computationally heavy. Frieren [ 28 ] reduces steps via rectified flow matching but may miss sparse, event-driven dynamics. MDSGen [ 24 ] employs masked diffusion for efficiency but still relies on multiple steps for fidelity. Despite advancements, existing methods struggle with efficiency, synchronization, and event-driven sound generation. Our TARO framework addresses these issues using flow-based transformers, timestep-adaptive alignment, and onset-aware conditioning to enhance audio quality, synchronization, and computational efficiency. Flow-based generative models. Flow-based generative models have gained traction for their efficiency and stability in image and audio synthesis, providing an alternative to score-based diffusion models. Flow matching [ 17 ] models a continuous vector field to smoothly transport samples from noise to data using ordinary differential equations (ODEs), achieving robust training and superior performance over Denoising Diffusion Probabilistic Models (DDPM) [ 12 ] . Expanding on this, MMDiT [ 9 ] employs rectified flow transformers, enabling scalable high-resolution image synthesis, while SiT [ 22 ] introduces interpolant transformers to balance flow and diffusion dynamics, improving generation quality. In audio synthesis, Voicebox [ 14 ] and Audiobox [ 27 ] leverage flow matching for speech and unified audio generation, while VoiceFlow [ 10 ] applies rectified flow matching for efficient text-to-speech. Despite these advances, most flow-based models focus on either image or audio domains, with limited exploration of video-to-audio generation. Our work bridges this gap by extending flow-based transformers to video-to-audio synthesis, incorporating timestep-adaptive alignment and onset-aware conditioning to improve synchronization and efficiency. Audio Representation Learning. Recent advances in audio representation learning have significantly improved the ability to extract meaningful features for audio understanding, offering valuable insights for generative models. Self-supervised learning frameworks such as wav2vec 2.0 [ 2 ] capture robust speech representations, while CLAP [ 30 ] employs contrastive language-audio pretraining to learn general-purpose audio semantics. Transformer-based models like BEATs [ 5 ] leverage masked spectrogram learning for audio classification, whereas EAT

Read the complete text