FOL ∙ \bullet AI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment

Riccardo F. Gramaccioni ♭∗ , Christian Marinoni ♭∗ , Emilian Postolache ♯ , Marco Comunità ♮ , Luca Cosmo ♯ , Joshua D. Reiss ♮ , and Danilo Comminiello ♭, ♭ Sapienza University of Rome, Italy ♯ Ca’ Foscari University of Venice, Italy ♮ Queen Mary University of London, UK · 2024

Read FOL ∙ \bullet AI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment by Riccardo F. Gramaccioni ♭∗ , Christian Marinoni ♭∗ , Emilian Postolache ♯ , Marco Comunità ♮ , Luca Cosmo ♯ , Joshua D. Reiss ♮ , and Danilo Comminiello ♭, ♭ Sapienza University of Rome, Italy ♯ Ca’ Foscari University of Venice, Italy ♮ Queen Mary University of London, UK (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15023.txt.

Introduction

Riccardo Fosco Gramaccioni and co-authors present FolAI in a 2024 paper, a system that generates sound effects for video. Foley artists rebuild everyday sounds, footsteps, doors, crashes, to match what happens on screen, a slow manual craft. Teaching a computer to do it means solving two problems at once: the sound must fit the action in meaning, and land at the right moments.

FolAI splits the job in two stages. First it reads the video's motion and draws a smooth control signal that captures how intense and rhythmic the action is over time. Then a diffusion model composes the sound, guided by that timing scaffold and by semantic cues about what should be heard. The authors argue that separating when from what keeps generated effects locked to the picture, a step toward automated sound design that still respects a creator's intent.

Catalog record

CREATORRiccardo F. Gramaccioni ♭∗ , Christian Marinoni ♭∗ , Emilian Postolache ♯ , Marco Comunità ♮ , Luca Cosmo ♯ , Joshua D. Reiss ♮ , and Danilo Comminiello ♭, ♭ Sapienza University of Rome, Italy ♯ Ca’ Foscari University of Venice, Italy ♮ Queen Mary University of London, UK
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FOL ∙ \bullet AI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment Riccardo F. Gramaccioni ♭∗ , Christian Marinoni ♭∗ , Emilian Postolache ♯ , Marco Comunità ♮ , Luca Cosmo ♯ , Joshua D. Reiss ♮ , and Danilo Comminiello ♭, ♭ Sapienza University of Rome, Italy ♯ Ca’ Foscari University of Venice, Italy ♮ Queen Mary University of London, UK Abstract Traditional sound design workflows rely on manual alignment of audio events to visual cues, as in Foley sound design, where everyday actions like footsteps or object interactions are recreated to match the on-screen motion. This process is time-consuming, difficult to scale, and lacks automation tools that preserve creative intent. Despite recent advances in vision-to-audio generation, producing temporally coherent and semantically controllable sound effects from video remains a major challenge. To address these limitations, we introduce Fol·AI, a two-stage generative framework that decouples the when and the what of sound synthesis, i.e., the temporal structure extraction and the semantically guided generation, respectively. In the first stage, we estimate a smooth control signal from the video that captures the motion intensity and rhythmic structure over time, serving as a temporal scaffold for the audio. In the second stage, a diffusion-based generative model produces sound effects conditioned both on this temporal envelope and on high-level semantic embeddings, provided by the user, that define the desired auditory content (e.g., material or action type). This modular design enables precise control over both timing and timbre, streamlining repetitive tasks while preserving creative flexibility in professional Foley workflows. Results on diverse visual contexts, such as footstep generation and action-specific sonorization, demonstrate that our model reliably produces audio that is temporally aligned with visual motion, semantically consistent with user intent, and perceptually realistic. These findings highlight the potential of Fol·AI as a controllable and modular solution for scalable, high-quality Foley sound synthesis in professional and interactive settings. Supplementary materials, including samples and code, are accessible on our dedicated demo page at https://ispamm.github.io/FolAI . I Introduction Fig. 1: Example showing ground truth audio and video, predicted RMS envelope and generated audio. Sound is a fundamental component of audiovisual storytelling, capable of deeply altering the meaning and emotional impact of a scene. Traditionally, Foley artists and sound designers manually select, record, and synchronize sounds (like footsteps or object interactions) to match on-screen actions. This careful process ensures the desired narrative effect but is inherently time-consuming and requires significant effort. Recent advancements in deep learning, particularly in multimodal generative models [ 1 , 2 , 3 ] and Large Language Models (LLMs) [ 4 , 5 ] , have generated significant interest in automating and enhancing audio synthesis. This has led to the rise of Video-to-Audio (V2A) generation [ 6 , 7 , 8 , 9 , 10 ] , a research area focused on synthesizing audio that is semantically relevant and temporally aligned with an input video sequence. Initial efforts in this area often focused on semantic alignment, generating ambient sounds or soundtracks reflecting the overall mood [ 11 , 12 , 13 , 14 ] , but lacked the temporal precision needed for synchronizing specific sound effects. Achieving both appropriate semantic content and precise timing is crucial for effective V2A generation as temporal alignment is equally fundamental for realism, particularly when synchronizing distinct sound events like Foley effects. Even slight timing mismatches, on the order of milliseconds, can be perceived by the audience and disrupt the feeling of immersion [ 15 ] . Furthermore, although more recent end-to-end V2A models [ 7 , 10 , 16 , 17 , 18 , 19 ] aim for better synchrony, they often function as “black boxes”. This typically offers limited direct control for sound designers to refine the timing or creatively adjust the generated audio beyond basic post-processing. This lack of fine-grained control hinders their adoption in professional workflows where artistic input and adjustments are essential. To address these critical needs for robust temporal alignment and user control, we introduce FOL·AI, a novel two-stage generative framework designed for accurate V2A synthesis. Our approach separates the task into predicting the temporal structure and then generating the audio based on that structure and desired sound characteristics, as shown in Fig. 2 . The first stage analyzes the input video using frame features and optical flow to predict a Root Mean Square (RMS) envelope. This envelope represents the audio’s timing, intensity, and duration [ 20 ] . Importantly, this predicted envelope is human-readable and can be directly edited by the sound designer. This allows for fine-tuning the timing or even adding sounds for off-screen events before the audio is generated. The second stage employs a state-of-the-art diffusion model architecture to create the final sound. To ensure the audio matches the timing specified by the envelope, we introduce an additional conditioning network architecture that steers the diffusion process using the RMS envelope from the first stage. Users can control the timbre by providing semantic information through embeddings from reference audio samples, text descriptions and the input video. This modular design allows creatives to control when sounds occur and what they sound like. The FOL·AI framework successfully generates 44.1 kHz stereo audio of variable length that shows strong temporal and semantic synchronization with the input video, as qualitatively illustrated in Fig. 1 . We demonstrate the effectiveness of our model through careful objective evaluations on the widely used Greatest Hits V2A benchmark dataset [ 21 ] , where FOL·AI achieves state-of-the-art results compared to existing V2A models. Furthermore, recognizing the lack of suitable public datasets with high-quality audio and video for evaluating V2A models on specific, highly relevant sound design tasks like Foley, we introduce Walking The Maps . This new dataset focuses specifically on footstep generation, a key challenge and frequent task in the field [ 22 ] . Sourced from high-quality video game walkthroughs available online, it provides numerous video clips featuring clearly audible footstep sounds across various surfaces (like grass, concrete, wood) and character actions (walking, running), paired with corresponding high-definition video offering excellent temporal cues. The audio was carefully preprocessed using source separation techniques to isolate footstep sounds, creating clean targets suitable for robust V2A model training and evaluation. We make Walking The Maps freely available to the research community to facilitate progress and benchmarking in realistic automated Foley synthesis scenarios. Fig. 2: FOL·AI architecture consists of two distinct parts: the video model, that predicts an envelope representative for the audio directly from the input video, and the audio synthesis model for the controlled generation of the final audio effect. The generation is controlled temporally by the predicted RMS envelope through a DiT ControlNet, and semantically by CLAP and CAVP embeddings. The length of the output waveform can be controlled with \mathrm{seconds\_start} and \mathrm{seconds\_total} parameters. II Related Works Generating audio that is aligned with a silent reference video is salient task nowadays for multimedia post production. Due to recent developments in deep learning, many of the leading experts in the audiovisual industry are focusing their efforts on adopting deep learning models that can be integrated into the principal post production tools [ 23 , 24 , 25 , 26 ] . Despite these advancements, generating audio from video still

Read the complete text