Video Object Segmentation-aware Audio Generation

Ilpo Viertola, Vladimir Iashin, Esa Rahtu · 2025

Read Video Object Segmentation-aware Audio Generation by Ilpo Viertola, Vladimir Iashin, Esa Rahtu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26604.txt.

Introduction

Ilpo Viertola, Vladimir Iashin, and Esa Rahtu set a new task: make sound for a chosen object in a video, not for the whole scene. Their 2025 work serves film sound craft, where fine control over each source matters. Past models blend all sounds and often stress the wrong object.

Their model takes object masks plus video and text cues to steer the sound it makes. A new benchmark set of solo music performance videos, with masks, supports the task. Tests show large gains over top current methods for control and fidelity. Code, samples, and the data set are shared in public. The authors aim at pro-grade, object-level sound design.

Catalog record

CREATORIlpo Viertola, Vladimir Iashin, Esa Rahtu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-26604

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video Object Segmentation-aware Audio Generation Ilpo Viertola, Vladimir Iashin, Esa Rahtu Abstract Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on the entire video and do not provide precise methods for prioritizing a specific object within a scene, generating unnecessary background sounds, or focusing on the wrong objects. To address this gap, we introduce the novel task of video object segmentation-aware audio generation, which explicitly conditions sound synthesis on object-level segmentation maps. We present SAGANet, a new multimodal generative model that enables controllable audio generation for musical instruments by leveraging visual segmentation masks along with video and textual cues. Our model provides users with fine-grained and visually localized control over audio generation. To support this task and further research on segmentation-aware Foley, we propose Segmented Music Solos, a benchmark dataset of musical instrument performance videos with segmentation information. Our method demonstrates substantial improvements over current state-of-the-art methods and sets a new standard for controllable, high-fidelity Foley synthesis for musical audio. Code, samples, and Segmented Music Solos are available at saganet.notion.site . keywords Artificial Foley, Multimodal, Generative Modeling 1 Introduction Multimodal audio generation focuses on synthesizing audio, given a conditional video feed, a textual description, or conditions in other modalities. These models can be utilized in Foley processing, where the goal is to produce a soundtrack for a video. For the artificial Foley models to thrive, e.g. , as a video post-processing tool, the end-user has to have high control over the synthesized result. In addition to controllability, the model has to produce high-quality samples with great semantic and temporal accuracy. Recent advances in diffusion ( Liu et al., 2023 ) and conditional flow matching (CFM) ( Tong et al., 2023 ) models have improved the audio quality of the artificial Foley models ( Cheng et al., 2025 ; Chen et al., 2025 ; Mo et al., 2024 ; Wang et al., 2024b ; Xu et al., 2024 ; Wang et al., 2024a ; Polyak et al., 2024 ) . However, existing models still lack precise user guidance and control. Although recent models ( Cheng et al., 2025 ; Chen et al., 2025 ; Polyak et al., 2024 ; Mo et al., 2024 ) introduce more modalities to guide the audio generation process, they still lack precision. In complex scenes, describing the target object using e.g. , text, quickly becomes unfeasible. To improve control in Foley generation, we propose a new audio synthesis task: video object segmentation-aware audio generation. This task focuses on nuanced control, highlighting the model’s ability to generate audio for a specific object in the video rather than for the full scene. To the best of our knowledge, we are the first to utilise visual segmentation information in the audio generation task. Existing state-of-the-art methods ( Cheng et al., 2025 ; Chen et al., 2025 ; Viertola et al., 2025 ; Wang et al., 2024b ; Polyak et al., 2024 ) focus on training the models from scratch, utilizing large uni- and multimodal datasets. It demands significant computational resources and quickly becomes infeasible in academic environments. For example, Movie Gen Audio ( Polyak et al., 2024 ) was pretrained on 384 high-end H100 GPUs for 14 days and later fine-tuned on 64 H100s for 24 hours. MMAudio ( Cheng et al., 2025 ) was trained on the same high-end GPU hardware, only utilizing fewer GPU hours per training run. Alternatively, recent works reduce training costs by using lightweight aligners or adapters to condition pretrained text-to-audio models on video sequences ( Wang et al., 2024a ; Mo et al., 2024 ; Jeong et al., 2025 ; Xing et al., 2024 ) . While these models are more efficient to train, they often struggle with temporal alignment, as fine-tuning text-to-audio models without built-in temporal control remains challenging. The concurrent work of Li et al. (2025) also explores object-aware audio generation. However, their focus on image-to-audio generation makes the approach unusable for artificial Foley. Our approach draws on recent advances in multimodal audio generation ( Cheng et al., 2025 ) and localized image captioning ( Lian et al., 2025 ) . To enable segmentation-aware audio generation for musical instruments, we develop a self-supervised control module on top of a pretrained network that enables users to select a specific object in a video to generate sound for. By training our control component on a small yet high-quality dataset, we achieve better controllability, temporal synchronization, and semantic quality compared to the original model, training only a fraction of the parameters. Although our model is trained on videos with a single audio source, it can generalize to scenes with multiple audio sources and generate audio for the target object. To train the segmentation-aware control module, we propose a high-quality dataset with sounding object segmentation maps and a high audio-video correspondence. The dataset consists of solo acts played on a variety of musical instruments. We curate the dataset based on Solos ( Montesinos et al., 2020 ) , AVSBench ( Zhou et al., 2022 ; Zhou et al., 2024 ) , and MUSIC21 ( Zhao et al., 2018 ; Zhao et al., 2019 ) datasets. We design a pipeline that generates short video clips given the original videos, ensuring that the target sounding object is present in both the auditory and visual modalities. Additionally, we extract visual segmentation maps of the target (sounding) object. We refer to this dataset as Segmented Music Solos. For testing purposes, we utilize the University of Rochester Multi-Modal Music Performance (URMP) dataset ( Li et al., 2018 ) . Our contributions can be summarized as follows: i) a new audio synthesis task, namely video object segmentation-aware audio generation, ii) a video object segmentation-aware control for a state-of-the-art multimodal generative audio model, iii) we show that by training our model with single-source samples it can generate audio for target object in multi-source scenes, and iv) a new benchmark dataset, Segmented Music Solos, with sounding object segmentation information. This manuscript is based on a previous conference publication at German Conference on Pattern Recognition (GCPR) 2025, which we significantly consolidate here by elaborating on evaluation methods, adding ablation studies and subjective results, and conduncting a subjective evaluation. 2 Related Work 2.1 Artificial Foley Models Artificial Foley models have gained considerable popularity. Many published models are built on top of autoregressive transformer architecture ( Viertola et al., 2025 ; Mei et al., 2024b ; Sheffer and Adi, 2023 ; Iashin and Rahtu, 2021 ) . Also, another transformer-based approach is to utilize Masked Generative Image Transformer (MaskGIT) ( Chang et al., 2022 ) schema for the audio generation task ( Ziv et al., 2024 ; Pascual et al., 2024 ; Liu et al., 2024b ; Su et al., 2024 ) . Another popular approach is to utilize diffusion ( Xu et al., 2024 ; Luo et al., 2023 ; Chen et al., 2024 ) or flow-matching methods ( Wang et al., 2024b ; Cheng et al., 2025 ) . To avoid resource- and time-exhaustive training from scratch, prior work has explored using pretrained text-to-audio models for video-to-audio generation by training lightweight feature aligners or control modules between the modalities

Read the complete text