Cinematic Audio Source Separation Using Visual Cues
Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son Chung · 2026
Read Cinematic Audio Source Separation Using Visual Cues by Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son Chung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26113.txt.
Introduction
Zhang and co-authors present the first audio-visual method for splitting film sound into its parts. Cinematic separation breaks a mix into speech, music, and effects for dubbing or remastering. Past methods listen only to the audio. Their method treats the task as conditional generation with flow matching. Visual cues guide which sounds belong where. Since no real dataset has isolated tracks, they synthesize training pairs from wild audio and video. A dual-stream visual encoder fits this setup. The model transfers well to real films, they report.
Catalog record
| CREATOR | Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son Chung |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65058 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26113
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cinematic Audio Source Separation Using Visual Cues Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son Chung Abstract Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams ( e.g . , facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at https://cass-flowmatching.github.io . 0 0 footnotetext: * Equal contribution, {\dagger} Corresponding authors. 1 Introduction Cinematic audio is composed of layered sound elements such as speech, music, and sound effects, which collectively enrich storytelling and immersion. The goal of Cinematic Audio Source Separation (CASS) is to separate a mixed movie audio into these three distinct tracks (Fig. 1 ). CASS enables a wide range of applications, including multilingual dubbing, film remastering, audio editing, and accessibility enhancement. As video streaming platforms continue to grow, the need for CASS becomes increasingly important to enable automatic tools that precisely control individual sound components. Figure 1 : Illustration of the Cinematic Audio Source Separation (CASS) task. The audio stream from a movie is separated into distinct tracks: speech, sound effects, and music. While related audio separation tasks, such as speech separation Hershey et al. 2016 ; Chen et al. 2017 ; Yu et al. 2017 and music demixing Défossez et al. 2019 ; Rouard et al. 2023 ; Défossez 2021 ; Mariani et al. 2024 , have seen significant progress, CASS remains underexplored. The introduction of the Divide and Remaster (DnR) dataset Petermann et al. 2022 formalized CASS as a three-way separation problem and initiated new research Watcharasupat et al. 2023 ; Uhlich et al. 2024 ; Watcharasupat et al. 2024 . However, existing methods are purely audio-based and overlook a defining characteristic of film: its audio-visual nature. In cinematic content, sound is often tightly coupled with visual events. Speech typically co-occurs with lip movements, and sound effects frequently align with object interactions or visual actions. Prior work in audio-visual learning has shown that using visual context, such as facial motion, body movement, and scene composition, significantly improves audio separation and enhancement Afouras et al. 2018a ; Ephrat et al. 2018 ; Gao and Grauman 2021 ; Gao et al. 2018 ; Tzinis et al. 2021 ; Chen et al. 2023 . Yet, to our knowledge, visual information has not been utilized in the context of CASS. The primary motivation of this paper is to achieve cinematic audio source separation using visual cues. Accordingly, we first formulate CASS as a conditional generative modeling problem. We adopt conditional flow matching Lipman et al. 2023 as our generative framework, which has demonstrated strong performance in both image Esser et al. 2024 ; Hu et al. 2024 and audio Le et al. 2024 ; Guo et al. 2024 ; Wang et al. 2024 ; Jung et al. 2024 generation tasks. Our model generates clean speech, sound effects, and music tracks from the mixture, conditioned on both audio and visual inputs. Second, despite the strong potential of visual cues, applying them to CASS remains challenging. A key obstacle is the lack of publicly available audio-visual datasets with clean source tracks, which are difficult – if not impossible – to obtain from real films. This raises an important question: Can we leverage individually available in-the-wild audio-visual data to train an effective audio-visual CASS model? To address this, we propose a novel training data synthesis strategy. In the absence of paired datasets such as films and their clean sound source tracks, we construct a pipeline that synthetically pairs individually available audio and video sources, e.g . , using facial videos for speech and scene video clips for sound effects. This results in a dual-stream video setup that provides controllable, source-specific visual supervision, for which we also design a visual feature extractor to leverage this setup. Importantly, although our model is trained using this synthetic dual-video configuration, we demonstrate that it generalizes effectively to real-world cinematic content without architectural changes, highlighting the practicality and robustness of our training approach. In summary, our contributions are: • We introduce the first framework for audio-visual cinematic audio source separation (AV-CASS), incorporating visual cues to separate speech, sound effects, and music in film audio. • We formulate CASS as a generative task using conditional flow matching, enabling flexible and principled modeling of multimodal audio decomposition. • We propose a training data synthesis strategy, along with a dedicated visual encoder, to enable training without the need for original source-separated film data. • We achieve strong performance on synthetic data, real-world cinematic examples, and standard audio-only CASS benchmarks. 2 Related Work 2.1 Cinematic Audio Source Separation Cinematic Audio Source Separation (CASS) was formalized by Petermann et al. 2022 ; Petermann et al. 2023 , introducing the Divide and Remaster (DnR) dataset for the separation of speech, sound effects, and music in film audio. BandIt Watcharasupat et al. 2023 applied Band-split RNNs Luo and Yu 2023 to this task, achieving improved performance over earlier methods. More recently, DnRv3 Watcharasupat et al. 2024 expanded the dataset with multilingual content and introduced mixing strategies aligned with industrial audio production pipelines. DnR-nonverbal Hasumi and Fujita 2025 further expands the dataset by incorporating nonverbal sounds, such as laughter and screams. While these efforts have advanced audio-based CASS, they remain limited to audio-only learning. To our knowledge, no prior work has explored audio-visual approaches for CASS, despite the inherently multimodal nature of cinematic content. This gap largely stems from the difficulty of acquiring datasets with both isolated audio tracks and temporally aligned video. In contrast, we propose the first audio-visual CASS framework and introduce a training data synthesis pipeline that constructs audio-visual training samples from in-the-wild video sources, without requiring ground-truth source audio with film datasets. Our approach enables effective audio-visual learning and generalizes well to real-world cinematic content. 2.2 Audio-Visual Source Separation Incorporating visual information has proven highly effective in sound source separation. Prior work has shown that visual cues such as lip movements align strongly with spoken content Chung and Zisserman 2017 ; Ma et al. 2022 , while facial features provide cross-modal biometric information that enhances speech separation Afouras et al. 2018a ; Ephrat et al. 2018 ; Owens and Efros 2018 ; Afouras et al. 2020 ; Chung et al. 2020 ; Nagrani et al. 2018 . Beyond speech, visual information from instrument motion or class-level appearance cues has also been leveraged to improve music separation Zhao et al. 2018 ; Gao and Grauman 2019 ; Gan et al. 2020 ; Chatterjee et al. 2021 . More generally, recent works