What’s Making That Sound Right Now? Video-centric Audio-Visual Localization

Hahyeon Choi, Junhoo Lee, Nojun Kwak · 2025

Read What’s Making That Sound Right Now? Video-centric Audio-Visual Localization by Hahyeon Choi, Junhoo Lee, Nojun Kwak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04667.txt.

Introduction

Hahyeon Choi, Junhoo Lee, and Nojun Kwak ask what makes that sound right now in a 2025 paper. Sound-source hunts in video have long worked frame by frame, which misses how a sound grows and moves. Past sets also kept sources in view and lone, a far cry from real clips.

Their test set, called AVATAR, is built on video with fine time marks. Four cases span lone sounds, mixed sounds, many sources, and sounds from off screen. Their model reads time as a first-class clue. Stock methods stumble on the moving cases, which shows the gap the set names.

Catalog record

CREATORHahyeon Choi, Junhoo Lee, Nojun Kwak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What’s Making That Sound Right Now? Video-centric Audio-Visual Localization Hahyeon Choi, Junhoo Lee, Nojun Kwak Abstract Audio-Visual Localization (AVL) aims to identify sound-emitting sources within a visual scene. However, existing studies focus on image-level audio-visual associations, failing to capture temporal dynamics. Moreover, they assume simplified scenarios where sound sources are always visible and involve only a single object. To address these limitations, we propose AVATAR, a video-centric AVL benchmark that incorporates high-resolution temporal information. AVATAR introduces four distinct scenarios -- Single-sound, Mixed-sound, Multi-entity, and Off-screen -- enabling a more comprehensive evaluation of AVL models. Additionally, we present TAVLO, a novel video-centric AVL model that explicitly integrates temporal information. Experimental results show that conventional methods struggle to track temporal variations due to their reliance on global audio features and frame-level mappings. In contrast, TAVLO achieves robust and precise audio-visual alignment by leveraging high-resolution temporal modeling. Our work empirically demonstrates the importance of temporal dynamics in AVL and establishes a new standard for video-centric audio-visual localization. 1 1 1 https://hahyeon610.github.io/Video-centric_Audio_Visual_Localization/ 1 Introduction Humans and animals perceive the world by integrating multiple sensory modalities. In particular, vision and hearing complement each other, enabling the localization and identification of sound sources across a broad spatial field. This capability is essential for filtering relevant auditory information, such as distinguishing a speaker in a crowded environment or detecting a predator’s location from its roar. Inferring spatial information from multimodal cues has broad applications in areas such as robot navigation, AR/VR, and video analysis. Audio-Visual Localization (AVL) seeks to replicate this perceptual ability by identifying sound sources within a visual scene. A core approach in AVL research leverages the natural co-occurrence of audio and visual signals, using self-supervised learning to align these modalities and extract joint embeddings. This paradigm has led to significant advancements in AVL [ 23 , 33 , 35 , 6 , 1 , 25 ] , enabling models to effectively utilize large-scale web video datasets without requiring manual annotations. However, despite this progress, existing AVL research faces two key limitations, largely stemming from the structural constraints of current AVL benchmarks. The first limitation is that AVL research primarily focuses on image-level audio-visual associations rather than extending to video-based analysis. Most benchmarks [ 29 , 7 ] adopt an annotation approach where annotators watch the entire video and label all sound-emitting objects in a single frame, effectively treating that frame as representative of the entire video. Consequently, existing methods [ 30 , 12 , 32 , 24 ] operate on single-frame inputs, neglecting temporal dynamics. While this approach assesses spatial understanding within static images, it fails to capture temporal variations. Real-world AVL tasks require tracking moving sound sources and handling dynamic changes over time, making spatiotemporal modeling essential for robust performance. Table 1 : Comparison of Audio-Visual Localization Datasets. A comparison of existing AVL datasets in terms of dataset scale, annotation type, and the scenarios they cover, including Single-sound, Mixed-sound, Multi-entity, and Off-screen. \lx@sectionsign Statistics for AVSBench and AVSBench-Semantic are based on the test set due to supervised use. Scenario Dataset # Videos # Frames # Categories Avg. Length Annotation Annotation type Single-sound Mixed-sound Multi-entity Off-Screen Flickr-SoundNet [ 29 ] 5,000 5,000 50 20.0s bbox Image ✓ ✗ ✗ ✗ VGGSS [ 7 ] 5,158 5,158 221 10.0s bbox Image ✓ ✗ ✗ ✗ Epic Sound Object [ 14 ] 3,172 9,196 30 1.0s bbox Image ✓ ✗ ✗ ✗ Extended Flickr-SoundNet [ 20 ] 292 292 50 15.4s bbox Image ✗ ✗ ✗ ✓ Extended VGGSS [ 20 ] 5,537 5,537 221 10.0s bbox Image ✗ ✗ ✗ ✓ IS3 [ 31 ] 3,240 3,240 118 10.0s bbox & instance segm. Image ✗ ✓ ✗ ✗ § AVSBench [ 36 ] 804 4,020 23 5.0s segm. Chunk ✓ ✓ ✗ ✗ § AVSBench-Semantic [ 37 ] 1,554 11,520 70 7.4s semantic segm. Chunk ✓ ✓ ✗ ✗ \mathbf{AVATAR} (Ours) 5,000 24,266 80 10.0s bbox & instance segm. Video ✓ ✓ ✓ ✓ The second limitation is the oversimplified assumptions in existing AVL benchmarks [ 36 , 31 , 29 , 7 ] . These benchmarks assume that sound-emitting objects are always visible and typically involve only a single active source. However, real-world scenarios often feature multiple simultaneous sound sources, and sound-emitting objects may be outside the visual field. These constraints limit model generalization to complex audio-visual environments. Although some studies attempt to address these issues, they primarily rely on negative pair construction using mismatched image-audio pairs [ 20 ] or synthetic mixed-audio generation [ 31 ] , providing only partial solutions. To overcome these limitations, we propose advanced A udio- V isual localiz A tion benchmark for a spatio- T empor A l pe R spective in video (AVATAR), a video-centric AVL benchmark that evaluates models on their ability to localize sound sources with high temporal resolution. As summarized in Table 1 , AVATAR introduces four key evaluation scenarios that reflect real-world complexity: Single-sound, Mixed-sound, Multi-entity, and Off-screen. Furthermore, we introduce T emporal-aware A udio- V isual L ocalization model for fine-grained vide O understanding (TAVLO), a novel AVL model that explicitly incorporates temporal information. Experimental results demonstrate that existing methods struggle with tracking temporal changes, as they rely on global audio features and static frame mappings. In contrast, TAVLO is robust to temporal variations, leveraging high-resolution temporal information for precise audio-visual alignment. Our model effectively localizes sound sources even in complex environments, demonstrating significant improvements over prior approaches. 2 Related Work 2.1 Audio-Visual Localization AVL has primarily been studied through self-supervised learning, leveraging the natural co-occurrence of auditory and visual signals. Various approaches have been proposed to enhance AVL performance [ 30 , 32 , 12 ] . EZ-VSL [ 21 ] formulates AVL as a multiple instance learning [ 19 ] problem, introducing a cross-modal multiple-instance learning loss to improve training. SSL-TIE [ 18 ] enhances self-supervised localization by incorporating transformation invariance and equivariance. ACL-SSL [ 24 ] integrates CLIP [ 27 ] to explore multimodal learning in AVL. To address the limitations of purely self-supervised methods, DMT [ 10 ] employs semi-supervised learning to improve generalization, while SLAVC [ 20 ] explores weakly supervised strategies for precise localization with limited annotations. Beyond single-source scenarios, several studies have extended AVL to multi-source environments [ 26 , 15 , 13 , 22 ] , broadening its applicability to more complex auditory scenes. Figure 1 : Model-Driven Labeling and Verification Process. CAV-MAE [ 9 ] and YoloV8 [ 28 ] generate candidate bounding boxes with audio class predictions, refined through Audio-guided Bounding box Filtering. Human verification filters the regions before SAM [ 16 ] performs instance segmentation, followed by a final verification for accuracy. 2.2 Audio-Visual Localization Benchmarks The most widely used datasets, Flickr-SoundNet [ 29 ] and VGGSS [ 7 ] , are derived from SoundNet [ 3 ] and VGGSound [ 5 ] , respectively. In the egocentric domain, Epic Sound Object [ 14 ] introduces an AVL benchmark that considers viewpoint variations and occlusion challenges arising from wearer-environment interactions. Other benchmarks explore misalignment scenarios and multi-source settings. Extended datasets [ 20 ] are designed to evaluate model performance under audio-visual misalignment by generating synthetic negative samples through cross-pairing audio and

Read the complete text