Seeing Speech and Sound: Distinguishing and Locating Audios in Visual Scenes

Hyeonggon Ryu*, Seongyu Kim*, Joon Son Chung, Arda Senocak · 2025

Read Seeing Speech and Sound: Distinguishing and Locating Audios in Visual Scenes by Hyeonggon Ryu*, Seongyu Kim*, Joon Son Chung, Arda Senocak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18880.txt.

Introduction

A person scanning a room hears a voice and a clatter at once and knows where each came from; machines usually handle speech and other sounds in separate passes. Hyeonggon Ryu, Seongyu Kim, Joon Son Chung, and Arda Senocak built one model in 2025 that grounds both in the same visual scene, even when the sounds are mixed together.

Their mix-and-separate training feeds the model blended audio and teaches it to pull the strands apart into distinct embeddings, each tied to its source in the image. A new dataset tests exactly that mixed case, and the model beat prior methods on it. It also matched or beat specialists on standard segmentation and retrieval tasks.

Catalog record

CREATORHyeonggon Ryu*, Seongyu Kim*, Joon Son Chung, Arda Senocak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18880

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Speech and Sound: Distinguishing and Locating Audios in Visual Scenes Hyeonggon Ryu*, Seongyu Kim*, Joon Son Chung, Arda Senocak Abstract We present a unified model capable of simultaneously grounding both spoken language and non-speech sounds within a visual scene, addressing key limitations in current audio-visual grounding models. Existing approaches are typically limited to handling either speech or non-speech sounds independently, or at best, together but sequentially without mixing. This limitation prevents them from capturing the complexity of real-world audio sources that are often mixed. Our approach introduces a “mix-and-separate” framework with audio-visual alignment objectives that jointly learn correspondence and disentanglement using mixed audio. Through these objectives, our model learns to produce distinct embeddings for each audio type, enabling effective disentanglement and grounding across mixed audio sources. Additionally, we created a new dataset to evaluate simultaneous grounding of mixed audio sources, demonstrating that our model outperforms prior methods. Our approach also achieves comparable or better performance in standard segmentation and cross-modal retrieval tasks, highlighting the benefits of our mix-and-separate approach. Figure 1 : Simultaneous Audio-Visual Grounding. We introduce a model for simultaneous grounding of mixed audio types — including overlapping spoken language and non-speech sounds — within a visual scene. Our framework takes audio-visual grounding to the next level, overcoming the limitations of existing methods that handle audio types independently or sequentially without overlapping. † † footnotetext: ∗ These authors contributed equally to this work. 1 Introduction We live in a world rich with multi-sensory signals. Our interactions with it rely on our natural multimodal perception, drawing on auditory and visual cues as well as spoken language to gain a complete understanding of our surroundings. While we can actively limit or focus our field of view in visual scenes, auditory signals are a mixture of different types of sounds, such as speech, music, and a variety of environmental or object sounds, all entering our ears as a combined mix without any filtering. Despite this complexity, humans can still discern specific types of sounds and associate them with their visual counterparts — for example, the sound of a ‘‘lion roaring’’, someone saying the word ‘‘lion’’, or the song ‘‘Nants’ Ingonyama’’ 1 1 1 Nants’ Ingonyama is the iconic opening song from The Lion King https://www.youtube.com/watch?v=CF-c1K3WWg4 linked with the visual of a lion . Remarkably, we do this without any direct supervision. This natural audio-visual association enables us to locate which object a spoken word refers to or identify which object is producing a sound within a visual scene. Inspired by the human ability to learn these localization skills from natural audio-visual correspondences without explicit supervision, extensive studies [ 32 , 33 , 34 , 35 , 36 , 21 , 20 , 11 , 28 , 29 ] have aimed to replicate this understanding in machine perception. However, most existing methods focus on understanding relationships between images and either spoken utterances or generic sounds, where each approach handles only one type of audio at a time but not both together. Recently, Hamilton et al . [ 9 ] proposed a model that visually grounds both spoken language and non-speech sounds within an image scene. This model is trained to disentangle different types of audio, such as environmental sounds or spoken words, and to locate their sources within a visual context, all without supervision. While pioneering, this work has several limitations. First, the model handles either speech or non-speech audio individually, rather than mixed audio sources, limiting its ability to process real-world audio, which often contains overlapping sounds. Second, the model does not achieve effective disentanglement in distinguishing between different types of auditory signals ( e.g . , speech and generic sounds). Although each of the heads in the audio encoder primarily activates for a specific audio type, these activations function as a mere selection mechanism for each type, rather than effectively controlling the flow of information between types — resulting in information leakage. Consequently, the multi-heads learn features of the opposite audio type, which impairs disentanglement and makes the model vulnerable when confronted with mixed audio inputs. To address these limitations, we propose a unified method that simultaneously grounds both spoken language and non-speech sounds within a visual scene ( Figure 1 ). Our model tackles both challenges using a “mix-and-separate” approach. We introduce a novel mixed audio-visual alignment objective that jointly learns correspondence and disentanglement , aligning representations of multiple audio types with corresponding visuals in images. Our framework includes shared audio and visual encoders within each modality, facilitating the transfer of audio-visual interactions across learning objectives. Unlike traditional mix-and-separate approaches, we do not generate separated audio; instead, we obtain embeddings from each specialized head of the audio encoder that represent the disentangled sound sources of different types using contrastive learning. These learning objectives enable our framework to process audio mixtures, eliminating the need for multiple audio types to appear sequentially or individually during training and inference. Additionally, this approach improves disentanglement within the multi-head structure. The primary motivation of this paper is to achieve simultaneous grounding of multiple audio types; accordingly, we aim to evaluate our model’s ability to perform this task. However, no existing dataset is suitable for this purpose. To address this, we created a new dataset by extending the IS3 dataset [ 36 ] — which provides segmentation masks for multiple sound-producing objects with corresponding audio — by adding spoken utterances to each sample. Evaluation on this dataset demonstrates that our model outperforms existing methods in simultaneous sound and speech grounding, a task that requires handling mixed audio inputs. Additionally, by achieving superior performance on standard segmentation and cross-modal retrieval experiments on the datasets from [ 9 ] , we show the benefits of our self-supervised mix-and-separate approach. Our main contributions are as follows: • We propose a method for simultaneously grounding both speech and non-speech audios within a visual scene. • We introduce a novel audio-visual alignment objective that uses audio mixtures and jointly learns correspondence and disentanglement using contrastive learning. • We provide a new dataset, Extended-IS3 , for simultaneous sound and speech grounding and demonstrate that our model outperforms existing methods. 2 Related Works Sound source localization. Audio-visual sound source localization is the task of identifying objects, events, or areas within a visual scene based on corresponding audio cues. In the deep learning era, Senocak et al . [ 32 , 33 ] pioneered this field by introducing a cross-modal attention mechanism with contrastive learning to align audio and visual information. As this research field advanced, diverse techniques have emerged, including hard sample mining [ 3 ] , multiple-instance contrastive learning [ 21 ] , negative-free strategies [ 38 ] , momentum encoders [ 20 ] , intra-modality similarity learning [ 39 ] , and the use of multiple positives in contrastive learning [ 35 ] . Recently, CLIP has also been incorporated into sound source localization without using text [ 27 ] . Most prior methods focus on single-source localization, while another line of research addresses multiple sound source localization with generic sounds [ 15 , 30 , 17 , 23 , 19 , 18 ] . In contrast to these models, which primarily localize generic sounds, our work targets both generic

Read the complete text