Segmenting Collision Sound Sources in Egocentric Videos
Kranti Kumar Parida, Omar Emara, Hazel Doughty, Dima Damen · 2025
Read Segmenting Collision Sound Sources in Egocentric Videos by Kranti Kumar Parida, Omar Emara, Hazel Doughty, Dima Damen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13863.txt.
Introduction
Kranti Kumar Parida, Omar Emara, Hazel Doughty, and Dima Damen define a new computer vision task: finding which objects made a collision sound in a first-person video. A crash or clatter comes from two objects meeting, so its sound depends on both, and head-mounted footage is cluttered with small, brief events. Their method learns with weak supervision, guided by the audio.
The approach builds on the CLIP and SAM2 foundation models and adds a first-person cue: objects held in the hands are likely actors in any collision. It beats rival methods by three to nearly five times on the overlap score across two new test sets drawn from EPIC and Ego4D videos.
Catalog record
| CREATOR | Kranti Kumar Parida, Omar Emara, Hazel Doughty, Dima Damen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13863
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Segmenting Collision Sound Sources in Egocentric Videos Thanks: This work was done when Kranti was at University of Bristol. Kranti Kumar Parida, Omar Emara, Hazel Doughty, Dima Damen Abstract Humans excel at multisensory perception and can often recognise object properties from the sound of their interactions. Inspired by this, we propose the novel task of C ollision S ound S ource S egmentation (CS3), where we aim to segment the objects responsible for a collision sound in visual input ( i.e . video frames from the collision clip), conditioned on the audio. This task presents unique challenges. Unlike isolated sound events, a collision sound arises from interactions between two objects, and the acoustic signature of the collision depends on both. We focus on egocentric video, where sounds are often clear, but the visual scene is cluttered, objects are small, and interactions are brief. To address these challenges, we propose a weakly-supervised method for audio-conditioned segmentation, utilising foundation models (CLIP and SAM2). We also incorporate egocentric cues, i.e . objects in hands, to find acting objects that can potentially be collision sound sources. Our approach outperforms competitive baselines by 3 × 3\times and 4.7 × 4.7\times in mIoU on two benchmarks we introduce for the CS3 task: EPIC-CS3 and Ego4D-CS3. 1 Introduction Humans effortlessly interpret the world around them using both visual and auditory cues. Sounds from physical interactions, such as a spoon clattering against a bowl or a hammer striking a nail, carry rich information about these actions or events. Despite their richness and frequency in everyday interactions, these collision sounds are rarely studied in audio-visual modelling. Such collision sounds are especially informative in egocentric video where the interaction sounds are clearly captured by wearable microphones, even in cluttered or visually ambiguous scenes. Auditory cues reveal the nature of interacting objects, enabling quick and intuitive reasoning for humans and embodied agents alike. Figure 1 : Task Definitions. (Top) Sound source localisation segments one object given its sound. (Middle) Multi-sound source localisation segments multiple objects from a mixture of their individually distinct sounds. (Bottom) We introduce a novel task, CS3, to segment the sources of collision sound by identifying the objects involved in the interaction, based on the impact sound. Prior works localising sound sources [ 46 , 7 , 47 , 22 , 37 ] have made progress in linking sounds to objects. However, these efforts focus on isolated, clearly audible objects, like a dog barking or a car passing, where sound maps directly to a single object (Fig. 1 top). Multi-sound source localisation extends this by localising multiple objects producing sounds at the same time, however each has a distinct sound ( e.g . a dog barking and cat meowing, Fig. 1 middle). While the recoded audio is the mixture of both sounds, each object still produces an independent sound that can be separated from the mixed audio. Moreover, datasets for these tasks often have minimal visual clutter or noise [ 46 , 7 , 55 ] . In contrast, we introduce the task of C ollision S ound S ource S egmentation (CS3) (Fig. 1 , bottom), where given a collision sound and a corresponding image the goal is to provide pixel segmentations of the object(s) producing this collision sound. We focus on egocentric audio-visual clips [ 14 , 20 ] where collision sounds are prominent [ 24 ] . Collision-induced sounds pose fundamentally different challenges. First, these sounds are not produced by individual objects but by interactions between two objects or parts of the same object. The collision sound is defined by the characteristics of both objects as well as the type and force of interaction; and is not a linear combination of two sounds. For example, a metal spoon sounds different hitting a ceramic plate than hitting a plastic container. Second, collisions in egocentric videos often occur in cluttered environments with frequent occlusions. Large-scale segmentation annotations are difficult to obtain in these settings. We thus design a weakly supervised method that requires only collision clips and can segment collision sound sources without segmentation annotations. Our approach uses a CLIP-based audio-conditioned segmentation model. Given most egocentric collisions involve hand-object interactions, we additionally utilise this prior with a hand-object interaction model that locates in-hand objects. We then verify the colliding objects to segment either two objects causing the collision, or a single object with colliding parts. Our contributions are threefold. First, we introduce the task of collision sound source segmentation (CS3), which aims to identify objects responsible for a collision using an audio-visual clip of the impact. Second, we curate two challenging CS3 datasets from large-scale egocentric datasets EPIC-Kitchens [ 14 ] and Ego4D [ 20 ] . Compared to existing sound source localisation datasets [ 46 , 7 , 55 ] , our scenes are more cluttered, objects are smaller ( \sim 10% of the image), and sounds arise from collisions rather than isolated sources. Third, we propose a method combining audio-conditioned CLIP-based segmentation with hand-object reasoning and show it significantly outperforms baselines on both datasets. 2 Related Works Audio-visual learning [ 1 , 2 , 32 , 39 , 40 , 38 , 52 ] exploits correlation between audio and visual modalities to learn aligned representations for various downstream tasks. In this work, we focus on localising objects involved in collision, and therefore review prior work on general sound source localisation [ 46 , 7 , 36 , 35 , 30 , 23 , 42 , 21 , 43 , 22 , 37 , 29 , 34 , 50 ] as well as audio-visual methods specifically addressing collision or impact sounds [ 3 , 41 , 54 , 49 , 48 , 17 , 18 , 12 , 13 , 4 , 51 ] . Sound Source Localisation identifies the image region responsible for a given sound [ 46 , 7 , 36 , 35 , 47 , 16 , 30 , 23 , 42 ] . A common strategy is to learn audio-visual correspondence via a contrastive loss on paired image-audio data [ 46 ] . Variants enhance performance by incorporating pre-trained object detectors [ 36 ] , suppressing irrelevant image regions [ 35 ] , using hard negatives [ 7 ] , increasing positive pairs [ 47 ] , adding optical flow [ 16 ] , or leveraging slot-attention [ 30 ] . While most methods target object sounds, [ 23 ] addressed action sounds in videos using geometric pooling of visual features and synthetic audio mixtures. Another direction adapts pre-trained CLIP-based vision-language models [ 44 ] for sound source localisation via audio-visual correspondence. The method [ 42 ] combines CLIP with supervised segmentation to improve localisation performance after adapting to audio inputs. Our approach is inspired by these works, following the same principle of weak supervision. Different to existing methods which assume a single sound source per image, we aim to localise collision sounds that often arise from multiple objects. Multi-sound Source Localisation aims to spatially localise multiple sounding objects in a scene, each corresponding to distinct a audio cue [ 21 , 43 , 22 , 37 , 29 , 34 , 50 ] . Hu et al . [ 21 ]