3D Audio-Visual Segmentation

Artem Sokolov, Swapnil Bhosale, Xiatian Zhu · 2024

Read 3D Audio-Visual Segmentation by Artem Sokolov, Swapnil Bhosale, Xiatian Zhu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02236.txt.

Description

Abstract Recognizing the sounding objects in scenes is a longstanding objective in embodied AI, with diverse applications in robotics and AR/VR/MR. To that end, Audio-Visual Segmentation (AVS), taking as condition an audio signal to identify the masks of the target sounding objects in an input image with synchronous camera and microphone sensors, has been recently advanced.

Catalog record

CREATORArtem Sokolov, Swapnil Bhosale, Xiatian Zhu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02236

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D Audio-Visual Segmentation Artem Sokolov, Swapnil Bhosale, Xiatian Zhu Abstract Recognizing the sounding objects in scenes is a longstanding objective in embodied AI, with diverse applications in robotics and AR/VR/MR. To that end, Audio-Visual Segmentation (AVS), taking as condition an audio signal to identify the masks of the target sounding objects in an input image with synchronous camera and microphone sensors, has been recently advanced. However, this paradigm is still insufficient for real-world operation, as the mapping from 2D images to 3D scenes is missing. To address this fundamental limitation, we introduce a novel research problem, 3D Audio-Visual Segmentation , extending the existing AVS to the 3D output space. This problem poses more challenges due to variations in camera extrinsics, audio scattering, occlusions, and diverse acoustics across sounding object categories. To facilitate this research, we create the very first simulation based benchmark, 3DAVS-S34-O7 , providing photorealistic 3D scene environments with grounded spatial audio under single-instance and multi-instance settings, across 34 scenes and 7 object categories. This is made possible by re-purposing the Habitat simulator [ 32 ] to generate comprehensive annotations of sounding object locations and corresponding 3D masks. Subsequently, we propose a new approach, EchoSegnet , characterized by integrating the ready-to-use knowledge from pretrained 2D audio-visual foundation models synergistically with 3D visual scene representation through spatial audio-aware mask alignment and refinement. Extensive experiments demonstrate that EchoSegnet can effectively segment sounding objects in 3D space on our new benchmark, representing a significant advancement in the field of embodied AI. Project page: https://surrey-uplab.github.io/research/3d-audio-visual-segmentation/ 1 Introduction Human perception of the real world, both visual and acoustic, predominantly occurs in three dimensions. Prior psychology literature [ 37 ] has highlighted humans’ remarkable ability to correspond across multiple modalities, often involving the association of events across these modalities. For instance, we can effortlessly ground emergent surround sound with its potential source in 3D visuals [ 26 ] . Inspired by this capability, a crucial aspect in the development of embodied AI systems is their ability to integrate cues from synchronous multimodal input streams and establish targets corresponding to their goals. In this work, we aim to build a machine model to achieve this multimodal correspondence, particularly targeted towards the task of audio-visual segmentation (AVS) in 3D. Albeit AVS has been widely explored within audio-visual scene analysis and correspondence learning, prominent research in this field has focused on 2D environments involving mono (single channel) sound sources, thus devoid of spatial presence entirely. In this paper, we take the first step towards exploring 3D AVS and introduce a large benchmark, 3DAVS-S34-O7 . Our exploration is rooted in a fundamental grounding problem: given an embodied agent equipped with a camera and a binaural microphone, can we teach the agent to obtain fine-grained localization of potential sounding objects (generally by predicting a segment-level mask of the object in 3D) while also utilizing spatial audio cues? (see Fig. 1 ) Furthermore, we extend our benchmark to include a more competitive multi-instance setup where, although multiple instances of the same object might be present in the scene, the goal is to segment only the sounding instance. This setup helps us testify to the efficacy of spatial presence harnessed from the input binaural audio samples. Recently, 3D Gaussian Splatting (3D-GS) [ 15 ] has emerged as a prospective method for modeling static 3D scenes directly from input RGB frames. Owing to its explicit Gaussian based representation, it has paved a natural pathway for 3D visual segmentation [ 13 , 41 , 33 ] . Deriving inspiration from human spatial memory in indoor environments, we design EchoSegnet , a purely training-free pipeline for 3D AVS within a 3D-GS representation. EchoSegnet leverages 2D foundation models (namely SAM [ 17 ] and ImageBind [ 9 ] ) to first obtain 2D AVS masks on the input RGB frames. These 2D AVS masks are further used to segment the Gaussians in the learned 3D-GS representation to obtain multi-view masks to achieve a consistent 3D segmentation. Figure 1: Comparison of the existing 2D AVS task with our proposed 3D AVS. Former task utilizes single channel audio to generate pixel-level masks of the potential sounding object in the input RGB frame. 3D AVS on the other hand is aimed at generating 3D masks (from which multi-view consistent 2D masks can be rendered) while utilizing multichannel (spatial) audio. To summarize, we make the following contributions : (1) the first 3D audio-visual segmentation benchmark composing of fairly complex indoor room scenes with integrated spatial sound cues; (2) a training-free AVS framework, EchoSegnet , capable of syncing across sequential frames from 3D environments; (3) a novel Audio-Informed Spatial Refinement Module AISRM , designed to enhance 3D segmentation and resolve ambiguities in complex, multi-instance environments by leveraging spatial audio intensity maps. We perform a comprehensive evaluation of EchoSegnet on the proposed 3DAVS-S34-O7 for both single-instance and multi-instance scenarios, along with an ablative comparison with existing 2D AVS models, highlighting their shortcomings in aligning audio-visual cues within 3D scenes -establishing their adaptation to 3DAVS-S34-O7 as non-trivial. 2 Related Work Audio-visual segmentation Existing AVS methods cater to 2D scenes with mono audio as inputs to identify audible visual pixels associated with a given audio signal [ 43 , 12 , 22 , 27 , 34 , 42 ] and are typically trained on thousands of manually annotated 2D segmentation masks. Although there have been recent improvements on reducing the dependence on annotated AV masks using weakly supervised [ 26 , 20 ] or entirely unsupervised methods [ 5 , 19 , 39 , 7 ] , there has not been any effort extending the task of AVS particularly to 3D scenes, with spatial audio cues. To address this gap, we propose the first benchmark for 3D AVS harnessing existing embodied AI platforms (Habitat simulator [ 32 ] ) to capture visual and (binaural) acoustic cues for sounding objects placed in 3D indoor scenes. We believe this lays a prominent groundwork for systematic evaluation of future embodied systems for 3D segmentation. 3D scene representations Point-based rendering techniques, initiated by [ 10 ] , utilize point-based explicit representation where each point affects a single pixel. Zwicker et al. [ 44 ] advanced this with ellipsoid-based rendering (splatting), allowing mutual overlap to fill image holes. In the absence of given geometry, Mildenhall et al. [ 24 ] explored neural implicit representation, NeRF, predicting view-dependent radiance via implicit density fields. 3D Gaussian Splatting (3D-GS) [ 15 ] , a novel-view synthesis method, employs explicit point-based representation, contrasting with NeRF’s volumetric rendering. Owing to its real-time high-quality rendering capabilities, 3D-GS has been applied to various domains, including simultaneous localization [ 14 , 23 ] , content generation [ 36 ] , and 4D dynamic scenes [ 18 , 38 ] , among others. In this work, we utilize the explicit representation from learned 3D-GS and decompose 2D masks of potential sounding objects to obtain consistent 3D masks. 3 Dataset Our proposed 3DAVS-S34-O7 is profoundly motivated towards simulating real-world indoor scenes, in terms of the visual quality of the scenes as well as the acoustic response generated by the objects placed within it. In the context of our 3D AVS task, we define an observation as O = \{(v_{i},a_{i},m_{i})\}_{i=1}^{n}

Read the complete text