RA-SSU : Towards Fine-Grained Audio-Visual Learning with R egion- A ware S ound S ource U nderstanding

Muyi Sun, , Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun · 2026

Read RA-SSU : Towards Fine-Grained Audio-Visual Learning with R egion- A ware S ound S ource U nderstanding by Muyi Sun, , Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09809.txt.

Introduction

Muyi Sun and coauthors define a finer audio visual task: naming and outlining exactly which region makes each sound, frame by frame. Past work stayed coarse, asking only whether sound and sight match or where a source sits. They built music and everyday life data sets with masks and text for each frame. Their SSUFormer model tops the new benchmark for both outlining and describing sources.

Catalog record

CREATORMuyi Sun, , Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51435
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09809

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RA-SSU : Towards Fine-Grained Audio-Visual Learning with R egion- A ware S ound S ource U nderstanding Muyi Sun, , Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun Abstract Audio-Visual Learning (AVL) is one fundamental task of multi-modality learning and embodied intelligence, displaying the vital role in scene understanding and interaction. However, previous researchers mostly focus on exploring downstream tasks from a coarse-grained perspective (e.g., audio-visual correspondence, sound source localization, and audio-visual event localization). Considering providing more specific scene perception details, we newly define a fine-grained Audio-Visual Learning task, termed Region-Aware Sound Source Understanding (RA-SSU) , which aims to achieve region-aware, frame-level, and high-quality sound source understanding. To support this goal, we innovatively construct two corresponding datasets, i.e. fine-grained Music (f-Music) and fine-grained Lifescene (f-Lifescene) , each containing annotated sound source masks and frame-by-frame textual descriptions. The f-Music dataset includes 3,976 samples across 22 scene types related to specific application scenarios, focusing on music scenes with complex instrument mixing. The f-Lifescene dataset contains 6,156 samples across 61 types representing diverse sounding objects in life scenarios. Moreover, we propose SSUFormer , a S ound- S ource U nderstanding Trans Former benchmark that facilitates both the sound source segmentation and sound region description with a multi-modal input and multi-modal output architecture. Specifically, we design two modules for this framework, Mask Collaboration Module (MCM ) and Mixture of Hierarchical-prompted Experts (MoHE) , to respectively enhance the accuracy and enrich the elaboration of the sound source description. Extensive experiments are conducted on our two datasets to verify the feasibility of the task, evaluate the availability of the datasets, and demonstrate the superiority of the SSUFormer, which achieves SOTA performance on the Sound Source Understanding benchmark. More details are available on our project page https://yixuan-wang1.github.io/AVUWEB/ . I Introduction In recent years, Audio-Visual Learning (AVL) has become a significant research direction in the field of multimodal learning [ 57 , 56 , 77 ] , encompassing various subtasks such as Audio-Visual Correspondence (AVC) [ 2 ] , Sound Source Localization (SSL) [ 50 ] , and Audio-Visual Event Localization (AVEL) [ 16 , 66 ] , etc. These subtasks aim to uncover the inherent connections within multi-modality data by combining audio and video information, accelerating the exploration of video understanding tasks [ 35 ] . However, existing AVL tasks are generally limited to holistic or coarse-grained information perception, such as audio-level and video-level category alignment, spatial sound source localization, and temporal event boundary detection. These coarse-grained approaches have limitations in complex and dynamic real-world scenarios, where finer scene details is essential for perception [ 57 ] . For instance, the AVC focuses on determining the match between audio and video, but it overlooks the spatial localization of the sound-producing objects. The SSL task is limited to region-level sound source localization, lacking detailed sound object understanding within scenes. The Audio-Visual Event Localization (AVEL) task advances by localizing audio-visual events in time using pre-defined event labels. However, it remains restricted to coarse-grained event classification and temporal localization, neglecting a deeper understanding of the form and semantic information for sound-producing objects. To address these limitations, in this paper, we propose RA-SSU , a fine-grained audio-visual learning task, which takes the granularities of space and time into account, with Region-Aware and frame-level Sound Source Understanding . Unlike previous coarse-grained tasks, RA-SSU is designed for task-specific scenarios, emphasizing not only spatial awareness localization, but also more precise semantic understanding of sound sources, thereby enhancing fine-grained perception capabilities for Sound-Source Understanding in complex situations. The architecture of the RA-SSU task and its differences compared to previous AVL tasks are illustrated in Fig. 1 . Moreover, by focusing on spatial awareness and detailed semantic descriptions of sound-producing objects, the RA-SSU task demonstrates strong potential to enhance the understanding of visual and auditory events. For instance, in task like Audio-Visual Retrieval [ 40 , 77 ] , RA-SSU could significantly improve the system’s ability to identify sounding objects, enabling more precise matching and retrieval of audio-visual content. In addition, in Audio-Visual Caption [ 49 ] , RA-SSU directly enhances the quality of generated descriptions, offering more accurate and detailed semantic information for automatic captioning. Therefore, the RA-SSU task provides broader potential and a wider range of application scenarios for downstream audio-visual tasks. Fig. 1: Comparisons of the Region-Aware Sound-Source Understanding task with several previous Audio-Visual Learning tasks. (a).Audio-Visual Correspondence. Holistic correspondence with audio-level and video-level alignment. (b).Sound Source Localization. Coarse-grained (spatial) and region-level sound source localization. (c).Audio-Visual Event Localization. Coarse-grained (temporal) event localization with frame boundaries. (d). Region-Aware Sound-Source Understanding. Fine-grained audio-visual learning with regional sound source localization and sound region description. To validate the feasibility of the RA-SSU task, we construct two fine-grained audio-visual datasets, fine-grained Music (f-Music) and fine-grained Lifescene (f-Lifescene) . The f-Music dataset focuses on music scenes, characterized by complex audio mixtures and background noise. In contrast, the f-Lifescene dataset encompasses more intricate everyday scenarios, featuring dynamic interactions among multiple sounding objects. Compared to music scenes, the everyday scenario dataset presents greater challenges due to its increased scene complexity and semantic richness. This complexity necessitates the models effectively manage intricate multi-modality information and achieve nuanced semantic understanding. These two datasets offer enhanced annotation precision and greater scene diversity, featuring frame-level sound source masks and detailed textual descriptions. This rich data support enables the models to train more effectively, improving their performance in complex scenarios compared to existing datasets. For the size of the datasets, f-Music contains 3976 samples, including 22 types of instrument sound scenes. F-Lifescene contains 6156 samples, including 61 types of ordinary living environments. For the data annotation, we utilize the interactive Large Vision Models [ 26 ] and Large Language Models [ 36 ] to obtain the initial rough masks and text description annotations. Then, we manually refine the annotation results of the algorithm, resulting in region-aware, frame-level, high-quality masks and descriptions. The illustrations of the selected data samples are listed in Fig. 2 . Fig. 2: Data samples from the created f-Music and f-Lifescene. The top three are from f-Music and down from f-Lifescene. For each sample, three frames are selected from 10 seconds. Each sample contains a 10s video and its frame-level sound source masks and descriptions. Specifically, the frame-by-frame description of a data portion varies with the change of the sounding object, as shown in the upper-right sample. (Zoom in for better details) To achieve the goal of the RA-SSU task, we design a unified framework, named SSUFormer , that accepts both video and audio as inputs, producing segmentation masks for the sounding objects and generating corresponding text descriptions. To enhance the spatial and semantic consistency of the sound-producing objects, we first propose a Mask Collaboration Module (MCM). In particular, MCM adaptively leverages the produced masks as spatial guidance to improve the semantic awareness of our framework. The interactive correlation between two output objectives is modeled by the regional collaborative learning. In this manner, MCM promotes our framework to yield high-fidelity textual captions with desirable semantic-spatial harmonic properties against the previous single-output counterparts. Considering the text descriptions should maintain the temporal consistency with the corresponding sequential visual frames, we further propose a Mixture of Hierarchical-prompted Experts (MoHE) module.

Read the complete text