Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation

Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, Di Hu · 2025

Read Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation by Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, Di Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13068.txt.

Introduction

Henghui Du and co-authors build one model that handles many sound-and-vision tasks together, in this 2025 paper. The tasks range from finding when or where a sound occurs to reasoning over whole scenes. Training them jointly usually breeds interference, since the data and goals differ.

Their model, Crab, makes the tasks cooperate on purpose. A rebuilt teaching data set spells out the reasoning that links the tasks, replacing bare word labels. Inside, a multi-head adapter lets facets of the mixed data be learned apart, then shared. The design aims at the unified grasp humans bring to sights and sounds.

Catalog record

CREATORHenghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, Di Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, Di Hu Abstract In recent years, numerous tasks have been proposed to encourage model to develop specified capability in understanding audio-visual scene, primarily categorized into temporal localization, spatial localization, spatio-temporal reasoning, and pixel-level understanding. Instead, human possesses a unified understanding ability for diversified tasks. Therefore, designing an audio-visual model with general capability to unify these tasks is of great value. However, simply joint training for all tasks can lead to interference due to the heterogeneity of audiovisual data and complex relationship among tasks. We argue that this problem can be solved through explicit cooperation among tasks. To achieve this goal, we propose a unified learning method which achieves explicit inter-task cooperation from both the perspectives of data and model thoroughly. Specifically, considering the labels of existing datasets are simple words, we carefully refine these datasets and construct an A udio- V isual U nified I nstruction-tuning dataset with E xplicit reasoning process ( AV-UIE ), which clarifies the cooperative relationship among tasks. Subsequently, to facilitate concrete cooperation in learning stage, an interaction-aware LoRA structure with multiple LoRA heads is designed to learn different aspects of audiovisual data interaction. By unifying the explicit cooperation across the data and model aspect, our method not only surpasses existing unified audio-visual model on multiple tasks, but also outperforms most specialized models for certain tasks. Furthermore, we also visualize the process of explicit cooperation and surprisingly find that each LoRA head has certain audio-visual understanding ability. Code and dataset: https://github.com/GeWu-Lab/Crab Figure 1 : We present Crab, a unified audio-visual scene understanding model with explicit cooperation, which can complete various audio-visual tasks. It is trained on an instruction-tuning dataset with explicit reasoning process, which clarifies the cooperative relationship among tasks. Furthermore, to alleviate the interference caused by the learning process of complex audiovisual data and facilitate concrete cooperation, an interaction-aware LoRA structure is designed to enable the model focus on different aspects of data interaction. † † * Intern at Tencent PCG, 🖂 Corresponding author. 1 Introduction “Alone we can do so little; together we can do so much.” — Helen Keller Over the past few years, many works have explored the utilization of audio and visual modality to enhance general scene understanding, thereby deriving multiple tasks, primarily categorized into temporal localization, spatial localization, spatio-temporal reasoning, and pixel-level understanding. These tasks include audio-visual event localization (AVE) [ 51 ] , audio-visual video parsing (AVVP) [ 52 ] , audio referred image grounding (ARIG) [ 47 ] , audio-visual question answering (AVQA) [ 27 ] , audio-visual segmentation (AVS) [ 65 ] and reference audio-visual segmentation (Ref-AVS) [ 56 ] , etc . Specifically, AVE and AVVP tasks focus on temporal localization, where the model needs to accurately predict the events occurring in the video and their temporal boundaries. The purpose of ARIG task is to endow models with grounding capabilities, enabling them to locate the position of sounding objects. The AVQA task requires the model to integrate temporal and spatial information to answer questions. To complete AVS and Ref-AVS tasks, the model needs to output segmentation mask of target objects at pixel level. While significant advancements have been made in exploring these tasks, these works [ 18 , 60 , 49 , 36 , 30 , 14 , 56 , 17 , 29 ] have mainly focused on completing specific task. By contrast, humans can generally possess a general perception and understanding for complex audio-visual scene, hence we also aspire to equip the model with similar capability. Unified general-purpose models can effectively transfer knowledge across tasks, and even perform unobserved tasks during training process. Early works primarily focused on unifying multiple language tasks, such as GPT2 [ 43 ] and PaLM [ 12 ] . Subsequently, large language models (LLMs) [ 53 , 61 , 11 ] have demonstrated human-level performance across various language tasks. Then researchers further extend LLMs into multimodal domain. A line of works [ 31 , 1 , 4 ] perform well on visual-language tasks. Recently, utilizing LLMs to complete multiple audio-visual tasks has become a goal pursued by the community [ 62 , 10 , 13 ] . They mainly accomplished this by naively curating instruction-tuning dataset and training jointly on multiple tasks, which is essentially an implicit approach to achieve inter-task cooperation. Considering the heterogeneity of audiovisual data and complex relationship among audio-visual tasks, it is difficult to more effectively achieve mutual cooperation among these tasks through such simple paradigm. For example, for two different types of tasks, temporal localization and spatial localization, instruction fine-tuning is effective when applied to one type of task individually, but directly training jointly on these two tasks could lead to interference. We argue that this problem can be addressed through explicit cooperation among tasks from both the data and model perspectives, thoroughly. On the one hand, considering the labels of existing datasets are relatively simple words or phrases, they are insufficient for different tasks to establish relationship. Therefore, it is essential to augment existing datasets to explicitly clarify the cooperative relationship among tasks. As shown in Fig. 1 , for spatio-temporal reasoning task, the label is no longer single words, but includes explicit reasoning process, such as the time when the first instrument makes a sound, the position of instruments, etc . These information can help establish cooperation relationship among tasks. On the other hand, implicit methods may cause interference in the learning stage of heterogenous audiovisual data, hence the models needs to focus on different aspects of audiovisual data interaction, facilitating concrete cooperation in learning stage. As shown in Fig. 1 , the model can focus on multiple interaction aspects such as temporal, spatial, and pixel-level, thereby facilitating concrete cooperation. To achieve this goal, we propose a unified audio-visual scene understanding model with explicit cooperation. Specifically, we first refine existing multiple audio-visual task datasets and construct an A udio- V isual U nified I nstruction-tuning dataset with E xplicit reasoning process ( AV-UIE ), which clarifies the concrete cooperative relationship among tasks. The learning objective of task is not only to predict simple labels, but also includes explicit reasoning process, which contains information from multiple spatio-temporal dimensions. The model can rely on these information to establish relationship among tasks. Then, to alleviate the interference and facilitate concrete cooperation in learning stage, an interaction-aware LoRA structure with multiple LoRA heads is designed. Each head is responsible for learning certain audiovisual data interaction aspect. By unifying the explicit cooperation across data and model aspect, our method not only surpasses existing unified audiovisual models on multiple tasks, but also outperforms the specialized models for certain tasks. Furthermore, we also visualize the process of explicit cooperation and surprisingly find that each LoRA head has certain audio-visual understanding ability. Our contributions can be summarized as follows: • Based on unification of audio-visual tasks, we propose an approach of explicit cooperation, which solves the problem of unified learning of complex audiovisual data. • We construct AV-UIE, an audio-visual unified instruction-tuning dataset with explicit reasoning process, which clarifies the cooperation relationship among tasks. • We design an interaction-aware LoRA structure

Read the complete text