AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering
Jiayu Zhang, Qilang Ye, Shuo Ye, Xun Lin, Zihan Song, Zitong Yu · 2025
Read AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering by Jiayu Zhang, Qilang Ye, Shuo Ye, Xun Lin, Zihan Song, Zitong Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18346.txt.
Introduction
Jiayu Zhang, with Shuo Ye and four co-authors, presents AV-Master in a 2025 paper on audio-visual question answering. The task asks models to answer questions using both sound and video together. Long scenes carry much redundant content, and fixed sampling misses the moments a question actually needs.
AV-Master samples adaptively over time, focusing step by step on the most relevant segments. It also weighs each modality, meaning sound versus vision, according to what the question needs. A dual-path contrastive loss, which pulls matching views together and pushes others apart, ties both dimensions together. Tests on four large benchmarks show clear gains on complex questions.
Catalog record
| CREATOR | Jiayu Zhang, Qilang Ye, Shuo Ye, Xun Lin, Zihan Song, Zitong Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18346
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering Jiayu Zhang, Qilang Ye, Shuo Ye, Xun Lin, Zihan Song, Zitong Yu Abstract Audio-Visual Question Answering (AVQA) requires models to effectively utilize both visual and auditory modalities to answer complex and diverse questions about audio-visual scenes. However, existing methods lack sufficient flexibility and dynamic adaptability in temporal sampling and modality preference awareness, making it difficult to focus on key information based on the question. This limits their reasoning capability in complex scenarios. To address these challenges, we propose a novel framework named AV-Master. It enhances the model’s ability to extract key information from complex audio-visual scenes with substantial redundant content by dynamically modeling both temporal and modality dimensions. In the temporal dimension, we introduce a dynamic adaptive focus sampling mechanism that progressively focuses on audio-visual segments most relevant to the question, effectively mitigating redundancy and segment fragmentation in traditional sampling methods. In the modality dimension, we propose a preference-aware strategy that models each modality’s contribution independently, enabling selective activation of critical features. Furthermore, we introduce a dual-path contrastive loss to reinforce consistency and complementarity across temporal and modality dimensions, guiding the model to learn question-specific cross-modal collaborative representations. Experiments on four large-scale benchmarks show that AV-Master significantly outperforms existing methods, especially in complex reasoning tasks. I Introduction Fig. 1: Illustration of the AVQA task and the comparison of our method with previous work. Our method employs dynamic adaptive focus sampling to capture key audio-visual segments and predicts modal preferences through global preference activation to enhance the model, ultimately generating the correct answer. Humans perceive the world through various modalities, such as vision, hearing, and touch. Inspired by such multisensory experiences, researchers have increasingly focused on a range of multimodal understanding tasks [ 1 , 2 , 3 ] . Among these tasks, Audio-Visual Question Answering (AVQA) [ 4 , 5 , 6 ] is a practical task with broad application prospects. AVQA utilizes both visual and auditory modalities and requires the model to discover the associations between them to answer various questions. This process involves dynamically understanding audio-visual segments and addressing question-specific modality preferences, significantly increasing the complexity of the task. For AVQA, we argue it is essential to focus on the following: (i) How can the model identify the most relevant visual and audio segments of the video related to the given question? (ii) How can the model extract cues from the most relevant modality information related to the given question? For (i), most current models employ discrete [ 7 , 5 ] or continuous [ 6 ] sampling methods. Although these methods have achieved promising results, they still have notable limitations. The former compromises the inherent temporal cues in audio-visual segments, resulting in hallucinations during fine-grained understanding. The latter partially alleviates this issue but introduces significant redundancy, which hinders further enhancement of model performance. For (ii), most mainstream methods are not specifically designed to differentiate between various input modalities but instead treat them merely as supplementary information. A small number of researchers [ 8 ] have recognized these challenges and opted to design fusion weights for features from different modalities during the multimodal interaction process, thereby biasing the model towards a particular modality. However, this method overlooks the challenges in learning significant differences among fine-grained features and requires the model to reinterpret modality preferences in complex multimodal semantics, without explicitly enhancing the learning process, thus providing limited guidance for improving the model’s performance. To solve these issues, we propose AV-Master, which can find the important fine-grained areas related to the current question within complex dynamic audio-visual scenes, and generate the optimal answer by integrating global preference information. As shown in Fig. 1 , unlike previous work [ 5 , 6 ] , AV-Master employs dynamic adaptive focus sampling to extract the fine-grained focus features from the audio-visual segments relevant to the current questioning scenario. The focus sampling combines the advantages of discrete ( e.g. , uniform [ 7 ] , top-k [ 5 ] ) and continuous ( e.g. , gaussian, gaussian-experts [ 6 ] ) sampling methods, allowing the model to capture all continuous time steps while significantly reducing redundant information obtained from sampling. Furthermore, we propose a global preference activation strategy focused on model modality preferences. This strategy determines the preference distribution of the model in different modalities through independent perception and improves the decision-making capabilities of the model. Compared to using dynamic weights at the multimodal fusion stage, our enhancement strategy focuses on the global information in the initial features. This avoids biases caused by decoding from complex multimodal features, while also complementing the fine-grained information obtained through temporal dynamic perception, achieving comprehensive audio-visual understanding. Our contributions are summarized as follows: • We propose a dual-path audio-visual learning model named AV-Master that enhances the understanding of audio-visual scenes by perceiving fine-grained details and global modality preferences related to questions, achieving cross-modal mapping from audio-visual signals to textual answers. • We introduce a dynamic adaptive focus sampling method that progressively performs adaptive learning from the input audio-visual segments during the encoding, enabling precise capture of the focal areas within a large number of redundant segments. • Extensive evaluation on four benchmark AVQA datasets demonstrates that our proposed AV-Master is superior and achieves new state-of-the-art performance compared to existing AVQA methods. The remainder of this paper is organized as follows. In Section II , we review the related work relevant to the research direction of this study. Section III provides a detailed description of the architecture of the proposed model. Section IV presents experimental results that validate the effectiveness of our proposed methods. Finally, in Section V , we conclude by summarizing the main contributions of this paper. II Related Work II-A Audio-Visual Scene Understanding In recent years, audio-visual scene understanding has emerged as a significant research area, garnering ever-increasing attention from the academic community. The core idea of this field originates from the human instinct to perceive the world through the synergy of multiple senses. Specifically, visual and auditory information are not only complementary but also often inseparable in understanding complex, dynamic environments. These two modalities are tightly linked through semantic consistency (e.g., the image of a dog matching the sound of its bark) and spatio-temporal correlation (a sound synchronizing with and emanating from its visual source). This interplay provides the key elements for achieving a sophisticated understanding of scenes that surpasses the capabilities of single-modality perception. Building on this foundation, researchers have explored numerous specific sub-tasks such as sound source localization [ 9 , 10 , 11 ] , action recognition [ 12 , 13 ] , event detection [ 14 , 15 , 16 ] , video parsing [ 17 , 18 , 19 ] , audio-visual source separation [ 20 , 21 ] , etc. These tasks aim to fully leverage the interaction and fusion of audio and visual information to overcome the limitations of unimodal perception, thereby enhancing the fine-grained understanding of dynamic audio-visual scenes. Within this broad research landscape, our work focuses on the more advanced cognitive task of audio-visual question answering. We achieve precise scene understanding and reasoning based on multimodal information by specifically emphasizing the temporal dynamic perception of audio-visual