Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval DOI: 10.1145/3746027.3754982 Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland. Proceedings of the 33rd ACM International Conference on Multimedi
Junan Lin, Daizong Liu, Xianke Chen, Xiaoye Qu, Xun Yang, Jixiang Zhu, Sanyuan Zhang, Jianfeng Dong · 2025
Read Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval DOI: 10.1145/3746027.3754982 Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland. Proceedings of the 33rd ACM International Conference on Multimedi by Junan Lin, Daizong Liu, Xianke Chen, Xiaoye Qu, Xun Yang, Jixiang Zhu, Sanyuan Zhang, Jianfeng Dong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04273.txt.
Introduction
Junan Lin and seven colleagues remind video search that ears help: their system weighs audio by usefulness before fusing it with sight and text, in a 2025 paper. Video moment retrieval finds the clip segment matching a query. Most systems ignore sound; a few fuse it blindly, though some audio is pure noise for the task.
Their model first guides vision and audio separately with the text query. A predictor, trained with pseudo-labels, scores how much the audio should count, damping noisy tracks. A fusion module then merges sound and picture at local, event, and global scales. A distillation trick covers videos whose audio is missing at run time. The team also built an audio-focused test set from a standard benchmark. Results set a new high for audio-aware retrieval. The moral is in the title: audio matters, when it matters.
Catalog record
| CREATOR | Junan Lin, Daizong Liu, Xianke Chen, Xiaoye Qu, Xun Yang, Jixiang Zhu, Sanyuan Zhang, Jianfeng Dong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval DOI: 10.1145/3746027.3754982 Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland. Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland ISBN: 979-8-4007-2035-2/2025/10 CCS: Information systems Multimedia and multimodal retrieval CCS: Information systems Video search Junan Lin, Daizong Liu, Xianke Chen, Xiaoye Qu, Xun Yang, Jixiang Zhu, Sanyuan Zhang, Jianfeng Dong Abstract . Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to the given query. To tackle this task, most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality. Although a few recent works try to tackle the joint audio-vision-text reasoning, they treat all modalities equally and simply embed them without fine-grained interaction for moment retrieval. These designs are counter-practical as: Not all audios are helpful for video moment retrieval, and the audio of some videos may be complete noise or background sound that is meaningless to the moment determination. To this end, we propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR. Specifically, after integrating the textual guidance with vision and audio separately, we first design a pseudo-label-supervised audio importance predictor that predicts the importance score of the audio, and accordingly assigns weights to mitigate the interference caused by noisy audio. Then, we design a multi-granularity audio fusion module that adaptively fuses audio and visual modalities at local-, event-, and global-level, fully capturing their complementary contexts. We further propose a cross-modal knowledge distillation strategy to address the challenge of missing audio modality during inference. To evaluate our method, we further construct a new VMR dataset, i.e. , Charades-AudioMatter, where audio-related samples are manually selected and re-organized from the original Charades-STA to validate the model’s capability in utilizing audio modality. Extensive experiments validate the effectiveness of our method, achieving state-of-the-art with audio-video fusion in VMR methods. Our code is available at https://github.com/HuiGuanLab/IMG . 1. Introduction Video Moment Retrieval (VMR) ( Anne Hendricks et al. 2017 ; Gao et al. 2017 ; Yang et al. 2024a ; Dong et al. 2024 ; Yang et al. 2024a ) aims to retrieve the part of the video that is relevant to the semantic of a given query. As a fundamental yet important task, it requires in-depth interaction between video and text semantics for accurate alignment and reasoning. Existing mainstream works ( Zhang et al. 2020b ; Zhang et al. 2020a ; Zhang et al. 2021b ; Yang and Wu 2022 ; Zhang et al. 2023 ; Yang et al. 2024b ; Dong et al. 2022 ; Dong et al. 2023 ) generally focus on naive visual and textual modalities and develop vision-text integration frameworks to retrieve the specific moment. However, in addition to the visual contexts, audio modality also contains valuable contexts within the video streams ( Bagchi et al. 2021 ; Pan et al. 2022 ; Mercea et al. 2022a ; Jiang et al. 2022 ; Mercea et al. 2022b ; Zhou and Lim 2021 ; Gogate et al. 2020 ) . Without considering the rich complementary contexts of the audio modality, previous VMR methods fail to distinguish different activities like “laughing" and “talking" that share a similar visual appearance. Therefore, exploring the interaction and fusion of audio, vision, and text modalities in VMR is a promising direction with research forward. To leverage audio information, several audio-based VMR methods ( Chen et al. 2023 ; Liu et al. 2022 ; Chen et al. 2020 ) have been proposed. However, these approaches typically extract features from audio, vision, and text modalities and apply a uniform aggregation strategy for joint reasoning, without considering their diverse contributions. For instance, PMI-LOC ( Chen et al. 2020 ) incorporates RGB, motion, and audio modalities, establishes pairwise interactions between modalities, UMT ( Liu et al. 2022 ) introduces a unified multimodal transformer framework for the integration of visual and audio information, while ADPN ( Chen et al. 2023 ) leverages the consistency and complementarity between audio and visual modalities for efficient audio fusion. Although these methods achieve relatively better performance than conventional VMR methods, they neglect that not all audios contribute to the final grounding as the audio of some videos may be complete noise or background sound. In practice, audio semantics exhibit considerable complexity and diversity, varying significantly across different scenarios. In certain instances, audio context serves as a valuable complement, enhancing the alignment with text semantics and facilitating accurate reasoning. Conversely, noisy audio can lead to erroneous textual associations. As shown in Figure 1 , for the first query “a person is laughing", leveraging audio context significantly aids in identifying the laughing action, which might be ambiguous using vision alone. However, for the second query “a person looks out a window", the audio modality provides no benefit and may even be detrimental, given that this action is primarily visually driven. Therefore, this motivates us to design a more dynamic audio-vision-text association framework that learns to selectively and adaptively aggregate appropriate contexts from audio and visual modalities for reasoning the specific text semantics. To this end, we make the first attempt to tackle a flexible audio-vision-text joint reasoning for the VMR task. In particular, we propose a novel Importance-aware Multi-Granularity fusion model (IMG) with three prediction branches: audio branch, visual branch, and audio-visual fusion branch. Initially, textual guidance is integrated separately with both visual and audio inputs. We then introduce an audio importance-aware module to tackle the issue of variable audio importance, which is crucial in vision-text pairs. This module is supervised by pseudo-labels derived from the retrieval loss of each branch. It effectively learns to assess the relative importance of audio compared to vision. Then, for the latter audio-vision context fusion, we design a multi-granularity fusion network, which establishes local-level and event-level to global-level audio-vision fusion, as a way to better discover key clues in audio for assisting text-specific activity understanding within video contents. In addition to using traditional retrieval loss for supervision, since the multi-modal fusion branch tends to show better performance than the individual vision/audio reasoning branch as the former fuses the positive contexts from both modalities, we also distillate the knowledge from the fusion branch to the weaker visual and audio branches, thus strengthening the performance of both branches and in-turn providing better feedback to the fusion branch to further improving the performance. Figure 1. (Top) Audio is a critical modality, outweighing the importance of vision. (Bottom) Audio is entirely irrelevant and considered noise relative to the vision. To sum up, the key contributions of our work are four-fold: • We propose a novel Importance-aware Multi-Granularity fusion network (IMG) to handle the audio-incorporated VMR task, which selectively fuses audio modal information of video samples at multiple granularities for final retrieval. • We introduce an audio importance predictor, guided by a loss-aware pseudo-importance generator during training, to identify and emphasize semantically relevant audio clues. This mechanism enables the model to selectively focus on informative audio cues while suppressing irrelevant or noisy background sounds. • We propose a cross-modal knowledge distillation strategy, which transfers knowledge from the more