Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

Zinuo Li, Xian Zhang, Yongxin Guo, Mohammed Bennamoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, Qiuhong Ke, University of Western Australia Alibaba Group Zhejiang Laboratory Monash University · 2025

Read Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM by Zinuo Li, Xian Zhang, Yongxin Guo, Mohammed Bennamoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, Qiuhong Ke, University of Western Australia Alibaba Group Zhejiang Laboratory Monash University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18110.txt.

Introduction

Zinuo Li and co-authors build a model that watches and listens to video at once, in a 2025 paper on finding specific moments in footage. People naturally combine sight, sound, and speech to locate a scene. Most models fuse sound poorly, which limits their grasp of long videos and the moments users ask about.

Their model, TriSense, joins vision, audio, and speech in one large language model. A connector piece reweights each sense based on the question, so the model still works when one input is missing. To train it, the team built a data set of over two million samples using an automated pipeline. Tests show broad skill gains.

Catalog record

CREATORZinuo Li, Xian Zhang, Yongxin Guo, Mohammed Bennamoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, Qiuhong Ke, University of Western Australia Alibaba Group Zhejiang Laboratory Monash University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18110

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM Zinuo Li, Xian Zhang, Yongxin Guo, Mohammed Bennamoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, Qiuhong Ke, University of Western Australia Alibaba Group Zhejiang Laboratory Monash University Abstract Humans naturally understand moments in a video by integrating visual and auditory cues. For example, localizing a scene in the video like “A scientist passionately speaks on wildlife conservation as dramatic orchestral music plays, with the audience nodding and applauding” requires simultaneous processing of visual, audio, and speech signals. However, existing models often struggle to effectively fuse and interpret audio information, limiting their capacity for comprehensive video temporal understanding. To address this, we present TriSense , a triple-modality large language model designed for holistic video temporal understanding through the integration of visual, audio, and speech modalities. Central to TriSense is a Query-Based Connector that adaptively reweights modality contributions based on the input query, enabling robust performance under modality dropout and allowing flexible combinations of available inputs. To support TriSense’s multimodal capabilities, we introduce TriSense-2M , a high-quality dataset of over 2 million curated samples generated via an automated pipeline powered by fine-tuned LLMs. TriSense-2M includes long-form videos and diverse modality combinations, facilitating broad generalization. Extensive experiments across multiple benchmarks demonstrate the effectiveness of TriSense and its potential to advance multimodal video analysis. Code and dataset will be publicly released. Figure 1 : TriSense supports segment captioning and moment retrieval for videos from audio, visual, and speech modalities, as well as any combination of them, covering a total of eight different tasks. 1 Introduction Human understanding of real-world events is inherently multimodal: we rely not only on vision but also on spoken language and audio cues to make sense of what is happening in a video. This integration allows us to interpret intention, emotion, and the significance of events—going beyond what is seen to include what is heard and said. However, existing Multimodal Large Language Models (MLLMs) often fall short of achieving this level of nuanced video temporal understanding. While advances in vision-language modeling and temporal localization have improved language–visual alignment [ 11 , 23 , 15 , 12 , 38 , 19 , 28 ] , most models still rely solely on visual inputs. As a result, they perform poorly on tasks requiring the integration of audio and speech—particularly when one or more modalities are missing, noisy, or contextually irrelevant. This stands in contrast to human perceptual robustness and significantly limits model generalization in real-world scenarios, such as accurately localizing events or generating rich multimodal descriptions. Challenges and Current Limitations. As highlighted above, the current state of multimodal temporal understanding remains limited. Building on these observations, two core challenges continue to hinder progress in multimodal temporal understanding: 1) Insufficient and Incomplete Training Data: Current datasets are often composed of short clips and lack large-scale, fully annotated examples across all three modalities—vision, audio, and speech—which are essential for effective multimodal pretraining [ 14 , 8 , 13 , 12 ] . This scarcity hampers the development of robust MLLMs. Moreover, real-world videos often contain incomplete or inconsistent modality coverage, due to factors like variable recording setups, intentional omissions (e.g., silent footage or background music), or the natural absence of certain signals in specific scenes. When models are trained predominantly on videos with all modalities present, they often fail at inference time when confronted with missing or degraded inputs—a common scenario in the wild. 2) Lack of Modality Adaptation: Current MLLMs are generally not equipped to assess the relative importance of each modality based on task or query context. Recent models such as LongVALE [ 10 ] and Qwen2.5-Omni [ 34 ] attempt to integrate multiple modalities but fall short in adaptivity. For instance, LongVALE compresses all modality tokens into a single representation, resulting in information loss and poor handling of missing modalities. It also lacks an adaptive dropout strategy, leading to unstable performance when modality availability varies. Qwen2.5-Omni introduces temporal positional embeddings, but still fails to capture fine-grained temporal dependencies, limiting its effectiveness on complex moment-level tasks, as demonstrated in our experiments. Key Contributions. We argue that understanding complex moments in video requires not only broader modality coverage but also an adaptive mechanism to selectively emphasize the most relevant modalities depending on the task and query. Our approach addresses these challenges through the following key contributions: 1) We introduce TriSense-2M , a large-scale multimodal dataset containing 2 million annotations. Each video instance in the dataset includes event-based annotations across vision, audio, and speech modalities, with flexible combinations and natural absences of modalities. The dataset supports a wide variety of scenes and includes long-form videos averaging 905 seconds—significantly longer than those in existing datasets—enabling deeper and more realistic temporal understanding. Importantly, queries are expressed in high-quality natural language, aligned with temporal annotations, and span diverse modality configurations to facilitate robust multimodal learning. 2) We propose TriSense , a triple-modality MLLM designed for both video segment captioning and moment retrieval under diverse modality configurations. As depicted in Figure 1 , TriSense is designed to handle multimodal video data with varying availability of vision, audio, and speech over temporal dimension. Crucially, it includes a Query-Based Connector that dynamically adjusts modality weights based on the query’s content and context. This allows the model to emphasize the most informative modalities (e.g., prioritizing vision if most relevant) while down-weighting irrelevant or missing ones—enabling strong performance even under incomplete modality conditions. 3) We conduct extensive experiments on two core tasks—video segment captioning and moment retrieval—across eight modality configurations, including zero-shot evaluation on public benchmarks. TriSense achieves strong performance on both the new TriSense-2M dataset and existing benchmarks, laying a solid foundation for future research in multimodal temporal video understanding. 2 Related Work 2.1 Video Temporal Understanding MLLMs Video temporal understanding focuses on modeling how events unfold over time within a video, enabling tasks such as moment retrieval, segment captioning, and dense video captioning. Vision-language models (VLMs) have demonstrated strong capabilities in solving real-world problems, including in zero-shot scenarios without task-specific fine-tuning. However, many of these models still face challenges when it comes to understanding temporal dynamics [ 36 , 5 ] . To address this, several models have been fine-tuned on video grounding datasets that emphasize temporal structure—such as TimeChat [ 27 ] , VTimeLLM [ 15 ] —to enhance their temporal reasoning abilities. More recently, Momentor [ 23 ] introduced a time encoder to correct errors caused by time token quantization, while VTG-LLM [ 12 ] employed specialized time tokens and temporal position embeddings to help video LLMs better capture the timing of events. In a different approach, TRACE [ 11 ] applied principles from causal language modeling to propose causal event modeling for videos. It also introduced a lightweight time tower for encoding temporal information, achieving solid performance in temporal understanding. Extending beyond vision-language modeling, Multimodal Large Language Models (MLLMs) integrate visual, audio, and speech modalities to enable richer video analysis. To improve performance, recent efforts have focused on incorporating these additional modalities. For example, LongVALE [ 10 ] compresses all modality tokens into a single token, but this leads to loss of fine-grained details such as audio context or speech intonation. Qwen2.5-Omni [ 34 ] , on the other

Read the complete text