TeMTG: Text-Enhanced Multi-Hop Temporal Graph Modeling for Audio-Visual Video Parsing DOI: 10.1145/3731715.3733495 Conference: The 15th ACM International Conference on Multimedia Retrieval; June 30–July 03, 2025; Chicago, USA ISBN: 979-8-4007-1877-9/25/06 CCS: Computing methodologies CCS: Artificial
Yaru Chen, Peiliang Zhang, Fei Li, Faegheh Sardari, Ruohao Guo, Zhenbo Li, Wenwu Wang · 2025
Read TeMTG: Text-Enhanced Multi-Hop Temporal Graph Modeling for Audio-Visual Video Parsing DOI: 10.1145/3731715.3733495 Conference: The 15th ACM International Conference on Multimedia Retrieval; June 30–July 03, 2025; Chicago, USA ISBN: 979-8-4007-1877-9/25/06 CCS: Computing methodologies CCS: Artificial by Yaru Chen, Peiliang Zhang, Fei Li, Faegheh Sardari, Ruohao Guo, Zhenbo Li, Wenwu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02096.txt.
Introduction
Yaru Chen and co-authors present a method to parse a video into its events, with both sound and image in play. The task is to name what happens in each part of a video and when, yet training often gives only weak labels for the whole clip.
Their method, TeMTG, adds text to sharpen the facts drawn from sound and image. Text forms are made by big pretrained models and fused with the raw features. A graph network that hops across time then models how segments link, near and far. On a standard data set the method hit top scores on most key measures, the authors report.
Catalog record
| CREATOR | Yaru Chen, Peiliang Zhang, Fei Li, Faegheh Sardari, Ruohao Guo, Zhenbo Li, Wenwu Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02096
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TeMTG: Text-Enhanced Multi-Hop Temporal Graph Modeling for Audio-Visual Video Parsing DOI: 10.1145/3731715.3733495 Conference: The 15th ACM International Conference on Multimedia Retrieval; June 30–July 03, 2025; Chicago, USA ISBN: 979-8-4007-1877-9/25/06 CCS: Computing methodologies CCS: Artificial intelligence CCS: Computer vision CCS: Computer vision tasks Yaru Chen, Peiliang Zhang, Fei Li, Faegheh Sardari, Ruohao Guo, Zhenbo Li, Wenwu Wang Abstract . Audio-Visual Video Parsing (AVVP) task aims to parse the event categories and occurrence times from audio and visual modalities in a given video. Existing methods usually focus on implicitly modeling audio and visual features through weak labels, without mining semantic relationships for different modalities and explicit modeling of event temporal dependencies. This makes it difficult for the model to accurately parse event information for each segment under weak supervision, especially when high similarity between segmental modal features leads to ambiguous event boundaries. Hence, we propose a multimodal optimization framework, TeMTG, that combines text enhancement and multi-hop temporal graph modeling. Specifically, we leverage pre-trained multimodal models to generate modality-specific text embeddings, and fuse them with audio-visual features to enhance the semantic representation of these features. In addition, we introduce a multi-hop temporal graph neural network, which explicitly models the local temporal relationships between segments, capturing the temporal continuity of both short-term and long-range events. Experimental results demonstrate that our proposed method achieves state-of-the-art (SOTA) performance in multiple key indicators in the LLP dataset. 1. Introduction In Audio-Visual Video Parsing (AVVP) ( tian2020unified ) task, our goal is not only to detect what events occur at what times, but also to determine which modality detects the event. Compared with other related tasks ( zeng2024anchor ; li2024boosting ; guo2024open ; li2022drcnet ) , a distinguishing characteristic of this task is the temporal asynchrony between events that occur in different modalities. As shown in Fig. 1 (a), we see cats for 10 seconds, while we hear it between 1-3 and 8-10 s, and we can still hear the sound of speech even if no one appears in the video. Therefore, events are often categorized into three types: audio events, visual events, and audio-visual events. AVVP is also trained under the weakly supervised scenario, which means that the model is trained using only limited video-level event labels. This setting increases the difficulty for models to learn the temporal details and modality correlations. AVVP can benefit a variety of real-world applications, such as intelligent video surveillance, content-based video indexing. Figure 1. (a) Illustration of the AVVP task. (b) Structure for Multi-Hop Temporal Graph (assume K=2 ). A baseline method ( tian2020unified ) was developed for the AVVP task by employing hybrid attention networks (HAN), where multi-modal multiple instance learning (MMIL) is used to aggregate the multi-modal temporal contexts, together with the identification and suppression of noisy labels for each modality. Subsequently, the researchers ( gao2023collecting ; sardari2024coleaf ) explored contrastive learning, distillation learning, and other techniques to connect semantically similar segments within and between modalities. With the emergence of large-scale pre-trained models, Lai et al. ( lai2023modality ) utilized pre-trained CLAP ( wu2023large ) and CLIP ( radford2021learning ) to extract features and generate segment-level pseudo-labels. Other researchers have explored the use of pseudo labels as references for the model to distinguish the semantics of each event that occurred in each segment ( zhou2024label ; fan2024revisit ) . Previous studies have explored the connections between the categories of events ( mo2022multi ; chen2024cm ) , but not the propagation and continuation of events in the temporal dimension. In addition, large-scale pre-trained models have been used to provide semantic information ( zhou2024advancing ; jiang2024resisting ) , which is used as pseudo labels and classification auxiliary. However, they have not been integrated deeply into feature representation. As a result, semantic consistency within the features cannot be guaranteed, causing potential misalignment in audio-visual feature fusion. Recently, text embeddings have been used to improve multimodal representation learning ( wang2024link ) . This method focuses primarily on encoding event-related semantics while neglecting background information, which, however, may contain crucial contextual cues to distinguish events. To tackle these challenges, we propose TeMTG, which combines text-enhanced semantic guidance with multi-hop temporal graph modeling for weakly supervised AVVP. We first introduce a fusion mechanism that leverages large-scale pre-trained models to generate segment-level pseudo labels and corresponding text embeddings, which are refined by a modality-specific multi-layer perceptron (MLP) ( taud2017multilayer ) to enhance semantic guidance and feature discriminability. Then, we construct a K-hop temporal graph to explicitly model segment-wise dependencies. By linking segments through K-hop edges, our model captures audio-visual correlations across time, improving temporal reasoning. Experimental results show that TeMTG effectively addresses AVVP limitations and achieves SOTA performance. 2. Proposed Methodology In the AVVP task, a video clip S can be divided into T segments, represented as S=\{A_{t},V_{t}\}_{t=1}^{T} , where A_{t} and V_{t} denote the audio and visual features of the t -th segment. The task requires segmenting events into three categories: audio events y_{t}^{a}\in\{0,1\}^{C} , visual events y_{t}^{v}\in\{0,1\}^{C} , and audio-visual events y_{t}^{av}\in\{0,1\}^{C} , where C is the total number of event classes. An event is considered audio-visual if it appears in both modalities simultaneously, i.e., y_{t}^{av}=y_{t}^{a}*y_{t}^{v} . During training, only weak video-level labels y\in\{0,1\}^{C} are provided. The goal of AVVP is, given a video clip divided into T segments of audio-visual features, to determine for each segment whether an event (among C possible classes) occurs in the audio modality, the visual modality, or both. Figure 2. TeMTG architecture: Audio and visual features are fused with text embeddings, aggregated, and then processed by multi-hop temporal graphs with multi-head residual GAT layers to model event dependencies. 2.1. Framework We adopt CoLeaF ( sardari2024coleaf ) as our baseline model and propose a novel multimodal optimization framework that integrates text enhancement and multi-hop temporal graph modeling. As shown in Fig. 2 , we first use the text encoder to generate text embeddings for the audio and visual stream of each video segment, respectively. These embeddings are then fused with the audio and visual features, respectively, through a feature fusion module. Next, the fused multimodal features are fed into the feature aggregation module which adopts self-attention and cross-attention to preserve unimodal feature learning and enhance cross-modal interactions. Then we construct a multi-hop temporal graph and propagate information using multi-head graph attention (GAT) ( velivckovic2018graph ) to model both short-term continuity and long-term dependencies among video segments. Finally, we employ MMIL pooling ( tian2020unified ) to aggregate temporal features and generate the final video-level predictions, including audio predictions P_{a} , visual predictions P_{v} , and joint audio-visual predictions P . As CoLeaF used two branches for feature aggregation, we placed our proposed multi-hop temporal GAT after each branch. 2.2. Text-Enhanced Multimodal Feature Fusion To enhance the semantic representation of the audio-visual features, we introduce text embeddings to provide explicit semantic guidance, effectively mitigating the limitations of weakly supervised learning. Specifically, inspired by ( lai2023modality ) , we first use CLAP and CLIP to generate pseudo labels at segment level p_{t}^{a},p_{t}^{v}\in\{0,1\}^{C} . Then we convert each pseudo label into a text description in the following format: " This is the sound of x audio event " or " This is the image of x visual event ". If a segment contains