Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing
Yaru Chen, Ruohao Guo, Liting Gao, Yang Xiang, Qingyu Luo, Zhenbo Li, Wenwu Wang · 2025
Read Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing by Yaru Chen, Ruohao Guo, Liting Gao, Yang Xiang, Qingyu Luo, Zhenbo Li, Wenwu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14097.txt.
Introduction
Yaru Chen and six co-authors improve audio-visual video parsing in a 2025 paper. Parsing means labeling which events in a video are heard, seen, or both, moment by moment. In the weakly supervised setting, training data carries only whole-video labels. A model must guess where each event sits in time. Earlier methods leaned on global predictions that wobble at segment level.
Two strategies stabilize the guesses. An exponential moving average teacher generates pseudo-labels for segments. The labels are chosen by adaptive thresholds or top picks, giving steady temporal supervision. A class-aware agreement loss then pulls audio and visual embeddings together. It acts only at segment-class pairs judged reliable. On the LLP and UnAV-100 datasets, the method reached the best reported results across several metrics.
Catalog record
| CREATOR | Yaru Chen, Ruohao Guo, Liting Gao, Yang Xiang, Qingyu Luo, Zhenbo Li, Wenwu Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14097
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing Yaru Chen, Ruohao Guo, Liting Gao, Yang Xiang, Qingyu Luo, Zhenbo Li, Wenwu Wang Abstract Weakly-supervised audio-visual video parsing (AVVP) seeks to detect audible, visible, and audio-visual events without temporal annotations. Previous work has emphasized refining global predictions through contrastive or collaborative learning, but neglected stable segment-level supervision and class-aware cross-modal alignment. To address this, we propose two strategies: (1) an exponential moving average (EMA)-guided pseudo supervision framework that generates reliable segment-level masks via adaptive thresholds or top- k k selection, offering stable temporal guidance beyond video-level labels; and (2) a class-aware cross-modal agreement (CMA) loss that aligns audio and visual embeddings at reliable segment–class pairs, ensuring consistency across modalities while preserving temporal structure. Evaluations on LLP and UnAV-100 datasets shows that our method achieves state-of-the-art (SOTA) performance across multiple metrics. † † address: 1 Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey, United Kindom 2 School of Intelligence Science and Technology, Peking University, China 3 College of Information and Electrical Engineering, China Agricultural University, China 1 Introduction Weakly-supervised audio-visual video parsing (AVVP) [ 1 ] aims to localize audible, visible, and audio-visual events in unconstrained videos. As illustrated in Fig. 1 , only video-level annotations are available during training, making it highly challenging to infer precise temporal boundaries and modality-specific events. This task is critical for applications such as audio-visual understanding, event detection, and segmentation [ 2 , 3 , 4 ] . Recent AVVP studies have explored strategies such as multi-instance learning for weak labels [ 1 , 5 ] , attention mechanisms to highlight informative segments [ 6 , 7 ] , and contrastive or collaborative learning to exploit audio–visual correlations [ 8 , 9 ] . Although these approaches improve performance, two key challenges remain. Figure 1: Illustration of the AVVP task. First, the lack of segment-level supervision, due to training with only video-level labels, makes it difficult to achieve stable learning. Previous works often propagate coarse labels to all segments or apply simple thresholding, which introduces noise [ 10 , 11 ] . Although large pre-trained models (e.g., CLIP [ 12 ] , CLAP [ 13 ] ) have been used to generate pseudo labels [ 14 , 15 ] , these are typically static and cannot be refined during training, leaving them prone to noise and domain mismatch. Thus, more reliable, dynamically updated pseudo supervision is needed. Second, most cross-modal approaches align modalities by maximizing global audio–visual similarity [ 8 , 16 ] , overlooking that different classes may occur in different modalities at different times. Without class-aware, segment-level alignment, models risk forcing mismatches between unrelated events, leading to suboptimal localization. Figure 2: Framework of E-CMANet. The EMA Teacher provides stable pseudo labels to guide the Student, and the CMA module enforces class-aware cross-modal alignment. The final loss combines Pseudo loss, AVVP loss, and CMA loss. To overcome these limitations, we proposed E-CMA, which includes two strategies. (1) Exponential moving average (EMA)-guided pseudo supervision [ 17 ] . We adopt a teacher–student framework where the teacher, updated via EMA of student parameters, periodically generates segment-level pseudo masks from frame-wise predictions using adaptive thresholds or top-k selection per class. This transforms noisy video-level labels into stable, dynamically updated supervision, reducing error propagation from static pseudo labels. (2) Class-aware cross-modal agreement (CMA) loss. Rather than enforcing global similarity between audio and visual embeddings, CMA aligns modalities only at confident temporal–class positions where both modalities strongly indicate the same event. This selective alignment prevents over-matching asynchronous content and provides fine-grained, event-consistent supervision. We evaluated our method on the widely used AVVP benchmarks, the LLP dataset [ 1 ] and also the weakly-labeled UnAV-100 dataset [ 18 ] . Experimental results show that our approach achieves state-of-the-art (SOTA) performance on multiple metrics, which highlights the importance of stable pseudo supervision and fine-grained cross-modal agreement for advancing weakly-supervised AVVP. 2 Proposed Methods 2.1 Problem Statement The goal of AVVP is to determine whether the event occurring in each time segment of a video is audio-only, video-only, or audio-visual event and to determine its category. Given a T second video \mathcal{X} that is partitioned into T non-overlapping segments, each one second long. We denote the sequence as \mathcal{X}=\{(x_{t}^{a},x_{t}^{v})\}_{t=1}^{T} , where x_{t}^{a},x_{t}^{v}\in\mathbb{R}^{d} represent the audio and visual features at time t and d represents the dimension of the features. For each segment, we define the event labels as y_{t}^{a}\in\{0,1\}^{C},y_{t}^{v}\in\{0,1\}^{C},y_{t}^{av}\in\{0,1\}^{C} , where C is the number of event classes. The audio-visual event occurs only when both modalities detect the same event at time t , that means y_{t}^{av}=y_{t}^{a}\odot y_{t}^{v} , where \odot denotes element-wise multiplication. AVVP is often a weakly supervised task, as only video-level labels are available during training, i.e. y\in\{0,1\}^{C} . In contrast, evaluation is performed with segment-level annotations, where fine-grained event boundaries are provided for both modalities. 2.2 Framework As shown in Fig. 2 , our framework builds upon the CoLeaF [ 8 ] baseline. We first extract audio and visual features using pre-trained CLAP [ 13 ] and CLIP [ 12 ] encoders. These features are refined by a Hierarchical Attention Network (HAN) [ 1 ] with self- and cross-attention to capture intra- and inter-modal dependencies. The updated segment representations are aggregated by (Multimodal Multiple Instance Learning) MMIL pooling [ 1 ] for video-level predictions. To enhance temporal supervision, we further introduce an EMA-guided teacher–student scheme. The teacher, updated by the exponential moving average, periodically generates stable segment-level pseudo masks that supplement weak video-level labels. In addition, a class-aware CMA loss is applied to enforce alignment between audio and visual embeddings in confident event segments. These novel designs enable more reliable segment-level learning and fine-grained cross-modal alignment. 2.3 Teacher-Guided Pseudo Supervision To address the lack of segment-level annotations and the limitations of fixed, non-adaptive pseudo labels, we introduce an EMA-guided teacher–student framework, where a slowly updated teacher generates reliable segment-level masks, which are then used to supervise the training of the student network. As shown in Fig. 2 , the student network follows the architecture we introduced in Section 2.2 , which includes CLAP and CLIP encoders, HAN aggregation, and MMIL pooling. It is updated by backpropagation with standard weakly-supervised objectives. The student network produces audio and visual probabilities \hat{P}^{a}_{t},\hat{P}^{v}_{t}\in[0,1]^{C} . Then we fuse them into a joint prediction vector, \hat{P}_{t}=\frac{1}{2}(\hat{P}^{a}_{t}+\hat{P}^{v}_{t}),\hat{P}_{t}\in[0,1]^{C} (1) The teacher network shares the same backbone architecture as the student network but is not optimized by gradient descent. Instead, its parameters are updated as an EMA [ 17 ] of the student’s parameters, which maintains a weighted average of the previous student parameters to form a more stable teacher model. Let \theta denote the parameters of the student at iteration k , and \theta^{\prime} denote the teacher parameters, thus: \theta^{\prime}_{k}=\alpha\theta^{{}^{\prime}}_{k-1}+(1-\alpha)\theta_{k} (2) where \alpha\in[0,1) is a momentum coefficient. This update ensures the teacher evolves smoothly, making its predictions more stable than the student network. To provide effective temporal supervision for the student, we transform the teacher’s segment-level predictions into binary pseudo masks that indicate reliable event occurrences. The teacher’s audio and visual predictions are first fused into a joint score: \tilde{P}_{t}=\frac{1}{2}(\tilde{P}^{a}_{t}+\tilde{P}^{v}_{t}),\tilde{P}_{t}\in[0,1]^{C} (3) From \tilde{P}_{t} , we can derive binary pseudo masks M_{t}\in\{0,1\}^{C} in