Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

Yongbiao Gao, Xiangcheng Sun, Guohua Lv, Deng Yu, Sijiu Niu, Shandong Computer Science Center (National Supercomputer Center in Jinan),Qilu University of Technology (Shandong Academy of Sciences), Jinan, China · 2024

Read Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing by Yongbiao Gao, Xiangcheng Sun, Guohua Lv, Deng Yu, Sijiu Niu, Shandong Computer Science Center (National Supercomputer Center in Jinan),Qilu University of Technology (Shandong Academy of Sciences), Jinan, China (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19563.txt.

Introduction

Yongbiao Gao and co-authors clean noisy labels for video parsing in a 2024 paper. The task asks a system to name sound and sight events in a video and mark when each starts and ends. Only whole-video labels are given, so many fine labels start out wrong.

Older cleaning tools run apart from the parsing model, which breaks the feedback between them. Their joint method trains both parts together through reinforcement learning. A reward built from parsing quality steers the cleaning policy directly. Tests beat standalone cleaning methods, and plugging the cleaner into other parsing models lifted their scores too.

Catalog record

CREATORYongbiao Gao, Xiangcheng Sun, Guohua Lv, Deng Yu, Sijiu Niu, Shandong Computer Science Center (National Supercomputer Center in Jinan),Qilu University of Technology (Shandong Academy of Sciences), Jinan, China
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing Yongbiao Gao, Xiangcheng Sun, Guohua Lv, Deng Yu, Sijiu Niu, Shandong Computer Science Center (National Supercomputer Center in Jinan),Qilu University of Technology (Shandong Academy of Sciences), Jinan, China Abstract Audio-visual video parsing (AVVP) aims to recognize audio and visual event labels with precise temporal boundaries, which is quite challenging since audio or visual modality might include only one event label with only the overall video labels available. Existing label denoising models often treat the denoising process as a separate preprocessing step, leading to a disconnect between label denoising and AVVP tasks. To bridge this gap, we present a novel joint reinforcement learning-based label denoising approach (RLLD). This approach enables simultaneous training of both label denoising and video parsing models through a joint optimization strategy. We introduce a novel AVVP-validation and soft inter-reward feedback mechanism that directly guides the learning of label denoising policy. Extensive experiments on AVVP tasks demonstrate the superior performance of our proposed method compared to label denoising techniques. Furthermore, by incorporating our label denoising method into other AVVP models, we find that it can further enhance parsing results. Keywords: Video Parsing, Reinforcement Learning, Label Denoising, Weakly-Supervised 1 Introduction The way humans perceive knowledge is through multiple modalities, including auditory, visual, tactile, and taste. Numerous studies have demonstrated that integrating these various senses can enhance natural knowledge perception [ 17 , 29 , 26 , 4 ] . Specifically, visual and auditory are particularly prevalent in conveying vast amounts of information during interactions with the environment. In recent years, the majority of research focus on visual scene understanding [ 15 , 33 ] or video parsing [ 24 , 23 ] , often overlooking other modalities. Prior works in multi-modal learning [ 3 , 39 ] primarily examine the synergy between text and visual modalities to acquire joint representations. However, auditory and visual data often coexist and provide complementary cues in human perception. Therefore, more recently, some works [ 31 , 41 , 5 , 45 , 25 , 27 , 18 ] investigate on localizing audio and visual event tasks, called Audio-Visual Video Parsing (AVVP), which aims to generate temporal audio, video, and audio-visual event labels. Most of existing works explore weakly-supervised learning for AVVP task since densely annotated event modality and category label with temporal boundaries is extremely expensive and time-consuming. For instance, [ 31 ] formulates the AVVP task as a Multimodal Multiple Instance Learning (MMIL) problem and introduces an attentive MMIL pooling method to adaptively explore useful audio and visual content from different temporal extents and modalities. While existing audio-visual parsing works often assume temporal alignment of audio and visual events, this alignment hypothesis may not hold in practical scenarios. As pointed out by [ 38 , 6 ] , ”some event clues that do not appear in one of the two modalities.” This is exemplified by the saying ”I have not seen the person, but I have heard his voice.” For example, in a basketball game video, even if the commentator cannot be seen, their voice and cheering can be heard, making sound signals crucial for comprehensive video understanding. Thus, the commentator’s speech serves as a noisy label for the visual modality, and the basketball is considered audio-specific noise in some video clips. This issue is defined as modality-specific noisy . Figure 1: The overall framework of our proposed joint training RLLD for AVVP. The label denoising module aims to generate the denoising policy and the task module feedback the joint reward to guide the label denoising module learning. Pioneering researchers in the field of audio-visual video parsing have made concerted efforts to address the modality-specific noisy problem. For example, an audio-visual modality exchange method [ 38 ] aimed at deriving precise modality-aware event labels is proposed to, thus safeguard models from being misled by ambiguous overall labels. In a similar vein, [ 6 ] tackle the modality-specific noisy issue by collaboratively computing a nosie estimator and forward modality losses. Leveraging the estimated noise ratios, they prioritize noisy samples based on forward losses. Notably, both the cross-modal and joint modal exchange mechanisms treat label denoising as a preliminary stage, disconnected from the subsequent AVVP process. This limitation precludes direct guidance of the denoising process by AVVP outcomes. Recently, VALOR [ 43 ] incorporates large-scale contrastively pre-trained models as the modality teachers to enhance learning with weak labels. An iterative optimization approach [ 27 ] using the EM algorithm is proposed to overcome the challenge of weak labels. However, the two most recent works ignore the modality-specific noisy problem. To address this gap, we introduce an innovative training strategy that integrates label denoising and audio-visual video parsing into a unified learning framework. This approach enables seamless interaction between denoising and parsing components, leading to more robust and accurate audio-visual video analysis. In this paper, we introduce a novel Reinforcement Learning-based Label Denoising (RLLD) framework for audio-visual video parsing. The objective of RLLD is to enhance the identification of noisy labels and facilitate label denoising, thereby improving the overall performance of AVVP. Figure 1 provides an overview of the RLLD framework. The core component of RLLD is a reinforcement learning agent that is trained to identify noisy labels. This agent is guided by a joint reward signal derived from both the validation results and the soft inter-reward. This approach ensures that the denoising agent can maintain prediction accuracy while reducing convergence time. In addition, RLLD integrates label denoising and AVVP into a unified training framework. This integration enables the direct mitigation of label denoising and AVVP, facilitating more effective learning and optimization. RLLD not only directly identifies noisy labels from a single modality, eliminating the need for computationally expensive cross-modal comparison, but also trains the denoising agent using validation results from the AVVP model. This approach ensures a more direct impact on enhancing AVVP performance, leading to superior temporal localization outcomes. We validate the effectiveness of RLLD on the standard AVVP dataset under several settings. The results demonstrate that RLLD achieves better performance than existing state-of-the-art label denoising models and further enhances the performance of state-of-the-art AVVP models. To summarize, our key contributions are as follows: • We propose to address the label denoising issue in AVVP by reinforcement learning to learn the noisy label discovery strategy. To the best of our knowledge, we are the first to introduce reinforcement learning into AVVP to solve the label denoising problem. • We carefully design the validation-based reward and the soft inter-reward functions to guide the label denoising strategy learning, which directly drives the label denoising for improving the performance of AVVP. • The experiments on the AVVP dataset validate the effectiveness of our method, which can alleviate modality-specific noise during the jointly training framework. 2 Related Work The traditional audio-visual learning primarily emphasize the representation learning [ 1 , 19 , 2 , 7 , 16 ] , video captioning [ 28 , 20 , 40 ] , audio-visual sound separation [ 35 , 8 , 11 ] , etc . For example, [ 2 ] propose a knowledge distillation-based procedure to transfer discriminative visual-knowledge from well established visual recognition models into the sound modality to get the

Read the complete text