Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup

Seokun Kang, Taehwan Kim · 2025

Read Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup by Seokun Kang, Taehwan Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02284.txt.

Introduction

Teaching a computer to name what happens in a video takes many labeled clips, and labels cost time and money. Seokun Kang and Taehwan Kim work on that problem in a 2025 paper, using both the picture and the sound of each clip. Sound has often been left out of such training, even though a splash or a shout can name the act.

Their trick blends two clips into one, guided by a map of where the sound in each clip comes from. That way the mixed clip keeps a sane link between what is seen and what is heard. On three known test sets the method beat prior work in its class. The gain shows that sound, used with care, can stretch a small set of labeled clips much further.

Catalog record

CREATORSeokun Kang, Taehwan Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02284

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup Seokun Kang, Taehwan Kim Abstract Video action recognition is a challenging but important task for understanding and discovering what the video does. However, acquiring annotations for a video is costly, and semi-supervised learning (SSL) has been studied to improve performance even with a small number of labeled data in the task. Prior studies for semi-supervised video action recognition have mostly focused on using single modality - visuals - but the video is multi-modal, so utilizing both visuals and audio would be desirable and improve performance further, which has not been explored well. Therefore, we propose audio-visual SSL for video action recognition, which uses both visual and audio together, even with quite a few labeled data, which is challenging. In addition, to maximize the information of audio and video, we propose a novel audio source localization-guided mixup method that considers inter-modal relations between video and audio modalities. In experiments on UCF-51, Kinetics-400, and VGGSound datasets, our model shows the superior performance of the proposed semi-supervised audio-visual action recognition framework and audio source localization-guided mixup. 1 Introduction Video is considered an important resource for deep learning vision research. Video data of different lengths and formats has become more readily available as large-scale video content offerings and user interactions on online platforms increase. This leads to active research on video understanding [ 44 , 23 , 24 , 26 ] and among them, video action recognition is one of the challenging tasks [ 52 , 8 , 14 , 22 , 36 , 56 ] . Unlike image classification, video action recognition is a challenging task that requires understanding both spatial and temporal information. This requires sufficient labeled training data, but labeling on video data is more difficult and time-consuming than images. Nevertheless, unlabeled video data is readily available, and semi-supervised learning (SSL) methods using labeled video and large-scale unlabeled video together are being actively studied [ 43 , 49 , 48 , 51 ] . Semi-supervised video action recognition research remains less explored compared to semi-supervised image classification [ 38 , 54 , 6 , 46 , 9 ] . Recent studies in SSL video recognition have actively explored various approaches beyond just using fixed, flexible, or even not predefined confidence-based thresholds [ 38 , 54 , 46 ] as proposed in the SSL image domain. These include leveraging pre-trained networks and large-scale unlabeled datasets [ 11 , 49 , 18 ] . In addition, research efforts are increasingly focusing on the utilization of additional modalities such as temporal gradient [ 48 ] and optical flow [ 50 ] obtained from video data or by introducing auxiliary networks [ 51 ] . Studies of semi-supervised learning for image and video use random augmentation techniques to generate weak and strong augmented images or videos and train the model through consistency regularization using both augmented [ 38 , 54 , 46 , 49 , 51 ] . In this context, weak augmentations typically involve random flips and rotations, while strong augmentations utilize RandAugment [ 10 ] that includes techniques such as CutMix [ 53 ] , which involves cutting and combining different images, and MixUp [ 55 ] , which mixes images through interpolation. However, SVFormer [ 49 ] points out the limitations of mixup and cutmix that do not produce the expected performance in the video domain, and to overcome these limitations, they propose TubeToken Mixup and Temporary Warping augmentation methods. However, there is still a limitation in that they cannot actively utilize all its information because only visual information is considered, not audio information. In semi-supervised video action recognition, using visual-audio modality has been under-explored. The only exception is AvCLR [ 3 ] , which employs audio-visual contrastive learning with ResNet architecture. However, despite originating from the same video clip, augmentations for video and audio modalities are conducted individually without profoundly considering the inter-modal relation between these modalities. In this paper, we propose the transformer-based framework for semi-supervised audio-visual action recognition with audio source localization-guided mixup. Specifically, we introduce a novel audio source localization-guided mixup method designed to preserve the inter-modal relations that tend to be overlooked when applying mixup or cutmix to visual and audio modalities. In most cases, visual and audio information occur simultaneously and are interrelated in videos. By preserving this inter-relationship within videos, models can better understand and reflect real-world situations. In addition, we leverage contrastive learning between video and audio modalities in the video clip. Incorporating audio source localization guided mixup with contrastive learning can improve performance substantially over existing state-of-the-art methods. In summary, our contributions are as follows: • We study semi-supervised audio-visual action recognition, which has been under-explored. To the best of our knowledge, we propose, for the first time, a visual-audio semi-supervised video action recognition approach based on the transformer model. This allows for the maximization of the use of visual and audio information inherent in videos. • We propose a novel audio source localization-guided mixup that, unlike mixup or cutmix, preserves the interrelation between the video and audio modalities. • In experiments, our proposed model outperforms the existing state-of-the-art model on three different benchmark datasets in even challenging scenarios with only a few labeled samples available. 2 Related Works 2.1 Semi-supervised Image Classification Deep learning-based image classification has achieved significant performance improvements by leveraging large-scale annotated datasets [ 41 , 20 , 15 , 16 , 13 , 27 ] . However, annotating requires considerable time and resources. To address this, semi-supervised learning (SSL) methods, which use a few labeled datasets and large-scale unlabeled datasets, have been introduced. These approaches aim to reduce annotating time and resource consumption while improving performance. The most commonly used techniques include consistency learning through data augmentations [ 42 , 4 ] and the use of pseudo labels [ 21 , 12 ] . Consistency learning focuses on training the model to make consistent predictions about augmented images, while pseudo-labeling uses the model’s predictions on unlabeled data as pseudo-labels. Additionally, various studies use consistency learning and pseudo-labeling together to reduce bias from pseudo-label via a thresholding approach. For example, FixMatch [ 38 ] uses fixed predefined thresholds, FlexMatch [ 54 ] employs flexible thresholds, and FreeMatch [ 46 ] uses thresholds that are not predefined but are also flexible. 2.2 Semi-supervised Video Recognition As the creation and utilization of video data increase across various platforms, research on video understanding is becoming increasingly active [ 44 , 23 , 24 , 26 , 11 ] . Among them, video recognition [ 30 , 5 , 25 , 28 , 8 , 1 , 22 , 36 , 56 ] is one of the challenging tasks. Similar to image classification, video recognition requires large amounts of annotated data. However, videos have more complex information than images, which needs significant annotation time and resources. To address this challenge, various studies have been conducted to extend semi-supervised learning techniques used in image classification to video recognition [ 51 , 2 , 43 , 47 ] . VideoSSL [ 18 ] extends the existing semi-supervised image classification methods to the video modality. In addition, MvPL [ 50 ] has introduced a multi-modal

Read the complete text