JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts

Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, Jun Won Choi · 2024

Read JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts by Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, Jun Won Choi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13708.txt.

Introduction

Finding who does what in a video, and where, is harder than naming the action alone. Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, and Jun Won Choi built a detector in 2024 that listens and reads scene descriptions as well as watching.

Their system, JoVALE, is the first in its class to fuse sound, images, and captions from a large image-captioning model around each actor. A transformer network built for the purpose weighs how actors and their mixed context interact from frame to frame. On three standard benchmarks the added senses lifted accuracy to new records. The authors argue that an actor's surroundings, in words and sound, often decide what the action is.

Catalog record

CREATORTaein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, Jun Won Choi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, Jun Won Choi Abstract Video Action Detection (VAD) entails localizing and categorizing action instances within videos, which inherently consist of diverse information sources such as audio, visual cues, and surrounding scene contexts. Leveraging this multi-modal information effectively for VAD poses a significant challenge, as the model must identify action-relevant cues with precision. In this study, we introduce a novel multi-modal VAD architecture, referred to as the Joint Actor-centric Visual, Audio, Language Encoder (JoVALE). JoVALE is the first VAD method to integrate audio and visual features with scene descriptive context sourced from large-capacity image captioning models. At the heart of JoVALE is the actor-centric aggregation of audio, visual, and scene descriptive information, enabling adaptive integration of crucial features for recognizing each actor’s actions. We have developed a Transformer-based architecture, the Actor-centric Multi-modal Fusion Network, specifically designed to capture the dynamic interactions among actors and their multi-modal contexts. Our evaluation on three prominent VAD benchmarks—AVA, UCF101-24, and JHMDB51-21—demonstrates that incorporating multi-modal information significantly enhances performance, setting new state-of-the-art performances in the field. Code — https://github.com/taeiin/AAAI2025-JoVALE Introduction Video action detection (VAD) is a challenging task that aims to localize and classify human actions within video sequences. VAD generates bounding boxes with action scores for a keyframe by analyzing the sequential frames around the keyframe. This task differs from Action Recognition task, which classifies the action for a given video clip, and from Temporal Action Detection task, which identifies the intervals of particular actions within a video clip. Humans rely on various sources of information to detect actions, including visual appearance, motion sequences, actor postures, and interactions with their environment. Numerous studies have demonstrated that leveraging such multi-modal information can significantly enhance action recognition performance ( Kazakos et al. 2019 ; Gao et al. 2020 ; Xiao et al. 2020 ; Nagrani et al. 2021 ) . Audio, in particular, offers valuable information, providing both direct and indirect contextual cues for action recognition. For example, sounds directly linked to actions, like speech, gunshots, or music, can help identify corresponding actions. Additionally, environmental sounds can indirectly suggest relevant actions, such as the sound of waves indicating beach-related activities. Therefore, incorporating audio data alongside visual data can improve the performance and robustness of VAD. Several action recognition methods have successfully utilized both audio and visual data ( Gao et al. 2020 ; Xiao et al. 2020 ; Nagrani et al. 2021 ) . While multi-modal information has shown promise for action recognition tasks, its application in VAD presents significant challenges. Action instances in videos are dispersed across both temporal and spatial dimensions, and the contextual cues necessary for their detection are similarly spread throughout the video. It is crucial to accurately link these actions and contextual features to ensure robust VAD performance. For example, the sound of a piano might help identify a ‘playing piano’ action but would be irrelevant for detecting a ‘talking with others’ action within the same scene. Therefore, the piano sound should be selectively used to detect the ‘playing piano’ action and not the ‘talking with others’ action. Moreover, effectively integrating multi-modal information from various sources is another key to enhancing VAD performance. Despite its potential, the use of audio-visual information for VAD is still relatively unexplored in current research. Figure 1: Overview of JoVALE: (top-left) The proposed JoVALE integrates audio, visual, and scene-descriptive features using an AMFN. (bottom-left) JoVALE leverages a VLP model fine-tuned on an image captioning task to generate scene-descriptive features. (right) AMFN encodes high-level interactions between multi-modal features through MFE and MFA. Another valuable resource for VAD identified in this study is the prior general scene-descriptive knowledge gained through vision-language foundation models. Vision Language Pre-training (VLP) models have shown substantial success by leveraging extensive multi-modal data sourced from the web, public databases, and various corpora. These models excel in capturing complex relational structures between text and images, enabling them to adapt to a variety of downstream tasks in either a zero-shot or one-shot manner. With their ability to understand images, the features derived from VLP can significantly enhance VAD performances. In this research, we further investigate a VAD approach that capitalizes on the rich language context provided by VLP models. This paper introduces a novel multi-modal VAD approach referred to as the Joint Actor-centric Visual, Audio, Language Encoder (JoVALE). JoVALE is the first method to leverage audio and visual modalities alongside language context to localize and classify actions in videos. At the core of JoVALE is the actor-centric modeling of multi-modal contextual information. The key concepts of JoVALE are illustrated in Fig. 1 . JoVALE begins by generating densely sampled actor proposal features using an off-the-shelf person detector. These actor proposal features are then processed by the Actor-centric Multi-modal Fusion Network (AMFN), which aggregates relevant contextual information from both audio and visual features. Furthermore, AMFN integrates scene-descriptive knowledge acquired from the VLP model, BLIP ( Li et al. 2022a ) , to enrich the action representation. To fully leverage multi-modal information for VAD, JoVALE effectively models the relationships among actors, temporal dynamics, and various modalities through AMFN. The AMFN captures their complex interactions through successive updates of Action Embeddings across multiple Transformer layers. It comprises two main components: Multi-modal Feature Encoding (MFE) and Multi-modal Feature Aggregation (MFA). The MFE module jointly encodes Action Embeddings and Multi-modal Context Embeddings for each modality, achieving computational efficiency through the use of Temporal Bottleneck Features. These Temporal Bottleneck Features provide a compact representation of the temporal changes across all actors. Following this, the MFA module aggregates the Action Embeddings from each modality in a weighted fashion. The resulting features are fed into the subsequent Transformer layer for final action detection. We evaluated JoVALE on three popular VAD benchmarks: AVA ( Gu et al. 2018 ) , UCF101-24 ( Soomro, Zamir, and Shah 2012 ) , and JHMDB51-21 ( Jhuang et al. 2013 ) . By effectively combining audio, visual, and scene-descriptive context information, JoVALE significantly outperforms the baseline on these benchmarks. On the challenging AVA dataset, JoVALE records a mean Average Precision (mAP) of 40.1%, achieving a substantial improvement of 2.4% over the previous best method, EVAD ( Chen et al. 2023 ) . Our contributions can be summarized as follows: • We present a simple yet effective multi-modal VAD architecture that utilizes the audio-visual information present in videos. Our main approach is Actor-Centric Feature Aggregation, which adaptively attends to the multi-modal context essential for detecting each action instance. There are only a few studies that have explored the use of audio-visual context for VAD. • We are the first to introduce a VAD approach that incorporates general scene-descriptive knowledge inferred from a Vision Language Foundation model. • We propose an efficient architecture that effectively models complex relationships among actors, temporal dynamics, and modalities. Our modeling approach differs from existing VAD methods, which typically combine semantic actor features or predicted scores from each modality in a straightforward manner. Related Work Video Action Detection Various VAD methods have been proposed, which can be broadly classified into two main approaches: end-to-end and two-stage methods. End-to-end methods predict both the action location and class

Read the complete text