EdgeOAR: Real-time Online Action Recognition On Edge Devices

Wei Luo, Deyu Zhang, Ying Tang, Fan Wu, Yaoxue Zhang · 2024

Read EdgeOAR: Real-time Online Action Recognition On Edge Devices by Wei Luo, Deyu Zhang, Ying Tang, Fan Wu, Yaoxue Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01267.txt.

Introduction

Wei Luo and co-authors built EdgeOAR in 2024, a system that names human actions in live video on small edge devices. Online action recognition must judge each moment at once, with almost no delay. Older methods wait for a full clip, so they react too late for live feedback.

EdgeOAR adds a light feature module that sharpens both time and space cues from the first frames of a stream. A training loop teaches that module to learn from the start of each video. A fusion step then weighs entropy and agreement between cues to choose when to exit early. On a standard action data set the method cut delay and energy use by more than ninety-nine percent against a top rival, while keeping fair accuracy.

Catalog record

CREATORWei Luo, Deyu Zhang, Ying Tang, Fan Wu, Yaoxue Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EdgeOAR: Real-time Online Action Recognition On Edge Devices Wei Luo, Deyu Zhang, Ying Tang, Fan Wu, Yaoxue Zhang Abstract This paper addresses the challenges of Online Action Recognition (OAR), a framework that involves instantaneous analysis and classification of behaviors in video streams. OAR must operate under stringent latency constraints, making it an indispensable component for real-time feedback for edge computing. Existing methods, which typically rely on the processing of entire video clips, fall short in scenarios requiring immediate recognition. To address this, we designed EdgeOAR, a novel framework specifically designed for OAR on edge devices. EdgeOAR includes the Early Exit-oriented Task-specific Feature Enhancement Module (TFEM), which comprises lightweight submodules to optimize features in both temporal and spatial dimensions. We design an iterative training method to enable TFEM learning features from the beginning of the video. Additionally, EdgeOAR includes an Inverse Information Entropy (IIE) and Modality Consistency (MC)-driven fusion module to fuse features and make better exit decisions. This design overcomes the two main challenges: robust modeling of spatio-temporal action representations with limited initial frames in online video streams and balancing accuracy and efficiency on resource-constrained edge devices. Experiments show that on the UCF-101 dataset, our method EdgeOAR reduces latency by 99.23% and energy consumption by 99.28% compared to state-of-the-art (SOTA) method. And achieves an adequate accuracy on edge devices. I Introduction Action recognition aims to recognize human activities such as push-ups, running, and walking within a video segment. [ 1 , 2 ] This capability serves as a key component for numerous edge applications. For end-users, particularly in the context of Online Action Recognition (OAR), it is crucial to deliver action recognition results from video segments captured by smartphones as quickly as possible. This meets the needs of downstream tasks, especially in scenarios such as emergency response in camera surveillance, real-time feedback in fitness applications, and providing immersive experiences in mobile games. This rapid processing is essential because delays can significantly impact user experience and the effectiveness of the application. Unfortunately, we identify a significant gap between existing practices and the goal of achieving optimal responsiveness. The current solution typically requires the complete video segments as a prior. Most action recognition tasks typically involve uniform sampling across the entire video for action modeling [ 3 , 4 , 5 , 6 , 7 ] . Some works improve the sampling process by selecting key frames to enhance recognition accuracy [ 8 , 9 ] , which still require a global perspective. Other works attempt early exit strategies, but limited to frames sampled from the entire video [ 10 , 11 ] . In an online scenario, the latency perceived under the existing solutions is the video segment length plus the duration required for video analysis. Typical video segment lengths are a few seconds, leading to inference latency of more than several seconds, which is an unacceptable delay for edge users. Even taking advantage of stronger computing facilities such as cloud servers, despite potential privacy concerns, does not bridge this gap due to the reliance on processing complete video segments. The root cause of this issue is that existing action recognition frameworks fail to fully exploit the predictive and recognition capabilities of the model. In contrast, human intelligence seamlessly integrates both abilities, enabling the determination of action types from observing only the initial segments of a video sequence, without necessitating the processing of the entire sequence. Inspired by human perception, in this paper, we propose the EdgeOAR framework. During the inference phase, EdgeOAR acquires action information from the initial frames of a video. Once the reliability threshold for action recognition is reached, our framework can directly deliver the recognition result. EdgeOAR ensures accuracy in edge devices, significantly reducing latency and energy consumption for action recognition tasks. Inferring action recognition results accurately from only the beginning of videos presents two new challenges. First, how to robustly model spatial-temporal action representation from online video streams. In online video streams, the lack of future information means that sufficient information to represent actions can only be captured in a limited number of initial frames. Given the diversity and complexity of actions, enhancing the model’s modeling capabilities while maintaining a certain level of generalization challenges the model’s capacity for accurate representation. Second, how to strike the balance between accuracy and efficiency on resource-limited edge devices. Deploying deep learning models on edge devices encounters resource constraints, particularly in computing power and energy consumption. Users expect highly responsive and accurate feedback in online applications. Using complex models to achieve powerful feature extraction can result in excessively long latency per frame, possibly even failing to complete the analysis before the video segment finishes playing, which is counterproductive. Achieving a balance between accuracy and efficiency on resource-limited edge devices presents another significant challenge. To address these challenges, we propose EdgeOAR, a general framework that enables real-time OAR on edge devices. Specifically, we have designed the Early Exit-oriented Task-specific Feature Enhancement Module (TFEM) which consists of two lightweight submodules: the Temporal Layering Shift Module (TLSM) and the MBs-guided Spatial Enhancement Module (MSEM). These modules optimize feature representations in the temporal and spatial dimensions, respectively, to enhance the overall performance of TFEM. To enhance its performance, we have developed an iterative training method that helps TFEM learn features from the beginning of the video. Additionally, we have designed the Inverse Information Entropy (IIE) and Modality Consistency (MC)-driven Fusion Module to fuse features from different modalities in both time and space, further improving accuracy and aiding the model in making better exiting decisions. Our design is edge-friendly and does not introduce excessive computational overhead. In summary, the main contributions are as follows: • We design EdgeOAR, a first online action recognition framework, implemented on Android with nearly 3,000 lines of Java [ 12 ] code (excluding test code). • Considering the limited computing power and resources of edge devices, we design a two-stage online framework and develop an iterative training approach to enable the backbone to gradually learn the features from the early part of the video. We then incorporate prior information to adaptively fuse inference results from different modalities and automatically exit based on a threshold. • To the best of our knowledge, EdgeOAR is the first online action recognition framework specifically designed for edge devices. Experiments show that on the two public datasets, EdgeOAR reduces latency by an average of 97.24%, power consumption by 26.56% and decreases accuracy by 3.26% compared to offline methods. II Background and Motivation Action recognition demands not only analyzing the content within each frame but also understanding the temporal relationships between frames to identify the overall pattern of the action. We first briefly introduced the encoding process of H.265 videos on edge devices and the relationship between the information contained in the videos and the motion content captured. Then, we outlined the motivation for early exit in action recognition and the feasibility of using multi-modal data to enhance action recognition accuracy. II-A Background: In the task of action recognition, motion representation, such as optical flow, serves as an important supplementary input. It provides motion cues at the pixel level, but is computationally complex. Fortunately, edge devices are equipped with dedicated codec chips that can compress image

Read the complete text