Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment

Jiaqi Wu, Simin Chen, Zehua Wang, Wei Chen, Zijian Tian, F. Richard Yu, Fellow IEEE, Victor C. M. Leung, life Fellow IEEE · 2024

Read Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment by Jiaqi Wu, Simin Chen, Zehua Wang, Wei Chen, Zijian Tian, F. Richard Yu, Fellow IEEE, Victor C. M. Leung, life Fellow IEEE (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00838.txt.

Description

Abstract As the volume of collected video data continues to grow, data-oriented cloud computing in Internet of Video Things (IoVT) systems faces increasing challenges, such as system latency and bandwidth pressure. Task-oriented edge computing, by shifting data analysis to the edge, enables real-time visual inference.

Catalog record

CREATORJiaqi Wu, Simin Chen, Zehua Wang, Wei Chen, Zijian Tian, F. Richard Yu, Fellow IEEE, Victor C. M. Leung, life Fellow IEEE
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment Jiaqi Wu, Simin Chen, Zehua Wang, Wei Chen, Zijian Tian, F. Richard Yu, Fellow IEEE, Victor C. M. Leung, life Fellow IEEE Abstract As the volume of collected video data continues to grow, data-oriented cloud computing in Internet of Video Things (IoVT) systems faces increasing challenges, such as system latency and bandwidth pressure. Task-oriented edge computing, by shifting data analysis to the edge, enables real-time visual inference. However, the limited computational power of edge devices presents challenges for effectively executing computationally intensive visual tasks. Existing methods struggle with the trade-off between high model performance and low resource consumption. For instance, lightweight neural networks often deliver limited performance, while Neural Architecture Search (NAS)-based model design incurs high computational and commercial costs at training Inspired by hardware/software co-design principles, we propose, for the first time, a co-design framework at both the model and system levels to optimize neural network architecture and deployment strategies during inference. This framework maximizes the computational efficiency of edge devices to achieve high throughput, enabling real-time edge inference. Specifically, we implement a dynamic model structure based on the re-parameterization principle, coupled with a Roofline-based model partitioning strategy to synergistically enhance the computational performance of heterogeneous devices. Furthermore, we employ a multi-objective co-optimization approach for the above strategies to balance throughput and accuracy. We also conduct a thorough analysis of the mathematical consistency between the partitioned and original models, in addition to deriving the convergence of the partitioned model. Experimental results show that, compared to baseline algorithms, our method significantly improves throughput (12.05% on MNIST, 18.83% on ImageNet) while also achieving higher classification accuracy. Additionally, it ensures stable performance across devices with varying computational capacities, highlighting its generalizability and practical application value. Simulated experiments further validate that our method enables high-accuracy real-time detection in IoVT systems, particularly for small object detection. I Introduction Real-time visual task inference is a core function of the Internet of Visual Things (IoVT) systems [ 22 , 23 ] , particularly for time-sensitive tasks such as object detection [ 25 ] and object tracking [ 26 ] . Real-time inference can effectively prevent task failures caused by latency. Currently, many IoVT systems rely on a centralized cloud computing architecture, as shown in Fig. 1 , where data collected by terminal sensing devices is transmitted to cloud servers for centralized inference. However, as the number of terminal devices increases and the volume of visual data grows exponentially, the long-distance transmission of large-scale data creates immense bandwidth pressure, severely compromising the system’s real-time performance [ 24 , 27 ] . Thus, this centralized cloud computing design principle is not well-suited for machine vision applications. Edge computing [ 28 ] , as an emerging computational paradigm, as illustrated in Fig. 1 , shifts data processing and analysis tasks closer to the data source, avoiding the need for massive data transmission, and thus holds greater potential for enhancing real-time system performance. [ 29 ] For example, studies [ 27 ] and [ 30 ] deploy lightweight object detection models on edge devices (e.g., drones) to achieve real-time power inspection, while study [ 31 ] deploys deep learning models on edge servers to avoid the latency issues associated with transmitting visual data to the cloud. However, a key challenge is limited computational resources of each edge device, which are insufficient to meet high-performance demands of complex visual tasks [ 32 ] . Although the computational power of edge devices has improved, it still lags significantly behind cloud servers, making it difficult to fully satisfy the computational requirements of visual inference tasks [ 33 , 34 ] . Despite neural network edge deployment solutions being proposed to address this issue [ 40 , 39 ] , there remain significant limitations in achieving real-time inference, where balancing performance and throughput proves challenging [ 38 ] . To explore this, we introduce several key edge deployment approaches and analyze their drawbacks: • General lightweight models (e.g., the Mobilenet family [ 42 ] and GhostNet [ 43 ] ): These models focus on lightweight design to reduce computational complexity, making them suitable for embedded devices with limited resources [ 41 ] . However, despite reducing computational load, their feature representation capabilities are limited, resulting in suboptimal performance for high-accuracy visual inference tasks . • Neural Architecture Search (NAS)-based methods : These methods employ automated network structure search strategies to design models optimized for specific edge devices [ 46 , 45 ] . While theoretically capable of customizing models for device-specific deployment, the process incurs significant computational and financial costs [ 44 ] . Moreover, NAS-generated models are typically device-specific, lacking generalizability across devices, which severely limits their practical application value. • Split learning approaches : Split learning methods [ 35 ] partition large-scale models across different devices for distributed sequential inference. Studies [ 27 ] and [ 37 ] introduce a split learning strategy for model distribution deployment in industrial IoVT systems to achieve real-time visual inference. Study [ 11 ] proposes a model partitioning deployment method based on roofline analysis to enhance inference throughput. However, relying solely on the model partitioning strategy still has limited effectiveness in maximizing the device’s computational performance to improve throughput. Fig. 1 : The comparison between centralized cloud computing and edge computing approach in the IoVT system. Centralized cloud computing is a typical data-driven design, requiring all data to be uploaded to a central server for inference. In contrast, edge computing is a task-oriented paradigm, where data analysis tasks are offloaded from the cloud to the edge, closer to the data collection points, in order to reduce system latency. To address these issues, we propose, for the first time, a co-design method for neural network architecture and edge deployment tailored for real-time visual inference in IoVT systems. This approach performs multi-objective joint optimization of dynamic model architecture design and deployment strategies to enhance the compatibility between the model and edge devices, maximizing computational power and improving throughput while ensuring high accuracy. At the system level, we adopt a Roofline-based [ 56 ] model partitioning strategy that determines the optimal partitioning scheme based on the peak computational performance and memory bandwidth of the hardware, fully leveraging their computational resources to boost throughput. At the model level, we employ a dynamic architecture based on a reparameterization strategy [ 57 ] , which adaptively adjusts the neural network during inference to reduce computational complexity while maintaining model performance, further optimizing throughput. To simultaneously enhance throughput and maintain model accuracy, we utilize a multi-objective co-optimization approach that coordinates both model architecture and deployment strategies. Unlike traditional optimization methods, our approach dynamically adjusts the pre-trained model during inference, enabling it to adapt to different devices with greater generalizability, making it more suitable for IoVT systems where frequent dynamic device participation occurs. Experimental results demonstrate that, compared to six state-of-the-art (SOTA) methods, the proposed approach significantly improves throughput by an average of 12.05% on MNIST, 13.75% on CIFAR-100, and 18.83% on ImageNet, while achieving satisfactory classification accuracy. Additionally, it achieves excellent throughput and accuracy across different devices, highlighting its generalizability. Furthermore, we evaluated the practical application of

Read the complete text