Reducing Inference Energy Consumption Using Dual Complementary CNNs Journal: Future Generation Computer Systems

Michail Kinnas, John Violos, Ioannis Kompatsiaris, Symeon Papadopoulos · 2024

Read Reducing Inference Energy Consumption Using Dual Complementary CNNs Journal: Future Generation Computer Systems by Michail Kinnas, John Violos, Ioannis Kompatsiaris, Symeon Papadopoulos (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01039.txt.

Introduction

Running vision networks drains batteries on small devices, and shrinking models usually costs accuracy. Michail Kinnas and co-authors propose a paired design in 2024: two small networks that cover each other's weak spots. The first network answers when it is confident; the second is called only when the first is unsure.

A memory also stores answers for inputs seen before, skipping both networks on repeats. On a small NVIDIA board, the setup cut energy use by up to about eighty-six percent on a test set where each image appeared twice, while holding accuracy high. The authors present the pairing plus memory as a practical route to on-device vision that respects tight power budgets.

Catalog record

CREATORMichail Kinnas, John Violos, Ioannis Kompatsiaris, Symeon Papadopoulos
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01039

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reducing Inference Energy Consumption Using Dual Complementary CNNs Journal: Future Generation Computer Systems Michail Kinnas, John Violos, Ioannis Kompatsiaris, Symeon Papadopoulos Abstract Energy efficiency of Convolutional Neural Networks (CNNs) has become an important area of research, with various strategies being developed to minimize the power consumption of these models. Previous efforts, including techniques like model pruning, quantization, and hardware optimization, have made significant strides in this direction. However, there remains a need for more effective on device AI solutions that balance energy efficiency with model performance. In this paper, we propose a novel approach to reduce the energy requirements of inference of CNNs. Our methodology employs two small Complementary CNNs that collaborate with each other by covering each other’s "weaknesses" in predictions. If the confidence for a prediction of the first CNN is considered low, the second CNN is invoked with the aim of producing a higher confidence prediction. This dual-CNN setup significantly reduces energy consumption compared to using a single large deep CNN. Additionally, we propose a memory component that retains previous classifications for identical inputs, bypassing the need to re-invoke the CNNs for the same input, further saving energy. Our experiments on a Jetson Nano computer demonstrate an energy reduction of up to 85.8% achieved on modified datasets where each sample was duplicated once. These findings indicate that leveraging a complementary CNN pair along with a memory component effectively reduces inference energy while maintaining high accuracy. † † graphicalabstract: † † highlights: Introduced the concept of complementarity in Neural Networks. Proposed an on-device AI methodology based on the synergy of two small, complementary CNNs. This method is optimized for Edge devices with limited computational resources. For the collaboration of two small CNNs a confidence-based mechanism used. With this mechanism the system dynamically selects the most accurate predictions from the paired CNNs, enhancing the efficiency of on-device inference. Conducted experimental evaluations using four datasets on a Jetson Nano device, with power consumption monitored via a power meter. Demonstrated that the collaboration between the two smaller CNNs significantly reduces energy consumption and response time, while maintaining accuracy levels comparable to larger, state-of-the-art CNN models. 1 Introduction Energy efficiency is essential for edge devices in resource-constrained smart environments, where reducing energy consumption is critical for extending device life and lowering maintenance costs [ 1 ] . Although techniques like network condition estimation and transmission power adjustment have been explored to improve energy efficiency at the network edge, optimizing deep neural network (DNN) models for on-device AI applications remains underexplored [ 2 ] . DNNs have proven to be highly effective for solving complex problems in smart environments, leveraging advancements from various fields, such as computer vision [ 3 ] . As Internet of Things (IoT), smart homes, and edge computing devices become more widespread, there is an increasing need to run DNN models directly on edge devices to enable real-time processing and minimize latency [ 4 ] . However, deploying large DNNs on these devices presents significant challenges due to their limitations, such as low-power processors and limited memory, making it difficult to manage intensive computations and large storage needs [ 5 ] . Given that these devices typically operate on batteries, the high energy demands of DNNs can rapidly drain their power, reducing their operational longevity. To address these limitations, one strategy is to compress DNN models into smaller ones. Smaller models, with fewer parameters and layers, require less computational power and energy, making them more suitable for resource-constrained devices. However, their limited capacity can restrict their ability to capture complex patterns and nuances in data, potentially leading to lower accuracy on challenging tasks [ 6 ] . In contrast, large DNN models, with more parameters and layers, have a greater capacity to learn intricate features and relationships, often resulting in superior performance on complex problems. This increased complexity, however, demands substantial computational resources and memory [ 7 ] , posing a significant challenge for deployment on edge devices. Efforts to mitigate the computational and memory burdens of DNNs have led to numerous works in the area of DNN compression [ 8 ] . These aim to reduce the model’s computational and memory demands without compromising its performance. However, in practice, these efforts typically involve a trade-off between reducing model size and computational resources against the potential loss in model performance and predictive accuracy [ 9 ] . A wide range of smart environment applications, from homes to cities, are driven by advancements in the field of computer vision [ 10 ] . In computer vision, several methods have been developed to build efficient CNNs [ 11 ] . Quantization reduces the precision of weight values to conserve memory. Pruning involves the removal of redundant or insignificant neurons to decrease the computational load [ 12 ] . Convolutional filter compression and matrix factorization reduce model size by optimizing the structure of the network. Network Architecture Search (NAS) is another approach that finds DNNs optimized for individual devices, ensuring guaranteed performance. Additionally, knowledge distillation involves training a smaller student neural network to mimic the behavior of a larger teacher model, with the goal of maintaining performance while reducing model size [ 13 ] . All these approaches have several limitations, including a loss of accuracy in the compressed model, inefficient capturing of the teacher model’s knowledge, and often considerable computational requirements, increased complexity and time needed for training, fine-tuning, and knowledge distillation [ 14 ] . Our goal is to implement a CNN architecture that achieves the performance of a large deep CNN while maintaining the energy efficiency of smaller, more compact models. We explore deploying two small CNN models on resource-constrained devices in order to attain higher predictive accuracy than each model could achieve individually. To this end, we investigate the concept of complementarity. Given a pair of small CNN models trained on the same dataset, complementarity is present if each model correctly predicts distinct subsets of the dataset. This suggests that each model has learned different aspects of the dataset. By leveraging the correct predictions from each model and discarding the incorrect ones, we aim to maximize the coverage of accurate dataset predictions. To evaluate the validity of a prediction, we employed the confidence score based on the output logits of the model’s prediction. This provides a reasonably accurate estimate of prediction validity during inference, while having the least computational and energy requirements than other more advanced methods [ 15 ] . By utilizing this confidence score for both deployed models, we select the prediction that is deemed more accurate. To further reduce energy consumption, we investigate the possibility of bypassing classification by invoking the CNNs only if the same input was not previously encountered. We focus on perceptual hashing [ 16 ] , a technique that generates a hash value from an image’s visual content, enabling similar images to have similar hash values. This method facilitates rapid identification and comparison of images. By storing the hash and classification for each previous prediction, we are able to avoid unnecessary CNN inferences. While our experiments are conducted on CNN architectures, we also refer to DNNs throughout the manuscript because CNNs are a specific subcategory of DNNs. Our contributions include the

Read the complete text