RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone

Mustafa Munir, Md Mostafijur Rahman, Radu Marculescu · 2024

Read RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone by Mustafa Munir, Md Mostafijur Rahman, Radu Marculescu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10995.txt.

Introduction

Mustafa Munir, Md Mostafijur Rahman, and Radu Marculescu design a fast vision backbone for phones in this 2024 paper. Transformer models lead image tasks but run heavy for mobile chips, and hybrid designs still trail pure convolutional networks in speed. The team bets on a pure convolutional design built from a new multi-level dilated block.

Dilated convolutions spread their view wider than standard ones, and stacking several dilation levels lets near and far features interact. The resulting RapidNet family beat leading mobile rivals on classification, detection, and two kinds of segmentation, in accuracy, speed, or both. The smallest model topped seventy-six percent on a standard test while answering in under a millisecond on a phone chip. The authors say careful pure design still wins on mobile.

Catalog record

CREATORMustafa Munir, Md Mostafijur Rahman, Radu Marculescu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone Mustafa Munir, Md Mostafijur Rahman, Radu Marculescu Abstract Vision transformers (ViTs) have dominated computer vision in recent years. However, ViTs are computationally expensive and not well suited for mobile devices; this led to the prevalence of convolutional neural network (CNN) and ViT-based hybrid models for mobile vision applications. Recently, Vision GNN (ViG) and CNN hybrid models have also been proposed for mobile vision tasks. However, all of these methods remain slower compared to pure CNN-based models. In this work, we propose Multi-Level Dilated Convolutions to devise a purely CNN-based mobile backbone. Using Multi-Level Dilated Convolutions allows for a larger theoretical receptive field than standard convolutions. Different levels of dilation also allow for interactions between the short-range and long-range features in an image. Experiments show that our proposed model outperforms state-of-the-art (SOTA) mobile CNN, ViT, ViG, and hybrid architectures in terms of accuracy and/or speed on image classification, object detection, instance segmentation, and semantic segmentation. Our fastest model, RapidNet-Ti, achieves 76.3% top-1 accuracy on ImageNet-1K with 0.9 ms inference latency on an iPhone 13 mini NPU, which is faster and more accurate than MobileNetV2x1.4 (74.7% top-1 with 1.0 ms latency). Our work shows that pure CNN architectures can beat SOTA hybrid and ViT models in terms of accuracy and speed when designed properly 1 . ${}^{1}$ ${}^{1}$ footnotetext: Code: https://github.com/mmunir127/RapidNet-Official 1 Introduction The field of deep learning has witnessed remarkable advancements in computer vision in the last decade [ 48 ] , from image classification and object detection to generative vision tasks, such as image synthesis [ 26 , 2 , 10 ] and video synthesis [ 14 , 37 ] using generative adversarial networks (GANs) [ 15 ] and diffusion models [ 21 ] . This evolution has been fueled by diverse architectural paradigms, including Convolutional Neural Networks (CNNs) [ 31 , 19 , 24 , 25 , 39 ] , Vision Transformers (ViTs) [ 13 , 38 , 3 ] , and Multi-Layer Perceptron (MLP)-based [ 56 , 55 ] models. CNNs and MLPs interpret images as pixel grids, while ViTs represent them as sequences of patches [ 13 ] , enabling them to be processed by transformers [ 59 ] . ViTs also have global receptive fields and capture distant interactions within images, unlike CNNs which have local receptive fields [ 13 ] . The emergence of Vision Graph Neural Networks (ViGs), exemplified by models like ViG [ 16 ] , ViHGNN [ 17 ] , and MobileViG [ 46 , 1 ] , introduced graph-based approaches, which connect image patches through graph structures. While ViG-based models demonstrate their potential in capturing global object interactions, they incur large computational costs due to graph construction [ 47 ] . The demand for deploying powerful AI applications directly on mobile devices has led to the exploration of lightweight models [ 24 , 52 , 5 , 65 ] . Early efforts with CNNs on mobile platforms paved the way for hybrid CNN-ViT architectures, but the computational cost of the self-attention operation in ViTs poses significant challenges for mobile applications [ 43 , 44 ] . The exploration of ViTs, ViGs, and hybrid architectures for mobile devices has led to many advances in accuracy, but state-of-the-art (SOTA) results can still be achieved using only CNN-based models. One avenue to make CNN-based models competitive is dilated convolutions [ 64 ] . Dilated convolutions can increase the receptive field of a convolution operation, but with a lower cost than increasing the kernel size. This is because a dilated convolution effectively increases the kernel’s receptive field by inserting ”gaps” in between elements of the kernel by a dilation factor [ 64 ] . This means that a 3 \times 3 convolution with a dilation factor of 2 will have a receptive field equal to a 5 \times 5 convolution while using less parameters. Thus, we can use dilated convolutions for increasing the receptive field in our network, similar to how hybrid CNN-ViT and CNN-ViG architectures use ViTs and ViGs to increase their receptive field. Figure 1 : Comparison of accuracy on ImageNet-1K, Average Precision (AP) on MS COCO, and mean Intersection over Union (mIoU) on ADE20K . a) RapidNet achieves the best accuracy-GMACs tradeoff on all model sizes compared. b) RapidNet achieves the best AP-GMACs tradeoff on all model sizes compared. c) RapidNet achieves the best mIoU-GMACs tradeoff on all model sizes compared. GMACs are computed using a resolution of 224 \times 224 for a) and a resolution of 512 \times 512 for b) and c). In this work, we propose Multi-Level Dilated Convolution blocks to create a CNN-based architecture competitive with SOTA CNN, ViT, ViG, and hybrid models. Our newly introduced architecture, RapidNet, is faster, less computationally expensive in terms of GMACS, and/or more accurate compared to other mobile architectures as shown in Figure 1 . Indeed, our experimental results show that our proposed RapidNet architecture outperforms competing SOTA models across all model sizes for the tasks of image classification, object detection, and semantic segmentation. We summarize our contributions as follows: 1. We propose using Multi-Level Dilated Convolutions (MLDC) to enable processing features at different levels of dilation in parallel. MLDC expands the receptive field of convolutions, thus allowing for an efficient CNN-based alternative to ViG and ViT-based models. 2. We propose a novel efficient CNN-based architecture, RapidNet, which uses MLDC, reparameterizable large kernel depthwise convolutions [ 6 , 58 ] , and a large kernel feedforward network (FFN) [ 58 ] . 3. We conduct comprehensive experiments to demonstrate the effectiveness of the RapidNet architecture, which beats the existing efficient ViG, CNN, and ViT architectures in terms of top-1 accuracy, GMACs, and/or latency on ImageNet-1k [ 9 ] image classification, COCO [ 36 ] object detection, COCO [ 36 ] instance segmentation, and ADE20K [ 70 ] semantic segmentation. Specifically, our RapidNet-M model achieves a top-1 accuracy of 81.0% on ImageNet classification, 42.0 Average Precision (AP) on COCO object detection, and 41.5 mean Intersection over Union (mIoU) on ADE20K semantic segmentation. The paper is organized as follows. Section 2 covers related work on dilated convolutions and efficient computer vision. Section 3 describes Multi-Level Dilated Convolutions, the usage of a large kernel FFN, our RapidNet architecture, and the network configuration of RapidNet for different model sizes. Section 4 describes our experimental setup and results for ImageNet-1k image classification, COCO object detection, COCO instance segmentation, and ADE20K semantic segmentation. Lastly, Section 5 summarizes our main contributions. 2 Related Work In this section, we review dilated convolutions and previous work in the mobile computer vision space. 2.1 Dilated Convolution Dilated convolutions [ 23 ] introduce ”gaps” between kernel elements by a dilation factor. The ”gaps” are introduced by inserting zeros between each pixel in the convolutional kernel [ 64 ] , thus allowing for an expanded receptive field without increasing the number of parameters [ 64 , 45 ] . Figure 2 : Comparison of regular and dilated convolution. a) In a regular convolution with a kernel size of 3 in a 7 \times 7 image, we can see the convolution is applied to the 3 \times 3 patches in the grid. b) In a dilated convolution with

Read the complete text