CNNtention: Can CNNs do better with Attention?

Nikhil Kapila, Julian Glattki, Tejas Rathi · 2024

Read CNNtention: Can CNNs do better with Attention? by Nikhil Kapila, Julian Glattki, Tejas Rathi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11657.txt.

Introduction

Nikhil Kapila and co-authors ask whether adding attention helps convolutional neural networks sort images, in a 2024 project paper. Such networks have long been the standard tool for image classification. They pull out local features well, but they can miss the wider scene. Attention-based designs try to capture that global context.

The project trains plain networks and attention-augmented networks on the same image task. It then compares their accuracy, speed, and cost. The trade-offs are laid out in plain terms for each design. Readers get a clear sense of when attention earns its extra cost. The work began as a final course project in deep learning.

Catalog record

CREATORNikhil Kapila, Julian Glattki, Tejas Rathi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11657

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CNNtention: Can CNNs do better with Attention? Nikhil Kapila, Julian Glattki, Tejas Rathi Abstract Convolutional Neural Networks (CNNs) have been the standard for image classification tasks for a long time, but more recently attention-based mechanisms have gained traction. This project aims to compare traditional CNNs with attention-augmented CNNs across an image classification task. By evaluating and comparing their performance, accuracy and computational efficiency, the project will highlight benefits and trade-off of the localized feature extraction of traditional CNNs and the global context capture in attention-augmented CNNs. By doing this, we can reveal further insights into their respective strengths and weaknesses, guide the selection of models based on specific application needs and ultimately, enhance understanding of these architectures in the deep learning community. 1 Introduction Convolutional Neural Networks (CNNs) have been the state-of-the-art architecture for image classification for many years and revolutionized the field of computer vision CNNReview CNNReview2 . Through convolution and pooling layers, CNNs are able to extract, preserve and structure spatial information, making them suitable for dealing with images. However, the architectural biases introduced through these layers force CNNs to rely on local receptive fields, thereby, among other limitations CNNOtherLimitations , hindering the networks from capturing long-range dependencies and global context CnnLimitations . More recently, Vision Transformers (ViTs) have taken over the leaderboards on image classification tasks (see ImageNetBest or CIFAR10Best ), showing the strength of attention-based mechanisms, which excel at capturing global relationships and thereby overcome some of the limitations of CNNs ViTsVsCnns . 2 Background/Motivation It is important to note that, even though Vision Transformers (ViTs) vision-transformer are superior to CNNs and have taken over the leaderboards, there are certain limitations in ViTs that deter its use such as requiring significant computational resources and since deep learning often involves trade-offs, it’s crucial to factor this in when planning to deploy a solution especially on edge devices. Furthermore, ViTs require large datasets to learn well which makes it unsuitable for cases where we do not have large amount of data. This makes a strong case that combining the strength of both architectures can lead to a more robust and versatile model. This idea has already been explored in the past. In SqueezeExcitation , the authors employ Squeeze-And-Excitation blocks, which encode inter-channel relationships and thereby model channel-wise attention. The authors of SpatialTransformers and UNet try a different approach by focusing on spatial features. Trying to tackle the shortcomings of these methods, the authors of CoordinateAttention specifically focus on how attention-mechanisms can be used to model long-range dependencies. Newer papers, like CANet or EfficientLocalAttention , focus on the combination of these ideas and refine them further. 2.1 What are we doing? We add 3 different attention mechanisms within a ResNet20 and observe how CNNs behave. 2.2 Novelty While the effectiveness of attention has already been demonstrated across various CNN architectures and datasets by the respective authors. Our approach differs from previous works in mainly 2 ways: • Attention not added after every convolution : We do not apply attention after every convolution operation as seen in CBAM but rather after a sequence of different convolution operations. We do it this way to achieve an efficient trade-off which enables us to evaluate if we can add attentions with reduced computation overhead. In our architecture, we have added attention only 3 times in the full architecture as seen in Figure 2 . • No information given about positions : We apply attention directly to convolutional layers without explicitly incorporating positional encodings unlike other approaches. 3 Datasets 3.1 CIFAR-10 We use CIFAR10 dataset CIFAR10 as one of our datasets. Krizhevskii et al created the dataset as a reliably labeled subset of the 80 million tiny images 80milliontiny in order to show how semi-supervised learning and pre-training can improve classification models Krizhevsky2009LearningML . The 80 million tiny images dataset contained only weak labeled images scraped from the web, so a subset of semantically similar labels was corrected and filtered, resulting in a self-contained data set that consists of 60000 corrected, randomly-selected, colored and labeled 32×32 images with 10 classes of 6000 instances each, which show animals or modes of transportation. The creators already apply a 50k/10k train/test split, and we evaluate our models on the test set. For tuning, we employ a 45k/5k train/validation split and pick the best performing model. We reuse the data augmentation techniques introduced in deepresiduallearningimage , meaning that for the training set we randomly flip the images horizontally and take random crops of padded (with a padding of 4) versions of the images. Both the training and testing set are normalized based on values from Martyn . 3.2 MNIST We use the MNIST dataset MNIST , a standard benchmark which comprises of 70,000 grayscale images of handwritten digits (0-9), each sized 28×28 pixels. The data was originally collected by the National Institute of Standards and Technology and pre-processed to include resizing, normalization, and anti-aliasing. Pixel values represent intensity levels, and the dataset includes four incorrectly labeled examples MNIST_wrongLabels . For more details, refer to the dataset source. 4 Approach For our baseline, we create an implementation of ResNet-20 (20 layers) inspired from deepresiduallearningimage . 4.1 CIFAR-10 Baseline First, we reduce the number of output channels in the initial convolution from 64 to 16 and adjust the kernel size, stride, and padding from 7, 2, and 3 to 3, 1, and 1, respectively. Second, we remove the subsequent max-pooling operation, as it is not mentioned in the paper. Third, we lower the number of residual stages from 4 to 3 and use output channels sizes of 16, 32 and 64. Fourth, in each residual block, when the spatial dimensions and number of channels increase, which applies identity mapping by padding feature maps with zeros to handle dimension mismatches. After experimenting with different variations we settled on the concise implementation shown in Idelbayev18a . Lastly, we adjusted the final layer to 64 input features, as the final number of output channels is 64, and map these to the 10 classes of CIFAR-10. By training & evaluating the re implementation with the same hyperparameters mentioned in original paper (see Table 1 ), we achieve a test error of 9.04, closely matching the error of 8.75 mentioned in the original paper. Furthermore, as shown in Figure 1 , when training the model on the entire training set as in deepresiduallearningimage , we achieve similar training and test error curves compared to the original papers. Hyperparameter Value Batch Size and Epochs 128 batch size, 182 epochs Learning Rate 0.1 Schedule (gamma=0.1) Milestones: 91, 136 Optimizer and Momentum SGD, 0.9 Weight Decay 0.0001 Table 1 : Hyperparameters adapted from deepresiduallearningimage . Milestones (91, 136 epochs) correspond to 32k and 48k iterations with a batch size of 128. 4.2 MNIST Baseline We extend the CIFAR-10 model to MNIST by changing input of initial convolutions from 3-channels to 1-channel and use it as our baseline model. Please note that the only reason we use MNIST is to ease our GradCAM gradcam evaluations, baseline results in CIFAR-10 are sometimes confused between the spatial area of interest and background elements. By including a qualitative analysis with the cleaner 1-ch images of MNIST, we aim to provide a clearer

Read the complete text