Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks

David A. Danhofer · 2024

Read Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks by David A. Danhofer (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00288.txt.

Description

Abstract The crucial role of convolutional models, both as standalone vision models and backbones in foundation models, necessitates effective acceleration techniques. This paper proposes a novel method to learn semi-structured sparsity patterns for convolution kernels in the form of maskings enabling the utilization of readily available hardware accelerations.

Catalog record

CREATORDavid A. Danhofer
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks David A. Danhofer Abstract The crucial role of convolutional models, both as standalone vision models and backbones in foundation models, necessitates effective acceleration techniques. This paper proposes a novel method to learn semi-structured sparsity patterns for convolution kernels in the form of maskings enabling the utilization of readily available hardware accelerations. The approach accelerates convolutional models more than two-fold during inference without decreasing model performance. At the same time, the original model weights and structure remain unchanged keeping the model thus easily updatable. Beyond the immediate practical use, the effect of maskings on prediction is easily quantifiable. Therefore, guarantees on model predictions under maskings are derived showing stability bounds for learned maskings even after updating the original underlying model. 1 1 1 Code available at github.com/ddanhofer/Semi-Structured-Sparsity-CNNs 1 Introduction The increasing complexity of deep learning models [ 21 ] , their deployment in applications [ 5 ] , and the adoption of reflection incurring several inference passes per query, e.g., as in the O1 models from the GPT family [ 3 ] , shifts the relative amounts of resources spent during the model lifetime from the training to the inference stage [ 7 , 35 ] . It therefore becomes imperative to make models more efficient [ 46 ] . One way of achieving this is by spending a comparatively small, additional share of resources during training to learn a one-time modification of the model that lowers the model’s inference and thus lifetime cost [ 30 , 40 ] . First and foremost, such a modification is effective if it decreases the model’s computational and time cost at a relatively low additional training overhead while not affecting the prediction performance of the model negatively [ 22 ] . Additionally, there are other desirable properties of such one-time modifications: From an application perspective the achievable gain in efficiency is only useful if it can be leveraged easily, a well-known challenge, e.g., with sparsifying models [ 8 , 15 ] . Taking into consideration the increasing popularity of large, expensive to train, foundation models [ 16 ] or models employed in an online setting subject to continuous updates the proposed change should not affect the possibility to update the model, e.g., by changing the weights or architecture underlying the model. Ideally, if such a model is updated, the learned modification can even be reused under the constraint of the magnitude of change imposed by updating the model. Semi-structured sparse maskings satisfy the above properties by replacing the dense matrix operations usually required during inference by cheaper and faster operations on semi-structured sparse matrices [ 4 ] . While many works have demonstrated that sparse (pruned) submodels can solve the same task at almost no loss of performance [ 2 , 26 ] the sparsity of the models does not necessarily have to adhere to a specific pattern making it difficult to turn theoretically obtained computational speedups by saving on data loading and computational operations into practical efficiency gains [ 14 ] . Regular patterns are more “machine-friendly” inducing the desired efficiency a priori but limiting the choices for the sparse patterns, which thus need to be chosen carefully with the goal of minimizing the loss of inference performance in mind. This paper proposes a novel method of learning regularly sparse masking patterns for convolutions, key building blocks for state-of-the art Computer Vision (CV) models [ 25 ] and foundation models building on CV models as their backbone [ 38 ] . The proposed method • shows how to effectively use readily available hardware accelerations for semi-structured sparse matrices in convolution kernels to accelerate inference, • outperforms available heuristics for semi-structured sparsity showing that semi-structured sparsity masks can be learned with a fraction of the original training resources while incurring a negligible performance loss in CV classification tasks, • provides the additional advantage of not changing the original set of trained weights keeping models updatable and rendering the method especially attractive for use in large models, e.g., foundation models and in online settings, • induces an easily quantifiable change to the model’s prediction behavior and thus lends itself to settings where hard guarantees on model predictions are of interest. In the following section the adoption of semi-structured sparsity and sparsity in convolutional models are addressed. Section 3 of the paper covers modeling semi-structured sparsity in general, in convolutional models, and the theoretical implications of such model alterations in inference. The results of empirically testing the method on widely used convolutional architectures are presented in Section 4 followed up by a discussion of the method presented and a conclusion. 2 Related Work In the following the notion and adoption of semi-structured sparsity is introduced. Then, the implications on prediction performance and the computational challenges of sparsifying Convolutional Neural Networks (CNNs) are highlighted. Semi-Structured Sparsity Semi-structured sparsity to accelerate network inference has been introduced in [ 37 ] as N:M-sparsity requiring N out of M elements of a contiguous block to be zero (s. a. 3.1 ). Beyond the general case of N:M sparsity, the practically interesting special case of 2:4 sparsity has been considered in more detail [ 17 , 18 ] in which exactly half of the weights are pruned as illustrated in Figure 1 . This setting enables hardware acceleration via NVIDIA sparse tensor cores available from the NVIDIA Ampere architecture on via the TensorRT v8.0 library [ 36 ] . Since half the elements are zeroed out and thus negligible, the amount of data to load from memory is almost halved with the number of Floating Point Operations (FLOPs) needed to conduct an operation on the sparse matrix also decreasing, e.g., linearly for addition and element-wise operations and super-linearly for multiplication, decomposition etc. [ 44 ] . This way 2:4 sparse matrix operations compute the same effective operation while reducing the time needed by a factor of two [ 36 ] . The difficulty in turning a dense matrix into a 2:4 sparse matrix, however, lies in selecting the most useful two of the four weights in each quadruple. To this end [ 37 ] propose a permutation regime that allows for preserving the weights based on magnitude and assess the found pattern via a scoring mechanism, the efficacy score. The functionality is available via NVIDIA’s Apex library [ 33 ] . Notably, pruning via Apex requires finetuning the network again after pruning to achieve an inference performance comparable to that of the dense network in CV tasks, e.g., classification [ 36 , 37 ] , and therefore changes the original pretrained weights. Figure 1: A 2:4 sparse matrix of floating point values – obtained from a dense matrix and a sparse bit mask – and its equivalent structured representation containing only the non-zero entries and a 2-bit index preserving the structure taking up roughly only half the space. Sparsity in CNNs The state-of-the-art performance of CNNs in image-based and other tasks comes at the cost of a large memory footprint and computational cost. Pruning to obtain sparse networks is therefore a popular technique to decrease the computational and storage cost of deep neural networks [ 2 ] . Pruning

Read the complete text