Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsity
Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee †, Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee † · 2024
Read Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsity by Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee †, Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee † (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10059.txt.
Introduction
Kam and co-authors design a neural network accelerator called Panacea in this 2024 paper. Large networks run faster when numbers are stored at low precision and split into slices. Skipping zero slices saves energy, but only with symmetric quantization, which loses accuracy.
Asymmetric quantization keeps accuracy yet produces many nonzero slices that older accelerators cannot skip. Their AQS-GEMM design instead compresses the frequent nonzero slices and skips their operations. Two joint algorithm-hardware tricks raise slice sparsity in the activations further. The result aims to give large-scale inference both high accuracy and low energy use. The work targets the matrix multiplications that dominate neural network cost.
Catalog record
| CREATOR | Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee †, Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee † |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Panacea: Novel DNN Accelerator using Accuracy-Preserving Asymmetric Quantization and Energy-Saving Bit-Slice Sparsity Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee †, Dongyun Kam † , Myeongji Yun † , Sunwoo Yoo † , Seungwoo Hong † , Zhengya Zhang ‡ , and Youngjoo Lee † Abstract Low bit-precisions and their bit-slice sparsity have recently been studied to accelerate general matrix-multiplications (GEMM) during large-scale deep neural network (DNN) inferences. While the conventional symmetric quantization facilitates low-resolution processing with bit-slice sparsity for both weight and activation, its accuracy loss caused by the activation’s asymmetric distributions cannot be acceptable, especially for large-scale DNNs. In efforts to mitigate this accuracy loss, recent studies have actively utilized asymmetric quantization for activations without requiring additional operations. However, the cutting-edge asymmetric quantization produces numerous nonzero slices that cannot be compressed and skipped by recent bit-slice GEMM accelerators, naturally consuming more processing energy to handle the quantized DNN models. To simultaneously achieve high accuracy and hardware efficiency for large-scale DNN inferences, this paper proposes an Asymmetrically-Quantized bit-Slice GEMM (AQS-GEMM) for the first time. In contrast to the previous bit-slice computing, which only skips operations of zero slices, the AQS-GEMM compresses frequent nonzero slices, generated by asymmetric quantization, and skips their operations. To increase the slice-level sparsity of activations, we also introduce two algorithm-hardware co-optimization methods: a zero-point manipulation and a distribution-based bit-slicing. To support the proposed AQS-GEMM and optimizations at the hardware-level, we newly introduce a DNN accelerator, Panacea , which efficiently handles sparse/dense workloads of the tiled AQS-GEMM to increase data reuse and utilization. Panacea supports a specialized dataflow and run-length encoding to maximize data reuse and minimize external memory accesses, significantly improving its hardware efficiency. Numerous benchmark evaluations show that Panacea outperforms existing DNN accelerators, e.g., 1.97 × \times and 3.26 × \times higher energy efficiency, and 1.88 × \times and 2.41 × \times higher throughput than the recent bit-slice accelerator Sibia and the SIMD design, respectively, on OPT-2.7B, while providing better algorithm performance with asymmetric quantization. I Introduction Large-scale deep neural networks (DNNs) have been employed as powerful solutions for a variety of practical applications, including image classification [ 1 , 2 , 3 ] , natural language processing (NLP) [ 4 , 5 , 6 , 7 , 8 , 9 ] , AR/VR [ 10 ] , and robotics [ 11 ] . However, their extensive computations and external memory accesses (EMA) result in poor throughput and increased energy consumption during inferences [ 12 , 13 , 14 , 15 ] , leading to more significant challenges for a large-scale DNN’s deployment on resource-constrained edge devices [ 16 , 17 ] . Fig. 1: Accuracy comparison on recent works utilizing symmetric [ 18 , 19 , 20 , 21 , 22 ] and asymmetric quantization [ 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ] for activations in large-scale DNNs. To address these challenges, recent works [ 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 ] have developed energy-efficient DNN inference accelerators by employing algorithm-hardware co-optimization techniques such as weight pruning [ 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 ] and quantization [ 49 , 41 , 42 , 43 , 45 , 44 , 46 , 47 , 48 ] . Among these techniques, low bit-precision post-training quantization (PTQ) [ 43 , 45 , 44 , 46 , 47 ] stands out by significantly lowering the complexity of operations and memory accesses, even without necessitating costly fine-tuning with large datasets. This results from enabling integer general matrix-multiplications (GEMMs) by quantizing weights and activations. However, PTQ-based methods, e.g., symmetric and asymmetric quantization, still have challenges for large-scale DNN inferences on resource-constrained devices, as summarized below. Accuracy degradation of symmetric quantization: While symmetric quantization generally enables low bit-precisions, it can cause significant accuracy loss during large-scale DNN inferences, such as transformer models, which contain layers producing asymmetric distributions or long-tail distributions in input activation matrices [ 50 , 51 ] . Recent studies at the algorithm-level [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 31 , 32 , 30 ] have addressed the accuracy degradation issue by applying symmetric quantization to weights and asymmetric quantization to activations, as illustrated in Fig. 1 . Consequently, to capitalize on the large-scale model’s high accuracy fully, it is crucial to utilize low-bit asymmetric quantization for input activations. HW optimization challenge of integer GEMMs with asymmetric quantization: However, integrating integer GEMMs with asymmetric quantization makes it difficult to utilize recent optimization techniques, such as sparsity, for energy-efficient DNN inferences [ 52 , 30 ] . While integer GEMMs with symmetric quantization [ 53 , 54 , 55 , 56 ] , have actively supported sparsity even for dense DNN models, which have high bit-slice sparsity ( > 90%) in high-order slices after segmenting near-zero values into multiple bit-slices, integer GEMMs with asymmetric quantization cannot exploit the slice sparsity due to few zero slices. Unlike symmetric quantization centering values around zero, as shown in Fig. 2 , asymmetric quantization produces quantized distributions that are not centered around zero, generating many nonzero slices that cannot be directly compressed and skipped. Therefore, to simultaneously enhance accuracy and hardware efficiency, it is essential that a new GEMM operation and its hardware architecture efficiently address the numerous multiply-and-accumulate (MAC) operations and memory accesses associated with nonzero slices. Fig. 2: Examples of uniform quantization methods: 8-bit (a) symmetric and (b) asymmetric approaches. This paper proposes the first asymmetrically-quantized bit-slice DNN accelerator, Panacea , successfully achieving high accuracy and hardware efficiency for resource-limited devices. The contributions of Panacea are summarized as follows: • The A symmetrically- Q uantized bit- S lice GEMM (AQS-GEMM) is newly developed to compress high-order nonzero slices, which are frequently observed in asymmetric quantization, and skip MAC operations of compressed slices. To get exact GEMM results, the AQS-GEMM efficiently computes a compensation term for skipping compressed slices, while reusing loaded data and minimizing additional memory accesses. It finally reduces the number of MAC operations by 61% compared to the dense GEMM, enabling bit-slice sparsity. • To increase high-order (HO) slice sparsity in activations, Panacea utilizes two algorithm-hardware co-optimization methods: Z ero- P oint M anipulation (ZPM) and D istribution-based B it- S licing (DBS). They are activated during the PTQ calibration (algorithm-level) and exploit higher sparsity during inferences (hardware-level). The ZPM properly adjusts the zero-point value of activation to put more quantized values into a slice skip range, thereby increasing the HO slice sparsity by at most 33%p. To further increase the sparsity, the DBS categorizes quantized distributions into three types during the calibration. Based on the identified types, the DBS applies different bit-slicing rules to different layers during the inference step by using low-cost shifting units, increasing the HO slice sparsity by at most 56%p and