VQ4ALL: Efficient Neural Network Representation via a Universal Codebook
Juncan Deng, Shuaiting Li, Zeyu Wang, Hong Gu, Kedong Xu, Kejie Huang · 2024
Read VQ4ALL: Efficient Neural Network Representation via a Universal Codebook by Juncan Deng, Shuaiting Li, Zeyu Wang, Hong Gu, Kedong Xu, Kejie Huang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06875.txt.
Introduction
Juncan Deng and five co-authors present VQ4ALL, a way to store neural networks in far less memory. Big models are heavy, and a compression method called vector quantization shrinks them by replacing groups of weights with shared code words. Older versions built a separate code table for each layer, which limited savings and slowed memory access.
VQ4ALL instead builds one universal codebook shared by many networks. The team extracts that codebook with a density estimation method, then constructs low-bit versions of different networks by gradually updating how weights are assigned to code words. Because the codebook never changes, it can sit in the chip's built-in read-only memory. In tests, the method compressed networks more than sixteen-fold while keeping accuracy high across several architectures, the authors report.
Catalog record
| CREATOR | Juncan Deng, Shuaiting Li, Zeyu Wang, Hong Gu, Kedong Xu, Kejie Huang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06875
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VQ4ALL: Efficient Neural Network Representation via a Universal Codebook Juncan Deng, Shuaiting Li, Zeyu Wang, Hong Gu, Kedong Xu, Kejie Huang Abstract The rapid growth of the big neural network models puts forward new requirements for lightweight network representation methods. The traditional methods based on model compression have achieved great success, especially VQ technology which realizes the high compression ratio of models by sharing code words. However, because each layer of the network needs to build a code table, the traditional top-down compression technology lacks attention to the underlying commonalities, resulting in limited compression rate and frequent memory access. In this paper, we propose a bottom-up method to share the universal codebook among multiple neural networks, which not only effectively reduces the number of codebooks but also further reduces the memory access and chip area by storing static code tables in the built-in ROM. Specifically, we introduce VQ4ALL, a VQ-based method that utilizes codewords to enable the construction of various neural networks and achieve efficient representations. The core idea of our method is to adopt a kernel density estimation approach to extract a universal codebook and then progressively construct different low-bit networks by updating differentiable assignments. Experimental results demonstrate that VQ4ALL achieves compression rates exceeding 16 × \times while preserving high accuracy across multiple network architectures, highlighting its effectiveness and versatility. 1 Introduction Neural networks have become indispensable tools in modern computer vision tasks due to their exceptional performance. However, these models are often resource-intensive, requiring substantial memory and computational power. Deploying them on resource-constrained platforms, where multiple networks may need to run simultaneously, remains a significant challenge. Addressing this limitation requires two critical advancements: enabling more universal network representation and more efficient network compression. Traditional methods for reducing model size and enabling hardware acceleration typically employ techniques such as pruning and quantization. While these methods can achieve acceleration to some extent, they often suffer from significant performance degradation when aiming for extreme model compression. This is primarily because low-bit quantization introduces large quantization errors, often requiring mixed precision to maintain performance. Meanwhile, pruning leads to the loss of important weights, causing substantial structural differences in the pruned model. Both methods result in low hardware compatibility, making it challenging to design a unified accelerator architecture. Vector quantization is a more hardware-friendly method of model compression that can achieve a higher compression ratio. However, traditional vector quantization requires training a separate codebook for each neural network, leading to long training times. Additionally, when achieving higher compression ratios, larger codebooks are often needed, resulting in longer loading times when switching between tasks. To address these challenges, we propose that it is essential to explore the underlying commonalities across neural networks. By leveraging these shared logical patterns, we can construct multiple neural network models in a bottom-up fashion. This approach not only effectively reduces the model size but also facilitates the design of unified low-level hardware, enabling general-purpose acceleration capabilities. Furthermore, it significantly reduces model compression time and holds the potential for enhancing the interpretability of neural networks. In this paper, we introduce VQ4ALL, a VQ-based method that enables universal network representation and efficient network compression. VQ4ALL adopts a kernel density estimation approach to extract a universal codebook and then learn the differentiable assignments to gradually construct the low-bit neural network. The universal codebook can be stored in ROM, which significantly reduces the silicon area and eliminates the need for repeated codebook loading during fast task switching. Experimental results demonstrate that our approach not only achieves accuracy comparable to the original models across various networks but also significantly reduces storage requirements. Our main contributions can be summarized as follows: • We propose VQ4ALL, a novel VQ-based approach that enables universal network representation and efficient network compression. • Our method introduces a kernel density estimation method to generate a universal codebook and a differentiable assignment learning strategy to progressively build low-bit neural networks. • Our method achieves high compression rates (exceeding 16 \times ) while maintaining accuracy comparable to original models across multiple architectures, demonstrating its effectiveness and versatility. Figure 1 : The bottom-up pipeline of VQ4ALL. The Universal Codebook is randomly sampled from the kernel density estimation of the floating-point sub-vectors of several networks. Based on the codebook, differentiable candidate assignments are assigned to each type of network, all initialized with the same ratios. The progressive network construction strategy then calibrates the ratios and sets candidate assignments with high ratios as optimal. Finally, various low-bit networks are constructed with the same universal codebook. 2 Related Work Quantization is a commonly used method in network compression, which is mainly divided into two types: uniform quantization and vector quantization. Uniform Quantization. Uniform quantization converts floating-point weights into integer values using scale coefficients, applied either tensor-wise, channel-wise, or group-wise. Most current methods focus on 8-bit or lower precision, often leveraging second-order information or quantization errors to enhance the process. To improve the accuracy of quantized models, training-time quantization techniques have been proposed. For instance, EWGS [ 14 ] adjusts gradients by scaling them based on Hessian approximations for each layer, while PROFIT [ 24 ] employs an iterative process and freezes layers depending on activation instability. Quant-Noise [ 6 ] , a structured dropout approach, applies quantization to a random subset of weights, allowing any quantization technique to be used. This approach enhances the predictive power of the compressed model. However, uniform quantization incurs a larger error at extremely low bit-width quantization due to its limitation of reconstructing weights in equidistant distributions, and the quantization coefficients cannot be shared across different DNNs. Vector Quantization. Vector quantization (VQ) in neural networks was first introduced by Gong et al . [ 8 ] , who explore scalar, vector, and product quantization (PQ) for fully connected (FC) layers. Wu et al . [ 35 ] extend PQ to compress both FC and convolutional layers of CNNs by sequentially quantizing layers to reduce error accumulation. DeepCompression [ 9 ] introduces k-means clustering for model compression by clustering weights and assigning the same value to all weights within a cluster. Son et al . [ 31 ] apply vector quantization to 3 \times 3 convolutions and fine-tune centroids via gradient descent, using extra bits to encode filter rotations, thus creating compact codebooks. HAQ [ 34 ] utilizes reinforcement learning to search for the optimal quantization policy across various tasks, combining k-means clustering, similar to DeepCompression, with flexible bit-widths for different layers. BGD [ 32 ] uses PQ to compress both convolutional and FC layers through a clustering approach aimed at minimizing the reconstruction error of layer outputs, followed by end-to-end training of the centroids via distillation. PQF [ 23 ] applies rate-distortion theory to reorder weights, reducing clustering errors, and further compresses the network using an annealing algorithm. GOBO [ 36 ] first identifies outlier weights that are far from the average of each layer’s weights and stores them uncompressed, while clustering the remaining weights using an algorithm similar to K-means. DKM [ 2 ] employs a differentiable K-means formulation integrated with the original loss function to recover the network’s accuracy. VQ4DiT [ 4 ] suggests that fixed initial assignments lead to suboptimal updates of the codebooks, and therefore proposes a