Compression for Better: A General and Stable Lossless Compression Framework
Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangmin Liu, Wenguang Chen · 2024
Read Compression for Better: A General and Stable Lossless Compression Framework by Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangmin Liu, Wenguang Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06868.txt.
Introduction
Zhang, Cheng, Zhang, Liu, and Chen present a paper from twenty twenty-four on a theory for shrinking neural networks without losing skill. Compression usually trades size for accuracy in murky ways. Their framework defines a safe neighborhood where compression stays lossless. Differential math draws that boundary and higher-order limits around it. Quantization is recast as a grouped packing puzzle inside the safe zone. Decomposition gets automatic per-layer ranks from the same theory. Tests span many network types and datasets.
Lossless shrinkage is achieved without fancy tricks, they report. Efficiency rises alongside the size cuts. Code will be released in public. The authors frame the boundary idea as the missing systematic piece.
Catalog record
| CREATOR | Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangmin Liu, Wenguang Chen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06868
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Compression for Better: A General and Stable Lossless Compression Framework Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangmin Liu, Wenguang Chen Abstract This work focus on how to stabilize and lossless model compression, aiming to reduce model complexity and enhance efficiency without sacrificing performance due to compression errors. A key challenge is effectively leveraging compression errors and defining the boundaries for lossless compression to minimize model loss. i.e., compression for better. Currently, there is no systematic approach to determining this error boundary or understanding its specific impact on model performance. We propose a general L oss L ess C ompression theoretical framework ( LLC ), which further delineates the compression neighborhood and higher-order analysis boundaries through the total differential, thereby specifying the error range within which a model can be compressed without loss. To verify the effectiveness of LLC, we apply various compression techniques, including quantization and decomposition. Specifically, for quantization, we reformulate the classic quantization search problem as a grouped knapsack problem within the lossless neighborhood, achieving lossless quantization while improving computational efficiency. For decomposition, LLC addresses the approximation problem under low-rank constraints, automatically determining the rank for each layer and producing lossless low-rank models. We conduct extensive experiments on multiple neural network architectures on different datasets. The results show that without fancy tricks, LLC can effectively achieve lossless model compression. Our code will be made publicly. 1 Introduction The scale and complexity of Deep Neural Networks (DNNs) have rapidly increased, driving up memory and FLOPS demands. To tackle these challenges, model compression has become a crucial method for improving efficiency, reducing energy consumption, and speeding up inference. However, achieving effective compression without sacrificing performance remains a key challenge. As such, model compression must balance two critical objectives: maximizing the compression ratio while preserving model performance. In this work, we focus on post-training compression, where the model is compressed after training without requiring modifications to the training process. Its advantage is that the computing resource consumption is low and there is no need to adjust the model training process. In such scenarios, most existing compression schemes focus on maximizing the compression rate while trying to optimize the performance of the compressed model. Taking model quantization and matrix decomposition as examples, quantization significantly speeds up inference and reduces model size by using lower bit widths to represent tensors. For example, Banner et al. Banner et al. 2018 minimized tensor-level errors to partially mitigate the accuracy loss incurred during quantization, achieving model performance close to the unquantized original model in the low-bit case. HAWQ Dong et al. 2019 employed layer-wise sensitivity metrics to determine the precision of different layers, striking a good balance between error and compression ratio. On the other hand, matrix decomposition decomposes the weight matrix into two or more smaller matrices, using these smaller matrices during actual storage and computation. Compared to other compression methods, matrix decomposition has a solid mathematical foundation and can retain the latent information in the data. For example, Yu et al. Yu et al. 2017 combined feature map reconstruction and low-rank factorization to effectively reduce the number of parameters in fully connected layers. Hsu et al. Hsu et al. 2022 incorporated weighted Fisher information into singular value decomposition to reduce the model degradation after decomposition. Although these methods strive to reduce the performance degradation caused by compression at different compression rates, they still struggle to avoid accuracy drop-offs, whether under high or low compression ratios. For example, even at lower compression rates, model loss tends to increase, making it difficult to fully retain the performance of the original model. Unlike most existing methods that focus on reducing degradation under high compression conditions, our research focuses on lossless compression, i.e., exploring the range of compression errors that can be tolerated by a model, without compromising its performance. In this work, we first attempted universal lossless compression and proposed a LossLess Compression theory framework (LLC). LLC reveals the relationship between compression error and model gradient direction through full differential analysis, which can be used to guide the lossless compression. LLC also clearly defines the error neighborhood of lossless compression and determines the boundary through second-order Hessian analysis. We apply model quantization and matrix decomposition to the LLC framework: for quantization, we transform the quantization search problem into a grouped knapsack problem to improve computational efficiency while ensuring lossless quantization; for decomposition, we combine the compression error neighborhood and low-rank constraints to generate lossless low-rank models. Experiments show that LLC can effectively compress models without loss under multiple datasets and different network architectures while ensuring compression rate, and even obtains compressed models with lower loss than the original model. For example, LLC compresses the volume of the ResNet series model by nearly 70%, achieving better performance than the original model. Our contributions are as follows: 1) We propose a universal lossless compression framework that provides guidance on how compression errors can be used for lossless model compression. 2) We apply LLC to quantization and matrix decomposition: by transforming the quantization search problem into a knapsack problem, we ensure lossless compression; in matrix decomposition, we combine the error neighborhood with low-rank constraints to successfully generate lossless low-rank models. 3) Experimental results across multiple datasets and neural network architectures verify the effectiveness of the LLC framework. 2 Related Works 2.1 Quantization Quantization uses low-bit-width representations for tensors while maintaining their dense format, aiming to reduce model storage and computational overhead. In typical setups, mixed-precision quantization strategies are employed, where different layers are assigned varying bit-widths based on their sensitivity to quantization. This approach minimizes performance loss after compression. For example, HAQ Wang et al. 2019 used reinforcement learning to determine the quantization strategy for each layer, incorporating feedback from hardware accelerators to improve computational efficiency. AutoQ Lou et al. 2019 introduced a layered deep reinforcement learning (DRL) method that sequentially determines kernel bit-widths. HAWQ Dong et al. 2019 employed the top Hessian eigenvalues to measure each layer’s sensitivity to quantization, providing a relative sensitivity score, although bit-width allocation still relies on manual selection. HAWQ-V2 Dong et al. 2020 replaced this with the trace of the Hessian matrix. BRECQ Li et al. 2021 further introduced block-wise optimization, which used different granularities of quantization to significantly reduce the model degradation induced by quantization. While these methods narrow the performance gap between the compressed and original models in practice, model degradation is still difficult to fully avoid, even under 8-bit quantization. Moreover, although these methods are effective empirically, they lack a principled explanation of optimality. Furthermore, bit-width assignment for each layer leads to an exponentially growing search space, decreasing efficiency. 2.2 Decomposition Traditional decomposition methods, such as Singular Value Decomposition(SVD), CANDECOMP/PARAFAC(CP), and Tucker decomposition, involve decomposing model weight matrices and directly assigning the decomposed weights back to the original model. However, this approach often leads to significant increases in model loss, typically rising 5–10 times compared to the original model. To mitigate this issue, existing methods incorporate fine-tuning after decomposition, which entails retraining to reduce the loss. Yu et al. Yu et al. 2017 leveraged weight structure information by combining low-rank weight matrices and feature map reconstruction to reduce fully-connected layer parameters. Xu et al. Xu et al. 2019 integrated low-rank approximation