Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding
Alexander Conzelmann, Robert Bamler · 2025
Read Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding by Alexander Conzelmann, Robert Bamler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18758.txt.
Introduction
Alexander Conzelmann and Robert Bamler present a method to shrink stored neural networks in a 2025 paper. Growing model sizes strain small devices such as embedded sensors. Compression can help, as long as performance stays close to the original model. Their framework works after training and joins quantization, which reduces number precision, with entropy coding, which packs the result tightly.
They extend a classic layer wise loss with a cost term for the storage rate, estimated by a simple quadratic form. Exact local solutions follow a classic brain surgeon method for pruning and tuning. On several vision networks, the method cut bit rates by twenty to forty percent at the same performance as a leading codec. Code is shared in public.
Catalog record
| CREATOR | Alexander Conzelmann, Robert Bamler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding Alexander Conzelmann, Robert Bamler Abstract The ever-growing size of neural networks poses serious challenges on resource-constrained devices, such as embedded sensors. Compression algorithms that reduce their size can mitigate these problems, provided that model performance stays close to the original. We propose a novel post-training compression framework that combines rate-aware quantization with entropy coding by (1) extending the well-known layer-wise loss by a quadratic rate estimation, and (2) providing locally exact solutions to this modified objective following the Optimal Brain Surgeon (OBS) method. Our method allows for very fast decoding and is compatible with arbitrary quantization grids. We verify our results empirically by testing on various computer-vision networks, achieving a 20-40% decrease in bit rate at the same performance as the popular compression algorithm NNCodec. Our code is available at https://github.com/Conzel/cerwu . 1 Introduction While neural networks have achieved impressive results on various tasks, their large computational demands require many neural network applications to rely on server-side deployment. This increases latency and can cause concerns for end users regarding privacy and regulatory constraints ( saravananAdvancementsOnDeviceDeep2023 ; hohmanModelCompressionPractice2024 ) . Thus, there has been an increased effort to move machine learning pipelines back onto user devices, which usually involves techniques that reduce their resource demands hohmanModelCompressionPractice2024 . The works on neural network compression can be roughly divided into two groups: (1) works that aim to reduce the server-side costs of training liMemoryEfficientOptimizers2023 ; haoNeuZipMemoryEfficientTraining2024 , fine-tuning ( dettmersQLoRAEfficientFinetuning2023a ; qinAccurateLoRAFinetuningQuantization2024 ) or inference ( kwonEfficientMemoryManagement2023 ; hooperKVQuant10Million2024 ; parkInferenceOptimizationFoundation2024 ) . These methods often save computational costs or reduce GPU memory requirements. And (2) works that focus on embedded and edge devices , aiming to produce extremely small models still suitable for inference ( howardSearchingMobileNetV32019 ; guMiniLLMKnowledgeDistillation2023 ) , to reduce energy demands ( kroukaEnergyEfficientModelCompression2021 ; baskinCATCompressionAwareTraining2021 ) or to decrease the storage requirements ( choiUniversalDeepNeural2020 ; beckingNNCodecOpenSource2023 ) . While both groups use similar techniques, such as pruning, quantization and entropy coding, the methods might not be used interchangeably, as they target different architectures and scenarios. Our work explicitly focuses on the storage requirements of neural networks, which we argue is important for more widespread adoption of neural networks on edge devices. These devices often have limited storage capacity or network bandwidth for software updates, causing model-size issues with even the simplest of neural networks (even ResNets ( heDeepResidualLearning2016 ) have an uncompressed storage size of hundreds of megabytes). The importance of storage size has been reinforced with the introduction of the ISO/IEC 15938-17 standard, a technical standard for the compression of neural networks. So far, few other works explicitly target the storage size of neural networks. When focusing just on inference speed, quantization (i.e., saving each parameter with a lower precision) is often enough, as current GPU kernels only experience a speed-up for 4-bit or 8-bit quantization (see ( nagelWhitePaperNeural2021 ) ). However, to reduce storage size even further, a more powerful technique is to use entropy coding, which builds a probabilistic model of the data and then encodes the data points into bit strings whose length is proportional to their information content (negative log-probability), allowing us to encode parameters with “fractional bits” , in some cases even achieving rates below 1 bit per weight. In this work, we combine quantization with entropy coding, and propose a novel approach that produces highly compressible quantized layer representations by adding a quadratic rate-estimation to the layer-wise loss ( nagelAdaptiveRoundingPostTraining2020 ) , searching for rate-distortion optimal quantization choices, and then providing entropy-regularized optimal weight updates following the Optimal Brain Surgeon (OBS) ( hassibiOptimalBrainSurgeon1993 ) framework. In summary, our proposed network compression scheme • achieves extremely low storage cost for the compressed networks; • allows for extremely fast decoding of compressed networks; • produces quantized representations suitable for fast inference using integer arithmetic; and • is flexible in regards to the choice of quantization grid and entropy model used, allowing users to, e.g., trade off decompression or inference speed against model performance. We name our method CERWU ( C ompression with E ntropy- R egularized W eight U pdates). In the rest of this paper, we discuss related work ( section 2 ) and the information-theoretical background ( section 3 ), derive the proposed entropy-regularized weight updates ( section 4 ), and verify the effectiveness of our method empirically ( section 5 ) on various networks from the computer vision community. We conclude with a discussion of limitations of our method ( section 6 ). 2 Related Work Various target scenarios exist in network compression. Our method is a post-training compression method that reduces the storage size of a network. Other compression schemes focus instead on decreasing the memory footprint on GPUs shengFlexGenHighThroughputGenerative2023 ; kwonEfficientMemoryManagement2023 , reducing energy costs ( koAdaptiveWeightCompression2017 ; chmielFeatureMapTransform2019 ; kroukaEnergyEfficientModelCompression2021 ) , increasing inference speed liPENNIPrunedKernel2020 ; dettmersLLMInt88bit2022 or they require re-training for each tested compression setting baskinCATCompressionAwareTraining2021 ; chenEfficientQATEfficientQuantizationAware2024 . Most related work focuses on network quantization rather than compression (often aiming to improve inference speed). We discuss the difference between the two in section 3.1 and existing quantization methods in section 3.3 . There are fewer existing works with our focus on storage size. Han et al. ( hanDeepCompressionCompressing2016 ) combine pruning, quantization, weight-sharing and entropy coding to achieve very high compression factors. Universal Neural Network Compression ( choiUniversalDeepNeural2020 ) uses universal quantization ( zivUniversalQuantization1985 ) together with vector quantization. Wiedemann et al. wiedemannCompactComputationallyEfficient2020 propose to use an entropy-constraint together with a sparsification process, resulting in parameter representations that allow for high compression ratios. The Neural Network Compression and Representation Standard (ISO/IEC 15938-17) ( kirchhofferOverviewNeuralNetwork2022 ) , with its reference implementation NNCodec ( beckingNNCodecOpenSource2023 ) , defines a compression pipeline encompassing parameter reduction (pruning, sparsification, parameter sharing, et cetera), quantization, and entropy coding, as well as interoperability with well-known neural network exchange formats such as ONNX ( bai2019 ) . 3 Background 3.1 Compression vs. Quantization Neural network compression is often conflated with quantization, but the two are fundamentally different. In its most basic form, quantization reduces the precision of each network weight to a fixed (integer) bit-width (e.g., 4 or 8 bits per weight). By contrast, compression is a more global operation that aims to find the most compact binary representation of a network by packing the most relevant information contained in all weights into a bit string of shortest possible length (called bit rate ). Compression thus effectively maps weights to varying bit-widths, and modern compression methods ( rissanen1979arithmetic ; pasco1976source ; martin1979range ; duda2015use ; bamler2022understanding ) are not even constrained to bit boundaries, i.e., they can effectively assign fractional bit-widths to weights, including widths below 1 bit per weight, see section 3.2 below. The simplest way to combine quantization and compression is to first quantize the weights to a fixed bit-width and then apply lossless compression. However, this is empirically far from optimal (see ablation “CERWU- \lambda\!=\!0 ” and baseline “RTN+EC” in section 5 ) because quantizing without consideration of the compression mechanism produces poorly compressible quantized weights. Recent literature includes more advanced network quantization schemes that go beyond a fixed bit-width per weight, either by combining quantization with pruning