Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
1 st Longsheng Zhou, 2 nd Yu Shen · 2026
Read Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression by 1 st Longsheng Zhou, 2 nd Yu Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04988.txt.
Introduction
Zhou and Shen study an ordered pipeline for shrinking neural networks under tight CPU and memory limits. Proxies such as parameter count poorly predict real inference time, and unstructured sparsity can even slow CPU runs. Their pipeline targets measured latency with pruning, INT8 quantization-aware training, and knowledge distillation. INT8 training gives the largest runtime gain, while pruning mainly regulates capacity, so step order matters for speed.
Catalog record
| CREATOR | 1 st Longsheng Zhou, 2 nd Yu Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33122 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression 1 st Longsheng Zhou, 2 nd Yu Shen Abstract Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse-kernel overhead. Motivated by this gap between compression and acceleration , we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and VGG-16-BN). Across all settings, the ordered pipeline achieves a stronger accuracy–size–latency frontier than any single technique alone, reaching 0.99–1.42 ms CPU latency with competitive accuracy and compact checkpoints. Controlled ordering ablations with a fixed 20/40/40 epoch allocation further confirm that stage order is consequential, with the proposed ordering generally performing best among the tested permutations. Overall, our results provide a simple guideline for edge deployment: evaluate compression choices in the joint accuracy–size–latency space using measured runtime, rather than proxy metrics alone. I Introduction Deep neural networks (DNNs) have achieved strong performance in vision, language, and multimodal tasks, but modern models are often over-parameterized and expensive to run. This becomes a real obstacle on resource-constrained platforms such as mobile devices, embedded systems, and edge accelerators, where memory footprint, power budget, and inference latency are all limited. In these settings, simply shrinking the backbone (fewer layers or channels) can quickly lead to noticeable accuracy degradation. Model compression provides a more controlled way to trade a small amount of accuracy for substantial efficiency gains [ 3 ] . Recent surveys summarize a wide range of compression techniques and emphasize that deployment constraints (hardware backend, operator support, and runtime overhead) often dominate practical outcomes [ 4 ] . A comparative study by Marinò et al. further suggests that different compression families typically improve different parts of the accuracy–efficiency space and that hybrid strategies are often preferable to single techniques [ 12 ] . In this work, we focus on three widely used building blocks: pruning, quantization, and knowledge distillation (KD). Pruning removes redundant parameters and can substantially reduce effective capacity with limited accuracy loss under appropriate retraining [ 7 ] , while its practical speed benefits depend on sparsity structure and hardware support [ 2 ] . Quantization reduces precision and storage cost, enabling efficient inference on standard integer backends (e.g., INT8) [ 16 ] . KD transfers knowledge from a high-capacity teacher to a constrained student and is often used to recover accuracy after compression [ 8 , 13 ] . We deliberately restrict ourselves to these standard components, aiming for a minimal and reproducible pipeline rather than introducing specialized operators or elaborate training tricks. Despite steady progress, combining these tools in a reliable way is still tricky. Many hybrid approaches are tailored to particular architectures or datasets, or rely on tightly coupled joint objectives and careful hyperparameter tuning [ 9 ] . Other hybrid ideas target different model families with different deployment bottlenecks, which may not directly carry over to standard CNN inference [ 15 ] . Meanwhile, practitioners often prefer simple recipes built from standard components, because they are easier to reproduce and integrate into existing toolchains [ 11 ] . This motivates a concrete question: can we design a minimal hybrid pipeline that consistently improves the joint accuracy–size–latency trade-off on mainstream convolutional backbones under measured CPU runtime, without specialized sparse kernels or intricate training tricks? We propose a fixed three-stage recipe: global unstructured pruning \rightarrow INT8 quantization-aware training (QAT) \rightarrow knowledge distillation , where the ordering is part of the method and targets a consistent deployable form (sparse INT8). Unstructured pruning alone often yields limited wall-clock benefits on general-purpose CPUs without specialized sparse kernels , but it reduces the active weight set and stabilizes subsequent low-precision optimization. INT8 QAT contributes most of the latency reduction on standard backends, and KD is applied last to recover accuracy after the model has entered the constrained sparse INT8 regime. Overall, the stages play complementary roles, and we validate the importance of ordering via a controlled stage permutation study. We evaluate the pipeline on three backbone–dataset pairs: ResNet-18 on CIFAR-10, WRN-28-10 on CIFAR-100, and VGG-16-BN on CIFAR-10. We compare against pruning-only, quantization-only, and KD-only baselines, and analyze the joint accuracy–size–latency space using both scalar metrics and multi-dimensional visualizations. In addition, to align with common co-compression benchmarks in prior work, we report a literature-aligned comparison on ResNet-20/CIFAR-10 using relative BOPs. Our main contributions are summarized as follows: • A minimal ordered co-compression recipe with role separation. We propose a simple three-stage pipeline—global unstructured pruning \rightarrow INT8 QAT \rightarrow KD—built from standard components and evaluated at a consistent deployable endpoint (sparse INT8), avoiding reliance on specialized sparse kernels. • Controlled evidence that stage ordering is consequential. Keeping the same ingredients, the same training budget, and the same deployable form, we perform a controlled stage-order ablation by permuting the three stages, showing that ordering alone can lead to clear accuracy differences while leaving latency within a narrow range. • Deployment-driven evaluation with consistent gains across settings. We evaluate compression choices in the joint accuracy–size–latency space using measured CPU runtime, demonstrating that proxy metrics (e.g., parameter/FLOP reductions) may not reflect wall-clock behavior on standard backends. Across three CNN backbones and two datasets, the proposed ordering yields consistently favorable trade-offs over single-stage baselines; we additionally report a literature-aligned comparison using relative BOPs on ResNet-20/CIFAR-10 to corroborate the conclusions under a common proxy metric. Fig. 1 : Overview of the proposed hybrid compression pipeline. A dense FP32 teacher with weights \mathbf{W}^{(0)} is first pruned into a sparse FP32 student ( \mathbf{W}^{(p)}=\mathbf{M}\odot\mathbf{W}^{(0)} ), then optimized under INT8 quantization (PTQ/QAT) to obtain a sparse INT8 student \mathbf{W}^{(q)}=Q_{8}(\mathbf{W}^{(p)}) , and finally refined via knowledge distillation to produce the hybrid student \mathbf{W}^{(h)} in the compressed space. II Related Work We review the three standard ingredients used in our pipeline—pruning, quantization, and knowledge distillation—and then briefly discuss hybrid, multi-stage recipes. II-A Pruning Pruning removes parameters (or structures) that contribute little to the final prediction. Early magnitude-based pruning with retraining showed that CNNs can often be made much smaller with only minor accuracy loss. Later work studied pruning together with quantization, including structured or coordinated pruning–quantization schemes [ 6 ] . In our setting, we care more about preserving accuracy on compact backbones than about relying on sparse kernels for speed. We therefore use global unstructured pruning mainly as an accuracy-friendly way to reduce the number of active weights before quantization, rather than treating pruning as the main source of wall-clock acceleration [ 10 ] .Recent analyses further suggest that unstructured sparsity mainly reduces storage and may not translate to latency gains on general-purpose CPUs unless supported by specialized kernels, which motivates using