Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations

Dhananjay Saikumar, Blesson Varghese · 2025

Read Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations by Dhananjay Saikumar, Blesson Varghese (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15790.txt.

Introduction

Dhananjay Saikumar and Blesson Varghese diagnose why pruned neural networks fail in a 2025 paper. Pruning cuts weights from a network so it can run on small hardware. One shot methods cut once, then measure the damage. The usual story blames the loss of key weights. The authors find a different culprit: signal collapse, a fading of activation spread as signals pass through the cut network's layers.

Their fix, REFLOW, restores the signal spread without retraining any weights. It reveals good sparse networks hiding inside the original weights. With plain magnitude pruning plus REFLOW, a large image model jumped from near useless accuracy to near its full score on a standard test, while keeping only a fifth of its weights. The authors argue that tending the signal, not picking weights with care, is the key lever.

Catalog record

CREATORDhananjay Saikumar, Blesson Varghese
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15790

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations Dhananjay Saikumar, Blesson Varghese Abstract Neural network pruning is essential for reducing model complexity to enable deployment on resource-constrained hardware. While performance loss of pruned networks is often attributed to the removal of critical parameters, we identify signal collapse —a reduction in activation variance across layers—as the root cause. Existing one-shot pruning methods focus on weight selection strategies and rely on computationally expensive second-order approximations. In contrast, we demonstrate that mitigating signal collapse, rather than optimizing weight selection, is key to improving accuracy of pruned networks. We propose REFLOW that addresses signal collapse without updating trainable weights, revealing high-quality sparse sub-networks within the original parameter space. REFLOW enables magnitude pruning to achieve state-of-the-art performance, restoring ResNeXt-101 accuracy from under 4.1% to 78.9% on ImageNet with only 20% of the weights retained, surpassing state-of-the-art approaches. 1 Introduction Neural networks are widely used across applications like natural language processing [ 32 ] and computer vision [ 18 , 14 ] . However, their increasing size, often reaching billions of parameters [ 36 , 30 ] , presents significant computational and memory challenges, making deployment in resource-constrained environments impractical [ 26 ] . Network pruning has emerged as a key technique to reduce model complexity and enable faster inference [ 34 , 16 , 21 , 33 , 31 ] . The increase in neural network sizes [ 36 ] and the widespread availability of pre-trained models have shifted the focus of pruning strategies. Gradual pruning, which iteratively removes parameters with retraining [ 38 , 19 , 8 , 4 ] , is computationally expensive for large models, often requiring days or weeks of fine-tuning [ 1 ] . This becomes infeasible for modern networks, particularly in settings with limited training data, such as privacy-sensitive or low-data scenarios [ 28 ] . As a result, one-shot pruning, which compresses pre-trained models in a single step [ 28 , 37 , 1 , 29 , 6 ] , has emerged as a scalable and efficient alternative, directly addressing the runtime and computational cost of gradual pruning for large-scale models. Pruning methods fall into two categories: magnitude pruning (MP) and impact-based pruning (IP) . MP [ 11 , 24 , 10 , 9 ] removes small-magnitude weights, but may not be effective as magnitude alone may not reliably capture parameter importance [ 1 ] . IP methods, such as Optimal Brain Damage [ 20 ] , Optimal Brain Surgeon [ 13 ] , and their modern extensions, use loss-aware weight selection via second-order approximations to identify and remove low-impact weights, followed by weight updates to compensate for loss. Despite outperforming MP, these methods are computationally expensive as they rely on second-order approximations to select which weights to prune. This raises an important question: What drives the higher pruning quality produced by IP methods compared to MP? It is natural to attribute the success of IP methods to loss-aware weight selection , which identifies and preserves critical parameters for the network’s performance [ 20 ] . To test this assumption , we systematically decouple the two stages of IP: (i) weight selection to determine which weights to prune, and (ii) Hessian-based weight update(s) to adjust the remaining weights and reduce the impact on loss after pruning. Surprisingly, our analysis highlights that weight selection alone contributes minimally to the final performance. Variants of IP-based methods, namely WoodFisher, CBS, and CHITA, that only select weights (WoodFisher-S, CBS-S, CHITA-S) achieve comparable accuracy to MP (Figure 1 , left). The substantial performance improvement is achieved only after Hessian-based updates (Figure 1 , right). 0.5 0.6 0.7 0 10 20 30 40 50 60 Network Sparsity ( \kappa ) Accuracy Gain (%) Weight Selection WF-S CBS-S CHITA-S 0.5 0.6 0.7 Network Sparsity ( \kappa ) Weight Selection + Weight Updates WF-U CBS-U CHITA-U Figure 1: Comparison of test accuracy gain of impact-based pruning methods over magnitude pruning for a pre-trained MobileNet on ImageNet at different sparsity levels. Left: Selection-only pruning methods. Right: Pruning methods with weight updates show significant performance gains. These findings challenge the conventional assumption and leads to a further question: If weight selection—whether based on magnitude or loss-aware criteria—plays a minimal role, what fundamentally drives the performance degradation of one-shot pruned networks? Our investigation uncovers a new explanation: signal collapse . One-shot pruning alters the flow of activations through the network, progressively reducing their variance in deeper layers. This leads to signal collapse, where activations in the final layers become nearly constant, preventing the network from distinguishing between inputs. As a result, predictions become uniform, severely impairing performance. While accuracy loss due to pruning is conventionally attributed to the removal of critical parameters, our results reveal that signal collapse is the primary driver of degraded performance in pruned networks. The key insight from our work is that signal collapse can be mitigated and simple methods, such as MP, can surpass the performance of state-of-the-art IP-based methods. Our work REFLOW ( Re storing F low of Low -variance signals) mitigates signal collapse without requiring gradient or Hessian-based computations. REFLOW addresses the overlooked bottleneck of signal collapse, demonstrating that high-performing sparse sub-networks inherently exist within the original parameter space identified. REFLOW achieves substantial accuracy recovery when applied to networks with various neural network architectures pruned with MP. For instance, at 80% sparsity on ImageNet, ResNet-152 recovers from under 1% to 68.2%, and ResNeXt-101 improves from less than 4.1% to 78.9%. These results highlight that high-quality sparse models can be obtained by restoring activation flow rather than optimizing weight selection. REFLOW further reveals that high-performing sparse networks inherently exist within the original weights. Contributions. This work makes the following key observations and contributions: 1. For the first time, we identify signal collapse as the leading cause of accuracy loss in addition to the removal of critical weights, . 2. Signal collapse can be mitigated without updating any trainable weights. Our work REFLOW restores activation flow, enabling networks pruned by MP to outperform IP methods without gradient or Hessian computations . 3. We demonstrate that high-performing sparse sub-networks inherently exist in the original parameter space . Unlike IP methods, which rely on updating unpruned weights to find a solution outside the original parameter space, our approach addresses signal collapse to uncover these sub-networks directly within the original weights. 2 Background & Related Work This section provides the mathematical formulation of pruning and reviews existing work on pruning techniques. 2.1 Problem Setup Consider a pre-trained deep neural network (DNN) f(\theta;x) parameterized by \theta\in\mathbb{R}^{d} and input x . Pruning produces a sparse sub-network f(\theta\odot m;x) , where m\in\{0,1\}^{d} is a binary mask, and \odot denotes element-wise multiplication. Sparsity \kappa\in[0,1] is the proportion of parameters set to zero. Pruning assigns scores z\in\mathbb{R}^{d} to parameters importance, using methods ranging from simple weight magnitude to loss-aware based pruning scores. 2.2 Related Work Magnitude-Based Pruning (MP) is a simple and widely used pruning strategy [ 10 , 5 , 24 , 22 , 31 , 27 , 9 , 11 , 23 , 3 ] . MP ranks weights based on their absolute values: z_{i}=|\overline{\theta}_{i}|. (1) It prunes parameters with the smallest magnitudes, which is

Read the complete text