STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing

Shir Maon, Odelia Melamed, Adi Shamir · 2026

Read STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing by Shir Maon, Odelia Melamed, Adi Shamir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01126.txt.

Introduction

Maon, Melamed, and Shamir heal pruned networks by matching their inner states in this 2026 paper. Pruning cuts weights to speed a network up. Accuracy falls, so a healing phase usually follows. Their method, STARFISH, retrains the pruned net to align with the original net's internal representations. A tiny set of unlabeled examples is all the calibration it needs.

For the common case of half the weights cut, STARFISH lifts recovered accuracy by up to 22 percent over top methods on Vision Transformer networks, they report. The edge grows under harsh pruning. With three quarters of a DeiT-B network's weights gone, STARFISH uses only 0.4 percent of the training images as a calibration set in the recovery run. It recovers 82 percent of the dense model's accuracy. Rival methods reach only 40 percent. The lesson is that inner states carry the knowledge worth restoring. Healing can be fast when it aims at them.

Catalog record

CREATORShir Maon, Odelia Melamed, Adi Shamir
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01126

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing Shir Maon, Odelia Melamed, Adi Shamir Abstract Pruning is a process designed to reduce the number of weights in a large neural network. This can substantially speed up inference but might cause a considerable reduction in the model’s accuracy, and thus it is usually followed by a healing process that regains some of the lost accuracy. In this paper, we propose a new healing method, STARFISH , that can recover (most of) the accuracy of any pruned network efficiently. The main idea of STARFISH is to optimize the pruned network to align with the original network’s internal state representations using a tiny calibration set of unlabeled examples. For the common case of removing 50 % 50\% of the weights, STARFISH healing improves the recovered accuracy by up to 22 % 22\% over the state-of-the-art methods on ViT-based networks. Its advantage is even more pronounced under aggressive pruning. For example, after eliminating 75 % 75\% of the weights in a DeiT-B network for ImageNet, STARFISH uses only 0.4 % 0.4\% of the number of training images as a calibration set and recovers 82 % 82\% of the original dense accuracy, whereas competing recovery techniques reach only 40 % 40\% of the dense model accuracy. Open-source code is available here . 1 Introduction Modern neural networks achieve increasingly strong performance, often at the cost of a growing parameter count. This growth makes deployment expensive in terms of memory, latency, and energy consumption. Pruning addresses this problem by removing some weights, neurons, channels, attention heads, or other computational units from a pretrained model, producing a smaller and more efficient network while attempting to preserve the predictive performance of the original dense model. When performing pruning, the desired sparsity , namely the fraction of removed weights, plays a key role. While low sparsity saves only a limited amount of space and computation, high sparsity pruning often causes a substantial drop in performance. Early pruning and recovery breakthroughs report high accuracy with very high sparsity, removing up to 98.5\% of the weights in convolutional neural networks (CNNs; [ 24 ] ) [ 8 ] . Yet, modern large-scale pruning and recovery methods struggle to restore accuracy after high-sparsity pruning, and excel in lower-sparsity settings of 50\% of weights removed. Though it has been shown that an optimal pruning might not need recovery [ 21 , 25 ] , many pruning pipelines include a recovery procedure for the remaining weights, which is often the most expensive part of the pipeline. This line of work started in the simplest pruning method followed by retraining or fine-tuning of the remaining weights, often through multiple steps of low-sparsity pruning and healing schedules until the target sparsity is reached [ 11 , 37 ] . Inspired by the Lottery Ticket Hypothesis [ 7 ] , other iterative pruning variants include rewinding the surviving weights to an early training checkpoint, before retraining the sparse model [ 8 , 26 ] . As neural networks and the corresponding datasets grow larger, these procedures require substantial computing resources. In addition, they require access to the original training data and knowledge of its randomness, which are often unrealistic in modern settings, where training details are typically treated as proprietary knowledge, even in open-source models. Thus, a line of work of efficiency-motivated methods, focuses on avoiding full retraining and uses local curvature restoration [ 12 ] , which was later extended to approximate curvature restoration [ 22 , 1 , 29 , 33 , 9 ] . The latest recovery-focused methods aim to repair the pruned model directly, using a small calibration set and fast local approximations of the pruning damage based on the networks’ hidden states. Two such methods are SNOWS [ 20 ] , which performs layerwise recovery by preserving downstream representations using Hessian approximation, and the very recent CORP [ 35 ] (published in February 2026), which derives a closed-form representation recovery function for pruning in vision transformers. 0.5 0.63 0.69 0.75 0.8 0.85 0 20 40 60 80 Sparsity Level Top-1 Accuracy (%) DeiT3-H (QK+MLP) 0.5 0.63 0.69 0.75 0.8 0.85 Sparsity Level DeiT-B (QK+MLP) 0.3 0.5 0.7 0.75 0.8 Sparsity Level DeiT-B (MLP) 0.3 0.5 0.7 0.75 0.8 Sparsity Level DeiT-S (MLP) 7 Figure 1 : Comparison of the top-1 accuracy results after applying the STARFISH and CORP healing processes on different pruning scopes and sparsity levels for the DeiT-B, DeiT3-H, and DeiT-S architectures trained on ImageNet, both following CORP structured pruning. STARFISH recovery is more successful at all sparsity levels, reaching an even larger advantage as sparsity increases. In this paper, we present the STARFISH (fa ST A ccuracy R ecovery in pruned networks F rom I nternal S tate H ealing) method. The starfish is a marine animal with exceptional healing abilities that can regrow multiple arms even when completely cut off. Similarly, our STARFISH method excels at healing the performance even after the network undergoes aggressive pruning, regardless of the pruning method or type. We show that optimizing the directional alignment of the intermediate representations of the pruned network with those of the original network is a good proxy for accuracy restoration. Our contribution is built on similar objectives that were previously shown to better capture the models’ knowledge in the fields of knowledge distillation, self-supervised and unsupervised learning [ 27 , 28 , 34 , 30 , 3 , 10 , 6 ] , now using it as a simple post-pruning recovery mechanism, empirically showing that this objective is especially effective for high-sparsity pruning. In Figure 1 , we demonstrate this remarkable improvement for four modern transformer-based image classifiers [ 31 , 32 ] at various sparsity levels: our STARFISH method (in pink) is much better than the state-of-the-art recovery method CORP (in brown), especially in the most interesting regime of high sparsity pruning, where we denote the sparsity level as the fraction of weights removed. For the sake of completeness, we include the baseline accuracy when no healing is used (in dashed grey). In more detail, our main contributions are: 1. We present STARFISH, an efficient recovery method based on intermediate-representation alignment, which is independent of pruning types, methods, or network architecture. Building on previous hidden-state alignment research, we empirically show that optimizing it over the calibration set is increasingly more effective at improving test accuracy than direct output-based optimization as the sparsity level increases. 2. We show experimentally, on large image classifiers such as ViT [ 5 ] and MobileNet [ 14 ] , that our method is the first to efficiently restore even 92\% of the original accuracy at the very high sparsity level of 0.85 , in which the pruning process almost completely degrades the original accuracy. 3. For the more modest sparsity of 0.5 across different ViT-based architectures [ 31 , 32 ] , we show that our method outperforms the competing methods by a large factor of up to 31.5 percentage-points improvement in recovering the original network accuracy, reaching up to 99.8\% accuracy retention of the dense model accuracy. 2 Related work Trying to create more efficient machine learning models, the pruning line of work involves the removal of

Read the complete text