Elastic ViTs from Pretrained Models without Retraining

Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G.M. Snoek, Yuki M. Asano · 2025

Read Elastic ViTs from Pretrained Models without Retraining by Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G.M. Snoek, Yuki M. Asano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17700.txt.

Introduction

Walter Simoncini, with Michael Dorkenwald and three co-authors, makes vision models adjustable after training in a 2025 arXiv paper. Vision foundation models ship in only a few fixed sizes. Real deployments face varied compute budgets, so a fixed size often wastes power or starves the task. Their method, called SnapViT, prunes a trained Vision Transformer in one shot. Pruning means removing unneeded network parts.

The pruning scores parts using gradient data plus structure links across the network. An evolutionary algorithm approximates those links cheaply. No labeled data and no retraining are required. The method even works for models without a classification head. Across four model families, it beat prior pruning methods at many sparsity levels. Building an adjustable model took under five minutes on one GPU.

Catalog record

CREATORWalter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G.M. Snoek, Yuki M. Asano
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17700

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Elastic ViTs from Pretrained Models without Retraining Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G.M. Snoek, Yuki M. Asano Abstract Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: Single-shot network approximation for pruned Vision Transformers, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining-free. Experiments on DINO, SigLIPv2, DeIT, and AugReg models demonstrate superior performance over state-of-the-art methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: https://elastic.ashita.nl/ 1 Introduction † † footnotetext: * Denotes equal contribution. Recent advances in model architectures and training recipes have enabled the training of vision foundation models with several billion parameters [ 12 , 53 , 16 , 60 ] , achieving state-of-the-art results across a wide range of tasks. However, these models must operate under strict compute, latency, and cost constraints when deployed in real-world settings. Yet, only a limited set of model sizes e.g., 21M, 29M, 86M, 300M, 840M and 6.7B parameter vision transformers (ViTs) for the DINOv3 family [ 15 , 59 ] are made available, forcing users to select the largest model that still fits their requirements, a choice that can often be sub-optimal. Traditionally, challenges related to model flexibility have been tackled using knowledge distillation [ 27 ] . However, this strategy mandates a predetermined target architecture and relies on often non-public pretraining datasets, which can in turn undermine robustness and limit flexibility. In parallel, methods for elastic inference [ 14 , 7 , 29 , 70 ] have emerged to enable dynamic selection among multiple sub-networks at inference time. Yet, these methods necessitate networks designed with a predefined structure, such as nested Matryoshka [ 34 ] , and require such structures to be present during pretraining. This dependency restricts their applicability to existing or proprietarily pretrained models. An attractive alternative is structured pruning [ 37 , 23 , 24 ] , a technique that reduces memory and computational requirements, enabling models to be adapted to diverse deployment settings. Despite their promise, most pruning techniques are tailored to specific compute constraints and tasks [ 35 , 83 ] and typically require retraining, leaving a significant gap for developing universally adaptable models. To bridge this gap, we propose a novel structured pruning method, SnapViT , that operates in a post-pretraining setting and enables elastic inference. By that, we can extract a continuum of sub-networks from a single pretrained model, thereby enabling users to precisely tailor state-of-the-art models to their computational budget and task. To this end, we introduce a prunability score that enables effective pruning across varying sparsity levels in a single shot that is extremely fast (less than five minutes on one A100 GPU). This score facilitates selective pruning of transformer components (e.g., row-column combinations within feed-forward blocks [ 14 ] ) and larger structures, such as entire attention heads. Specifically, our prunability score is composed of two terms: (i) a gradient-based component, in line with previous works [ 37 , 35 , 80 ] , and (ii) a novel cross-network correlation score that approximates parameter sensitivity, as captured by the off-diagonal elements of the Hessian. While prior approaches have largely ignored this component due to its quadratic scaling with the number of parameters, we propose an efficient approximation using an evolutionary algorithm to estimate these correlations. Moreover, by basing the gradient term on a self-supervised loss, our method works on any pretrained model without requiring a classification head and generalizes well across diverse downstream target datasets. We evaluate pruned models across eight datasets, including ImageNet-1k, via k-nearest neighbor classification (k-NN), linear probing, and linear semantic segmentation using small and large ViTs from the DINO [ 9 , 59 ] , AugReg [ 61 ] , DeIT [ 64 , 65 ] , and SigLIPv2 [ 66 ] model families. Our method consistently matches or outperforms state-of-the-art approaches such as LAMP [ 38 ] , the LLM Surgeon [ 67 ] , FPTP [ 35 ] , SparseGPT [ 18 ] , SNIP Magnitude [ 31 ] , and NViT [ 76 ] across various sparsity levels, while generating all sparsities in a single shot. In particular, our method can prune DINOv1 ViT-B/16 to 40% sparsity, accelerating inference by 1.58x while keeping the accuracy degradation below 5%. We also show that our method is compatible with post-pruning weight correction and full fine-tuning, and outperforms or is competitive with state-of-the-art approaches in these setups. Finally, we demonstrate the importance of modeling cross-network correlations through multiple ablations and visualize the sparsity distribution across the network. Our contributions can be summarized as follows: • We introduce an effective, fast pruning strategy for pretrained ViTs, yielding elastic models that can adapt to any computational constraints. • We propose a novel strategy to approximate the off-diagonal components of the Hessian for network structures using a genetic algorithm. • We obtain state-of-the-art performance under considerable pruning without retraining or requiring any labels. 2 Related work Network efficiency To improve model efficiency, several techniques have been proposed, including pruning [ 22 ] , quantization [ 52 ] , and knowledge distillation [ 27 ] . Pruning, in particular, aims at eliminating the “unimportant” bits of the network while preserving model performance. Most pruning research [ 25 , 2 , 71 , 39 ] has focused on CNNs for image classification. Pruning methods can be classified into unstructured [ 22 ] , removing individual weights to yield irregular sparsity and high compression, often requiring specialized hardware to realize speedups, or as structured [ 39 ] , eliminating entire filters, channels, or other structures to enable practical acceleration on standard hardware. Finding the “unimportant” parts of the network has been done based on weight magnitudes [ 38 , 23 , 62 , 44 , 84 , 43 ] activations [ 62 , 81 , 82 ] , gradients [ 77 , 56 ] , or the model’s Hessian [ 37 , 24 , 67 , 68 , 35 , 63 , 45 , 69 ] . The latter is the most accurate, as it accounts for all second-order dependencies [ 37 ] ; however, computing the full Hessian is infeasible. Several tractable approximations have been introduced, such as diagonal [ 37 , 63 ] , block diagonal [ 36 , 18 ] , and block diagonal with K-FAC [ 67 , 68 ] . While these approximations efficiently capture local and intra-layer interactions, they inherently disregard inter-layer dependencies. To overcome this limitation, we introduce a black-box evolutionary algorithm that circumvents the need for explicitly computing the Hessian and can model intra-layer dependencies. Elastic inference

Read the complete text