Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, Gianni Franchi · 2025
Read Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers by Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, Gianni Franchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18358.txt.
Introduction
Firas Gabetni and his five co-authors address a practical worry about neural networks: knowing when a prediction might be wrong. Running many full copies of a model gives good uncertainty estimates, but it costs too much memory and time for large models. Their answer, Hydra Ensembles, builds varied members by pruning attention heads, the parts of a transformer that weigh which inputs matter most.
The pruned members are then merged through a new grouped attention design, so the combined model runs almost as fast as a single network and needs no training from scratch. Careful study showed that careless pruning can damage how well confidence matches reality, while the Hydra design keeps that balance. Across image and text tasks, the method matched or beat full ensembles, and it led on zero-shot image sorting as well.
Catalog record
| CREATOR | Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, Gianni Franchi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, Gianni Franchi Abstract Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles , an efficient transformer-based ensemble that prunes attention heads to create diverse members and merges them via a new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Experiments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, in zero-shot classification on ImageNet-1k, our approach surpasses state of the art methods, even without requiring additional training. 1 Introduction Deep neural networks (DNNs) excel in vision, language, and multimodal tasks, yet are prone to making overconfident errors ( Hein et al. 2019 ) . This unreliability in predictions is especially concerning in safety-critical domains such as healthcare and autonomous driving, underscoring the importance of studying Uncertainty Quantification (UQ) in DNNs ( Gawlikowski et al. 2023 ) . Several UQ approaches have been explored, including evidential methods ( Sensoy et al. 2018 ) , conformal prediction ( Mollaali et al. 2025 ) , Bayesian inference ( Kendall & Gal 2017 ) , and ensemble methods ( Lakshminarayanan et al. 2017 ; Laurent et al. 2023 ) . Currently, the most reliable method for UQ is Deep Ensembles ( Lakshminarayanan et al. 2017 ) that aggregates predictions from multiple independently trained models. Although highly accurate, Deep Ensembles is extremely costly, as it requires multiple rounds of pre-training and fine-tuning, storing several checkpoints, and performing as many number of forward passes as the number of models, making it both slow and memory-intensive, especially for foundation models (e.g, CLIP ( Radford et al. 2021 ) , BERT ( Devlin et al. 2019 ) ). Several efficient alternatives exist, such as MC Dropout ( Gal & Ghahramani 2016 ) , MIMO ( Havasi et al. 2020 ) , BatchEnsemble ( Wen et al. 2020 ) , and Packed Ensembles ( Laurent et al. 2023 ) , which lower computational costs by reusing weights or sharing parameters. However, they still require full pre-training, and scaling them to transformer-based ( Vaswani et al. 2017 ; Dosovitskiy et al. 2020 ) large-scale foundation models is computationally expensive. To still leverage the power of Deep Ensembles for UQ while reducing the inference cost, a naive approach is pruning or discarding unimportant weights of each constituent model. Contrary to conventional wisdom, in this work we show that commonly used pruning methods ( Molchanov et al. 2019 ; He et al. 2020 ) , despite their ability to preserve accuracy, can, in fact, harm calibration and lead to unreliable predictions. We prove theoretically and empirically under which conditions pruning degrades performance (predictive uncertainty), and consequently raise a pertinent question: How can we leverage ensembles of transformers for efficient and effective UQ? Motivated by this challenge, we develop a framework for ensembling pruned transformers for efficient and effective UQ, which is amenable to large-scale models like CLIP and BERT. In detail, our framework, Hydra Ensembles 1 1 1 In Greek mythology, the Hydra is a serpent-like creature with multiple heads, and it is famously known for its regenerative ability: when one head is severed, two more grow in its place. , aims to preserve the diversity of multiple models, like Deep Ensembles, while maintaining the inference cost close to a single model. Unique to our approach, we generate diverse models via pruning attention heads from a single pre-trained transformer model. These pruned subnetworks are then merged into a single model using Grouped Fully Connected (GFC) layers ( Lafage et al. 2025a ) . Importantly, unlike previous approaches ( Liu et al. 2021 ; Le et al. 2020 ) , Hydra Ensembles does not require training each member from scratch, and can even operate without fine-tuning, offering computational advantages both at training and inference. We evaluate our proposed Hydra Ensembles on three different classification tasks: image classification with ViT ( Dosovitskiy et al. 2020 ) , text classification with BERT ( Devlin et al. 2019 ) , and zero-shot classification with OpenCLIP-ViT ( Ilharco et al. 2021 ) . Our experimental results demonstrate that Hydra Ensembles is competitive to Deep Ensembles both in terms of accuracy and calibration metrics, while greatly reducing the inference costs. Notably, on zero-shot ImageNet-1K OOD benchmarks, Hydra Ensembles surpasses the state of the art ViLU ( Lafon et al. 2025 ) method (which requires training) by +1.3 AUROC, –3.5 FPR95, and +4 AUPR (Table 3 ). Our main contributions are: (i) We investigate the impact of pruning transformer-based models on UQ, and demonstrate both theoretically and empirically that naive pruning can lead to poorly calibrated uncertainty (Section 3.2 ). (ii) We introduce Hydra Ensembles , the first pruning framework specifically designed for UQ in transformer-based large-scale models (Section 4 ). (iii) We show that Hydra Ensembles delivers uncertainty estimates that are comparable to Deep Ensembles, while significantly reducing computational costs both during training and inference (Section 5 ). 2 Related Work Transformers and Uncertainty Quantification. Estimating epistemic uncertainty in DNN is challenging. A common approach is to approximate the intractable posterior distribution over the model’s weights. Bayesian methods such as Variational Inference ( Graves 2011 ; Ranganath et al. 2014 ) or Markov Chain Monte Carlo (MCMC) ( Chen et al. 2014 ; Neal 2012 ) are theoretically sound but often too computationally expensive to scale to large transformer models. Deep Ensembles ( Lakshminarayanan et al. 2017 ) remain the gold standard for accuracy and calibration, but their cost grows linearly with the number of models, making them impractical for very large architectures. Lighter alternatives such as MC Dropout ( Gal & Ghahramani 2016 ) , MIMO ( Havasi et al. 2020 ) , BatchEnsemble ( Wen et al. 2020 ) , MaskEnsembles ( Durasov et al. 2021 ) , and LoRA-Ensemble ( Mühlematter et al. 2024 ; Wang et al. 2023 ) reduce computation by sharing most of the backbone, but this limits the independence of ensemble members. Packed-Ensembles ( Laurent et al. 2023 ) maintain stricter independence but reduce per-member representation capacity. For CLIP ( Radford et al. 2021 ) and other Large vision-language models, post-hoc methods like BayesVLM ( Baumann et al. 2024 ) (Laplace approximation on the last layers) and ViLU ( Lafon et al. 2025 ) (adding a lightweight error-prediction head) have been explored. While convenient, post-hoc approaches generally yield weaker uncertainty estimates. Overall, ensemble-style methods provide stronger uncertainty but either require costly retraining or sacrifice diversity for efficiency. Hydra Ensembles offers a balance: it preserves member diversity, avoids retraining, and achieves near single-model inference cost. Network Pruning. Neural network pruning is widely used to reduce model size and computational cost while retaining predictive accuracy. Pruning removes less important weights to preserve accuracy ( Chauvin 1988