ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning
Zhendong Mi, Zhenglun Kong, Geng Yuan, Shaoyi Huang · 2025
Read ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning by Zhendong Mi, Zhenglun Kong, Geng Yuan, Shaoyi Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21987.txt.
Description
Zhendong Mi and three co-authors prune large language models with two activation-based signals, in work from 2025. Cutting model size saves memory and compute, but blunt pruning wrecks quality and slow calibration eats the savings. Their ACE method scores parts by how much pruning bends the output direction, using cosine similarity of activations, and by how much spread the outputs lose.
Keeping both signals preserves meaning while letting calibration run on shorter inputs. On well-known model families the method cut perplexity by up to eighteen percent against rivals and shortened pruning time by up to sixty-three percent. Speed and quality arrive together here, the authors argue.
Catalog record
| CREATOR | Zhendong Mi, Zhenglun Kong, Geng Yuan, Shaoyi Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21987