ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning

Zhendong Mi, Zhenglun Kong, Geng Yuan, Shaoyi Huang · 2025

Read ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning by Zhendong Mi, Zhenglun Kong, Geng Yuan, Shaoyi Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21987.txt.

Description

Zhendong Mi and three co-authors prune large language models with two activation-based signals, in work from 2025. Cutting model size saves memory and compute, but blunt pruning wrecks quality and slow calibration eats the savings. Their ACE method scores parts by how much pruning bends the output direction, using cosine similarity of activations, and by how much spread the outputs lose.

Keeping both signals preserves meaning while letting calibration run on shorter inputs. On well-known model families the method cut perplexity by up to eighteen percent against rivals and shortened pruning time by up to sixty-three percent. Speed and quality arrive together here, the authors argue.

Catalog record

CREATORZhendong Mi, Zhenglun Kong, Geng Yuan, Shaoyi Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21987

Browse the full library · About the Tower of Babel