Towards the Connection between Activation Sparsity and Flat Minima

Ze Peng, Jian Zhang, Lei Qi, Yang Gao, Yinghuan Shi* · 2026

Read Towards the Connection between Activation Sparsity and Flat Minima by Ze Peng, Jian Zhang, Lei Qi, Yang Gao, Yinghuan Shi* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25612.txt.

Description

Peng and colleagues explain why activation sparsity appears in the MLP blocks of trained Transformers. Earlier accounts blamed data or fitting, or leaned on strong assumptions that fail for deep models trained over many steps. They show sparsity equals augmented flatness divided by input norm times activation gradient. This ratio falls during training. They define derivative sparsity, which matches activation sparsity under ReLU but allows pruning in backward passes and stays more stable. Three tweaks that shrink the ratio yield much sparser activations on ImageNet-1K and C4 tests.

Catalog record

CREATORZe Peng, Jian Zhang, Lei Qi, Yang Gao, Yinghuan Shi*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE142706
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25612

Browse the full library · About the Tower of Babel