Towards the Connection between Activation Sparsity and Flat Minima
Ze Peng, Jian Zhang, Lei Qi, Yang Gao, Yinghuan Shi* · 2026
Read Towards the Connection between Activation Sparsity and Flat Minima by Ze Peng, Jian Zhang, Lei Qi, Yang Gao, Yinghuan Shi* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25612.txt.
Description
Peng and colleagues explain why activation sparsity appears in the MLP blocks of trained Transformers. Earlier accounts blamed data or fitting, or leaned on strong assumptions that fail for deep models trained over many steps. They show sparsity equals augmented flatness divided by input norm times activation gradient. This ratio falls during training. They define derivative sparsity, which matches activation sparsity under ReLU but allows pruning in backward passes and stays more stable. Three tweaks that shrink the ratio yield much sparser activations on ImageNet-1K and C4 tests.
Catalog record
| CREATOR | Ze Peng, Jian Zhang, Lei Qi, Yang Gao, Yinghuan Shi* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 142706 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25612