How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs
Emily Dent, Jared Tanner · 2026
Read How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs by Emily Dent, Jared Tanner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05779.txt.
Description
Deep networks train best when signals neither explode nor vanish across layers, and a body of theory called edge of chaos guides the choice of starting weights. Emily Dent and Jared Tanner study a less explored case: activations that induce heavy sparsity, zeroing a large region of values near the origin. For near linear activations, the classic theory pushes the layer variance toward zero with depth.
The authors prove a new phenomenon for the sparse case: starting points that lead to larger fixed Gaussian processes are actually better for training stability. That finding informs a simple new initialization strategy for fully connected and convolutional networks. With it, they train networks whose hidden layers run at up to ninety percent sparsity. The result matters for efficient computing, since sparse networks promise cheaper inference. In the sparse regime, shrinking variance is the wrong target.
Catalog record
| CREATOR | Emily Dent, Jared Tanner |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05779