How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs

Emily Dent, Jared Tanner · 2026

Read How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs by Emily Dent, Jared Tanner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05779.txt.

Description

Deep networks train best when signals neither explode nor vanish across layers, and a body of theory called edge of chaos guides the choice of starting weights. Emily Dent and Jared Tanner study a less explored case: activations that induce heavy sparsity, zeroing a large region of values near the origin. For near linear activations, the classic theory pushes the layer variance toward zero with depth.

The authors prove a new phenomenon for the sparse case: starting points that lead to larger fixed Gaussian processes are actually better for training stability. That finding informs a simple new initialization strategy for fully connected and convolutional networks. With it, they train networks whose hidden layers run at up to ninety percent sparsity. The result matters for efficient computing, since sparse networks promise cheaper inference. In the sparse regime, shrinking variance is the wrong target.

Catalog record

CREATOREmily Dent, Jared Tanner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05779

Browse the full library · About the Tower of Babel