Sparse Training of Neural Networks based on Multilevel Mirror Descent

Yannick Lunk, Sebastian J. Scott, Leon Bungert · 2026

Read Sparse Training of Neural Networks based on Multilevel Mirror Descent by Yannick Lunk, Sebastian J. Scott, Leon Bungert (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03535.txt.

Description

Training a neural network that stays sparse from the start could cut computing costs sharply, if the accuracy held. Yannick Lunk and 2 co-authors propose a dynamic sparse training algorithm in a 2026 paper. It builds on linearized Bregman iterations, a mirror descent method that naturally encourages sparsity. The challenge is exploring sparse structures without losing the sparsity itself.

Their algorithm alternates between static phases and dynamic updates of the sparsity pattern, and adaptively freezes the network structure in between. A multilevel optimization framing supplies convergence guarantees. On standard benchmarks the method produces highly sparse, accurate models. Theoretical operation counts fall to about 6 percent of dense training, and a sparsity-aware CPU implementation halves training time.

Catalog record

CREATORYannick Lunk, Sebastian J. Scott, Leon Bungert
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03535

Browse the full library · About the Tower of Babel