Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries

Chris Kolb, Tobias Weber, Bernd Bischl, & David Rügamer · 2025

Read Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries by Chris Kolb, Tobias Weber, Bernd Bischl, & David Rügamer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02496.txt.

Description

Making a neural network sparse usually means penalties that gradient descent cannot chew. Chris Kolb and co-authors take a side door in this 2025 paper. They split each weight into several multiplied factors, so smoothness returns. Ordinary penalties on the factors then act like sparsity penalties on the whole. They call the approach deep weight factorization.

The team proves the equivalence with non-convex sparse regularization and studies how training behaves, finding that the starting values and learning rate need special care. Across architectures and datasets, deep factorization beat its two-factor predecessor and popular pruning methods. The authors frame artificial symmetries in the weights as a tool, not a nuisance.

Catalog record

CREATORChris Kolb, Tobias Weber, Bernd Bischl, & David Rügamer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02496

Browse the full library · About the Tower of Babel