How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks
William T. Redman, Zhangyang Wang, Alessandro Ingrosso, Sebastian Goldt · 2024
Read How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks by William T. Redman, Zhangyang Wang, Alessandro Ingrosso, Sebastian Goldt (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06545.txt.
Description
Pruning a trained network by repeatedly cutting its smallest weights, then retraining, famously leaves lean subnetworks that still perform. William Redman, Zhangyang Wang, Alessandro Ingrosso, and Sebastian Goldt ask in 2024 why that pruning carves out local receptive fields, the patch-wise wiring of real visual brains, inside plain fully connected networks.
Their guess: pruning feeds on itself, steadily making the network's internal patterns less Gaussian, and skewed patterns favor local wiring. A new cavity method let them weigh how each single weight bends the pattern statistics. They show skewed input statistics are required for the fields to appear, and that pruning systematically deepens the skew. The loop explains a small mystery of the lottery ticket literature.
Catalog record
| CREATOR | William T. Redman, Zhangyang Wang, Alessandro Ingrosso, Sebastian Goldt |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06545