Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks
Jiahao Lu · 2025
Read Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks by Jiahao Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22192.txt.
Description
Jiahao Lu investigates what common training tricks really do to a neural network's internal preferences for pattern frequencies. Weight decay and dropout are standard regularizers, yet their effects on which frequencies a network learns were poorly understood. Lu builds a visual diagnostic system that tracks how weight frequencies evolve during training, and defines a Spectral Suppression Ratio to measure each regularizer's low-pass filtering strength.
Tests with ResNet-18 on CIFAR-10 show weight decay suppresses high-frequency energy buildup more than threefold compared with unregularized training. A trade-off emerges: weight-decayed models suffer more under broad Gaussian noise but resist high-frequency information loss, beating baselines by over six percent on blurred images. The results present generalization as a strong bias toward low-frequency structure.
Catalog record
| CREATOR | Jiahao Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22192