Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks

Jiahao Lu · 2025

Read Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks by Jiahao Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22192.txt.

Description

Jiahao Lu investigates what common training tricks really do to a neural network's internal preferences for pattern frequencies. Weight decay and dropout are standard regularizers, yet their effects on which frequencies a network learns were poorly understood. Lu builds a visual diagnostic system that tracks how weight frequencies evolve during training, and defines a Spectral Suppression Ratio to measure each regularizer's low-pass filtering strength.

Tests with ResNet-18 on CIFAR-10 show weight decay suppresses high-frequency energy buildup more than threefold compared with unregularized training. A trade-off emerges: weight-decayed models suffer more under broad Gaussian noise but resist high-frequency information loss, beating baselines by over six percent on blurred images. The results present generalization as a strong bias toward low-frequency structure.

Catalog record

CREATORJiahao Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22192

Browse the full library · About the Tower of Babel