Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?

Tom Jacobs, Chao Zhou, Rebekka Burkholz · 2025

Read Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? by Tom Jacobs, Chao Zhou, Rebekka Burkholz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12883.txt.

Description

Jacobs, Zhou, and Burkholz study how deliberate penalties during training interact with the hidden preferences a model develops on its own. Large models often generalize well for reasons traced to implicit bias, the direction training naturally pushes them. Practitioners also add explicit regularization such as weight decay, and the two forces act together.

The authors fold regularization into the mirror flow picture of training and identify three lasting effects: it shifts where solutions settle, changes the type of bias, and shrinks its range. They show the ideas at work on sparse coding, matrix sensing, attention layers, and LoRA tuning. One practical tip follows: turning weight decay off partway through training can improve generalization.

Catalog record

CREATORTom Jacobs, Chao Zhou, Rebekka Burkholz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12883

Browse the full library · About the Tower of Babel