The Implicit Bias of Logit Regularization

Alon Beck, Yohai Bar-Sinai, Noam Levi · 2026

Read The Implicit Bias of Logit Regularization by Alon Beck, Yohai Bar-Sinai, Noam Levi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12039.txt.

Description

Label smoothing and related tricks penalize logits directly, improving calibration and generalization by mechanisms that were poorly understood. Alon Beck and two co-authors analyze a general class of logit regularizers for linear classification, showing they induce an implicit bias: logits cluster around finite per-sample targets. For Gaussian data, or whenever logits cluster tightly enough, that clustering aligns the weight vector exactly with Fisher's Linear Discriminant.

In a signal-plus-noise model, the effect is sharp: the critical sample size falls by half, grokking sets in when noise is tiny, and generalization stays steady as noise rises. The results extend the theory of label smoothing and support a broader class of logit regularization methods. What looked like a calibration hack, the paper suggests, reshapes learning geometry in a principled way.

Catalog record

CREATORAlon Beck, Yohai Bar-Sinai, Noam Levi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12039

Browse the full library · About the Tower of Babel