A Theory of Generalization in Deep Learning

Elon Litman, Gabe Guo · 2026

Read A Theory of Generalization in Deep Learning by Elon Litman, Gabe Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01172.txt.

Introduction

Elon Litman and Gabe Guo present a new theory for why deep neural networks can learn from training data and still do well on new data. Their account centers on the neural tangent kernel, a tool that tracks how a network changes as it trains.

In their picture, the kernel splits the output space into a signal part and a vast noise part. Real signal is learned fast, while odd details of single examples are stored slowly in the noise part, where tests cannot see them. The theory ties together known puzzles such as benign overfitting, double descent, and grokking. It also yields a risk score from one training run alone, with no held-out data, and a tweak to the Adam optimizer that speeds up grokking and cuts memorization.

Catalog record

CREATORElon Litman, Gabe Guo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE179 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01172

Browse the full library · About the Tower of Babel