Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization

Yu-Han Wu, Pierre Marion, Gérard Biau, Claire Boyer · 2025

Read Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization by Yu-Han Wu, Pierre Marion, Gérard Biau, Claire Boyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03435.txt.

Description

Wu, Marion, Biau, and Boyer explained in twenty twenty-five why diffusion image models do not simply memorize their training data. Diffusion models learn by denoising, and the perfect solution to that task would copy training samples exactly. Yet real models show only moderate copying, even without explicit safeguards.

The authors credit large learning rates, the step size used in training. In the small-noise regime, the perfect score function is highly irregular. They prove a network trained with big enough steps cannot settle stably near that perfect solution. So the learned score stays safely distant, limiting memorization. Their proof uses one-dimensional data and two-layer networks for tractability. Experiments confirm the learning rate's key role beyond that simple setting.

Catalog record

CREATORYu-Han Wu, Pierre Marion, Gérard Biau, Claire Boyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03435

Browse the full library · About the Tower of Babel