Grokking of Diffusion Models: Case Study on Modular Addition

Joon Hyeok Kim Yonghyun Park Mattis Dalsætra Østby Jiatao GuDepartment of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia, PA 19104, USA {hozy, park19, mdals, jgu32}@seas.upenn.edu · 2026

Read Grokking of Diffusion Models: Case Study on Modular Addition by Joon Hyeok Kim Yonghyun Park Mattis Dalsætra Østby Jiatao GuDepartment of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia, PA 19104, USA {hozy, park19, mdals, jgu32}@seas.upenn.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17673.txt.

Description

Kim, Park, Østby, and Gu give a careful account of diffusion-model learning, using grokking in an image-based modular addition setting as a testbed. In a single-image regime, the model encodes inputs as cyclic codes and joins them at the attention-feedforward interface, finding trig identities. In a varied-image regime with high variety, step-by-step sampling splits the task into math and visual polish phases, with the switch tracked by an internal frequency-entropy signal. Overall, the study explains diffusion-model learning from inside.

Catalog record

CREATORJoon Hyeok Kim Yonghyun Park Mattis Dalsætra Østby Jiatao GuDepartment of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia, PA 19104, USA {hozy, park19, mdals, jgu32}@seas.upenn.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52350
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17673

Browse the full library · About the Tower of Babel