Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers

Pengqi Lu · 2026

Read Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers by Pengqi Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06169.txt.

Description

Pengqi Lu identifies a silent collapse mode that threatens very deep diffusion transformers. As these networks grow to hundreds of layers, token representations can homogenize while training still looks stable on the surface. Lu names the trigger event Mean Mode Screaming, a mean coherent backward shock that opens deep residual branches and pushes the network into a mean dominated state.

An exact split of the gradients into mean coherent and centered parts explains the collapse. Attention logit gradients also fade once values homogenize, compounding the problem. Lu's fix, Mean-Variance Split Residuals, gives the centered update its own gain and replaces the trunk mean with a leaky version. On a 400 layer model it prevented the crash that killed the baseline, and a 1,000 layer run stayed trainable.

Catalog record

CREATORPengqi Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06169

Browse the full library · About the Tower of Babel