Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers
Pengqi Lu · 2026
Read Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers by Pengqi Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06169.txt.
Description
Pengqi Lu identifies a silent collapse mode that threatens very deep diffusion transformers. As these networks grow to hundreds of layers, token representations can homogenize while training still looks stable on the surface. Lu names the trigger event Mean Mode Screaming, a mean coherent backward shock that opens deep residual branches and pushes the network into a mean dominated state.
An exact split of the gradients into mean coherent and centered parts explains the collapse. Attention logit gradients also fade once values homogenize, compounding the problem. Lu's fix, Mean-Variance Split Residuals, gives the centered update its own gain and replaces the trunk mean with a leaky version. On a 400 layer model it prevented the crash that killed the baseline, and a 1,000 layer run stayed trainable.
Catalog record
| CREATOR | Pengqi Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06169