Layer Collapse in Diffusion Language Models
Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei Liu · 2026
Read Layer Collapse in Diffusion Language Models by Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06366.txt.
Description
Alexander Conzelmann, Albert Catalan-Tatjer, and Shiwei Liu uncover a layer collapse pattern inside diffusion language models. These models rival autoregressive ones, yet their internal activation dynamics were poorly understood until now. In the LLaDA model studied, a few early layers show nearly identical activation patterns. One huge outlier dominates them and persists across long stretches of tokens. The structure looks redundant at first glance.
Removing that outlier wrecks the output, collapsing it into repetitive random token loops, so the apparent redundancy is load bearing. The authors trace the pattern to overtraining rather than undertraining, the reverse of what drives redundancy in autoregressive models. Practical tests follow: the diffusion model survived hard low bit quantization nearly unscathed while a rival autoregressive model collapsed.
Catalog record
| CREATOR | Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06366