Turning the Tide : Cross-Architecture Distillation for Diffusion Large Language Models
Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan · 2026
Read Turning the Tide : Cross-Architecture Distillation for Diffusion Large Language Models by Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26951.txt.
Description
Zhang and three colleagues present TIDE, the first framework for distilling diffusion language models across designs. Teacher and student here differ in design, attention, and tokenizer. TIDAL tunes distillation strength over training progress and diffusion step. CompDemo enriches teacher context via complementary mask splitting. Reverse CALM inverts chunk-level likelihood matching. Distilling large teachers into a much smaller student beats the baseline across eight benchmarks, with code scores well above the autoregressive baseline.
Catalog record
| CREATOR | Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50450 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26951