Turning the Tide : Cross-Architecture Distillation for Diffusion Large Language Models

Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan · 2026

Read Turning the Tide : Cross-Architecture Distillation for Diffusion Large Language Models by Gongbo Zhang, Wen Wang, Ye Tian, Li Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26951.txt.

Description

Zhang and three colleagues present TIDE, the first framework for distilling diffusion language models across designs. Teacher and student here differ in design, attention, and tokenizer. TIDAL tunes distillation strength over training progress and diffusion step. CompDemo enriches teacher context via complementary mask splitting. Reverse CALM inverts chunk-level likelihood matching. Distilling large teachers into a much smaller student beats the baseline across eight benchmarks, with code scores well above the autoregressive baseline.

Catalog record

CREATORGongbo Zhang, Wen Wang, Ye Tian, Li Yuan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50450
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26951

Browse the full library · About the Tower of Babel