Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

Fangyu Ding, Ding Ding, Sijin Chen, Kaibo Wang, Peng Xu, Zijin Feng, Haoli Bai, Kai Han, Youliang Yan, Binhang Yuan, Jiacheng Sun · 2026

Read Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes by Fangyu Ding, Ding Ding, Sijin Chen, Kaibo Wang, Peng Xu, Zijin Feng, Haoli Bai, Kai Han, Youliang Yan, Binhang Yuan, Jiacheng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23507.txt.

Introduction

Ding and ten co-authors replace masking in diffusion language models with deletion and insertion. Masked models waste compute on mask and pad tokens that carry no signal. Their DID models treat token deletion and insertion as the diffusion steps themselves. That frees length to vary with no padding at all. Insertion lets the model fix its slips mid-generation. A score-based trainer rates insert moves, backed by a dynamic program. Across settings, DID beats masked and insertion rivals on quality and speed, with no tuning.

Catalog record

CREATORFangyu Ding, Ding Ding, Sijin Chen, Kaibo Wang, Peng Xu, Zijin Feng, Haoli Bai, Kai Han, Youliang Yan, Binhang Yuan, Jiacheng Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE203623
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23507

Browse the full library · About the Tower of Babel