Learning Generation Orders for Masked Discrete Diffusion Models via Variational Inference
David Fox, Sam Bowyer, Song Liu, Laurence Aitchison, Raul Santos-Rodriguez, Mengyue Yang · 2026
Read Learning Generation Orders for Masked Discrete Diffusion Models via Variational Inference by David Fox, Sam Bowyer, Song Liu, Laurence Aitchison, Raul Santos-Rodriguez, Mengyue Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23968.txt.
Description
David Fox and five co-authors propose a variational inference framework for learning generation orders in masked discrete diffusion models. The 2026 paper addresses a central tension in such models. Parallel token generation promises speed, but the order in which masked tokens are filled shapes sample quality. Current methods rely on fixed heuristics.
The authors parameterize the approximate posterior over generation orders in a way that supports parallelism and efficient sampling during training. Early tests on a math reasoning dataset show competitive performance against heuristic sampling in the highly parallel regime, reaching notably higher accuracy than standard rivals in only a handful of generation steps. The work opens a principled route to learned, parallel-friendly generation orders, with further experiments needed to map its full gains.
Catalog record
| CREATOR | David Fox, Sam Bowyer, Song Liu, Laurence Aitchison, Raul Santos-Rodriguez, Mengyue Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-23968