Reasoning with Latent Tokens in Diffusion Language Models
Andre He, Sean Welleck, Daniel Fried · 2026
Read Reasoning with Latent Tokens in Diffusion Language Models by Andre He, Sean Welleck, Daniel Fried (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03769.txt.
Description
Diffusion language models reason about every undecided token at once, which costs computation but buys coherence. Andre He and 2 co-authors investigate that trade in a 2026 paper. These models are trained to predict a joint distribution over all unknown tokens, including ones not decoded in the current step. Ablating that joint prediction speeds inference while hurting quality.
The authors interpret the undecoded tokens as latent tokens, hidden reasoning capacity. Modulating their number trades inference speed against sample quality smoothly. They then add latent tokens to ordinary autoregressive models through an auxiliary multi-token prediction objective. The augmented models improve a lot on exactly the planning and global-coherence tasks where autoregressive models usually struggle.
Catalog record
| CREATOR | Andre He, Sean Welleck, Daniel Fried |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03769