Reasoning with Latent Tokens in Diffusion Language Models

Andre He, Sean Welleck, Daniel Fried · 2026

Read Reasoning with Latent Tokens in Diffusion Language Models by Andre He, Sean Welleck, Daniel Fried (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03769.txt.

Description

Diffusion language models reason about every undecided token at once, which costs computation but buys coherence. Andre He and 2 co-authors investigate that trade in a 2026 paper. These models are trained to predict a joint distribution over all unknown tokens, including ones not decoded in the current step. Ablating that joint prediction speeds inference while hurting quality.

The authors interpret the undecoded tokens as latent tokens, hidden reasoning capacity. Modulating their number trades inference speed against sample quality smoothly. They then add latent tokens to ordinary autoregressive models through an auxiliary multi-token prediction objective. The augmented models improve a lot on exactly the planning and global-coherence tasks where autoregressive models usually struggle.

Catalog record

CREATORAndre He, Sean Welleck, Daniel Fried
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03769

Browse the full library · About the Tower of Babel