How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov, Nikita Gushchin, Ilya Koziev, Alexander Korotin, Dmitry Vetrov · 2026

Read How to Train Your Latent Diffusion Language Model Jointly With the Latent Space by Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov, Nikita Gushchin, Ilya Koziev, Alexander Korotin, Dmitry Vetrov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07933.txt.

Description

Viacheslav Meshchaninov and six co-authors train the latent space and the diffusion model together for text generation. Latent diffusion draws interest for language because it denoises whole sequences in parallel, but everything depends on building the right latent space, which is usually fixed in advance by a separate autoencoder.

Their LDLM reshapes a pretrained language model's internal forms with a trainable encoder, and trains encoder, diffusion model, and decoder jointly. Naive joint training fails, producing a poor model. Their recipe adds a mean-squared decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and noise at the decoder input, with ablations showing each piece matters. On OpenWebText and LM1B it beat existing discrete and continuous language models while writing several times faster.

Catalog record

CREATORViacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov, Nikita Gushchin, Ilya Koziev, Alexander Korotin, Dmitry Vetrov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07933

Browse the full library · About the Tower of Babel