TextLDM: Language Modeling with Continuous Latent Diffusion
Jiaxiu Jiang Jingjing Ren Wenbo Li Bo Wang, Haoze Sun Yijun Yang Jianhui Liu Yanbing Zhang Shenghe Zheng, Yuan Zhang Haoyang Huang Nan Duan Wangmeng Zuo · 2026
Read TextLDM: Language Modeling with Continuous Latent Diffusion by Jiaxiu Jiang Jingjing Ren Wenbo Li Bo Wang, Haoze Sun Yijun Yang Jianhui Liu Yanbing Zhang Shenghe Zheng, Yuan Zhang Haoyang Huang Nan Duan Wangmeng Zuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07748.txt.
Introduction
Jiaxiu Jiang and twelve co-authors bring the visual diffusion recipe to language modeling. Diffusion transformers trained with flow matching in a learned latent space have unified image and video generation, but text is discrete, so the transfer is not obvious. Their system, TextLDM, first maps tokens into continuous latents with a transformer autoencoder.
The key finding is that reconstruction fidelity alone does not make latents good for generation; aligning them with a frozen pretrained language model's representations proved critical. A standard diffusion transformer then performs flow matching in that latent space, architecturally identical to its visual counterpart. Trained from scratch on OpenWebText2, TextLDM substantially beat earlier diffusion language models and matched GPT-2 under the same settings.
Catalog record
| CREATOR | Jiaxiu Jiang Jingjing Ren Wenbo Li Bo Wang, Haoze Sun Yijun Yang Jianhui Liu Yanbing Zhang Shenghe Zheng, Yuan Zhang Haoyang Huang Nan Duan Wangmeng Zuo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TextLDM: Language Modeling with Continuous Latent Diffusion Jiaxiu Jiang Jingjing Ren Wenbo Li Bo Wang, Haoze Sun Yijun Yang Jianhui Liu Yanbing Zhang Shenghe Zheng, Yuan Zhang Haoyang Huang Nan Duan Wangmeng Zuo Abstract Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding. 1 Introduction Central to multimodal modeling is the pursuit of a unified framework capable of seamlessly generating both textual and visual content. On the visual side, Diffusion Transformers (DiT) ( 26 ) trained with flow matching ( 19 ; 21 ) in a VAE latent space ( 29 ) have already unified image and video generation ( 7 ; 34 ) , establishing a dominant recipe: continuous latent space, DiT backbone, flow matching objective, classifier-free guidance (CFG) ( 13 ) , and carefully designed timestep schedules ( 7 ) . While some autoregressive (AR) methods ( 33 ; 35 ) have attempted to unify understanding and generation within a discrete token-based paradigm, the prevailing excellence of diffusion models in the visual domain still leaves a methodological gap in language modeling. If this same architecture could also perform language modeling effectively, it would provide a concrete foundation for unified multimodal generation and understanding. As illustrated in Figure 1 , language generation has traditionally been dominated by the AR paradigm, whereas visual generation has converged toward continuous diffusion modeling. Rather than debating the superiority of one paradigm over the other, this paper explores the feasibility of extending the successful visual diffusion recipe to text generation. We propose TextLDM, which instantiates language modeling within the DiT framework with minimal architectural modification. A Transformer-based VAE ( TextVAE ) maps each discrete token to a continuous latent vector, and a standard DiT ( TextDiT )—architecturally identical to its visual counterpart—performs flow matching in this latent space. Critically, as shown in Figure 2 , this approach in principle offers a distinct advantage in inference efficiency, providing a more constant-time generation profile. The central challenge we encounter is not in the diffusion backbone, but in the latent representation. Text is inherently discrete, and a VAE trained solely for token reconstruction can achieve near-perfect accuracy yet produce latents poorly suited for conditional denoising. Our ablations confirm this: configurations with virtually identical reconstruction accuracy may yield substantially different generation quality. The key bottleneck is representation effectiveness —whether the continuous latents support the downstream diffusion process—rather than reconstruction fidelity alone. Figure 1 : Comparison of language generation paradigms. Each sub-figure illustrates the generation process from left to right . (a) Autoregressive (AR) models generate tokens one by one in sequential order. (b) Discrete diffusion (e.g., LLaDA ( 24 ) ) iteratively unmasks tokens, where gray blocks denote <mask> tokens. (c) TextLDM (ours) performs flow matching in a learned continuous latent space, progressively denoising random noise into valid language representations. The uniform-colored blocks at the bottom represent the condition (context) latents. Figure 2 : Inference efficiency: the number of function evaluations (NFE) for AR models grows linearly with sequence length, while TextLDM achieves length-invariant NFE over a broad operating range (e.g., up to 1024 tokens). To address this, we introduce Representation Alignment (REPA) ( 38 ) , originally proposed for image DiT training, to the text VAE. By aligning the VAE encoder’s features with those of a frozen pretrained language model (Qwen3-1.7B ( 37 ) ), REPA shapes the latent geometry to be more amenable to diffusion-based generation, yielding substantial improvements in downstream quality without affecting reconstruction. We train all components from scratch on OpenWebText2 ( 8 ) and evaluate on text continuation across four benchmarks. TextLDM substantially outperforms prior continuous and discrete diffusion language models and matches GPT-2 baselines under the same settings. Comprehensive ablations validate that visual diffusion components—logit-normal scheduling and CFG—can seamlessly transfer to the language modeling. Our contributions are: • We propose TextLDM, which transfers the visual latent diffusion recipe (VAE + DiT + flow matching + CFG) to language modeling with minimal modification, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding. The entire system is trained from scratch without pretrained encoders or decoders. • We identify representation effectiveness as the key bottleneck for latent text diffusion, and introduce REPA-enhanced TextVAE to produce continuous representations suited for conditional denoising, substantially improving generation quality without affecting reconstruction. • Extensive experiments demonstrate that TextLDM achieves state-of-the-art results among diffusion language models on text continuation benchmarks and matches autoregressive baselines under identical settings. Ablations validate the effectiveness of each transferred visual diffusion component. 2 Related Work Diffusion Models for Visual Generation. Diffusion models ( 12 ; 32 ) have been unified with flow matching ( 21 ; 19 ) and extended to latent spaces ( 29 ) . Diffusion Transformers (DiT) ( 26 ) enabled scalable architectures, and the recipe of flow matching + VAE + DiT has become the standard for visual generation ( 7 ; 4 ) . Diffusion Language Models. Diffusion language models (see Figure 1 ) can be categorized into continuous and discrete approaches. Continuous methods ( 16 ; 18 ; 36 ; 11 ; 5 ; 10 ) apply diffusion in embedding or simplex spaces. LD4LG ( 22 ) and COSMOS ( 23 ) use latent diffusion with pretrained encoders or compressed latent spaces. Discrete methods ( 2 ; 30 ; 24 ) define diffusion over tokens directly. Block Diffusion ( 1 ) denoises token blocks, and CALM ( 31 ) augments AR with a diffusion head for chunk generation. Our method differs by performing flow matching in a learned continuous latent space with a standard DiT, requiring no pretrained encoder/decoder. Unlike chunk-based methods, we generate an entire passage in a single diffusion pass. Variational Autoencoders for Text. Prior text VAEs ( 14 ; 15 ; 20 ) typically rely on pretrained components or autoregressive decoders. Our TextVAE is trained from scratch with a non-autoregressive decoder and enhanced by REPA ( 38 ) , which was originally proposed to align DiT representations with pretrained vision encoders for image generation. We adapt REPA to align the VAE encoder with a frozen language model, ensuring the latent space is highly structured and semantically rich, which significantly enhances representation quality. 3