LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu · 2026

Read LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling by Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11748.txt.

Introduction

Chen and six colleagues present LangFlow, the continuous diffusion model matching discrete diffusion. Continuous diffusion excels at fast images, but prior language models lagged discrete rivals given sparse data. LangFlow links embedding diffusion models to Flow Matching via Bregman divergence. Its innovations are an ODE-based NLL bound, an information-uniform noise schedule motivating a tuned Gumbel scheduler, and revised training with self-conditioning. LangFlow rivals discrete models on perplexity measures, reaching 30.0 on LM1B and 24.6 on OpenWebText, and beats autoregressive baselines on 4 of 7 zero-shot benchmarks.

Catalog record

CREATORYuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100733
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11748

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu Abstract Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow , the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow 1 Introduction Diffusion models ( 45 ; 47 ; 22 ) have achieved remarkable success in generating data with continuous modalities, such as images ( 13 ; 39 ) , videos ( 23 ; 4 ) , molecular structures ( 52 ; 1 ) , and robot behaviors ( 11 ) . A key reason behind this success is the flexibility of continuous diffusion : it admits expressive latent trajectories, stable ODE/SDE-based sampling, and a mature toolbox of techniques such as self-conditioning, trajectory editing, and potential few-step acceleration via flow-based distillation. These properties make continuous diffusion a powerful and versatile paradigm for generative modeling. This success motivates extending diffusion models to categorical data, including graphs ( 51 ) and sequences ( 3 ) , contributing to the broader goal of unifying cross-modality data generation under a single architecture ( 54 ; 38 ) . The most prominent milestone yet to be conquered for categorical diffusion is language modeling, for which diffusion has morphed into two distinct species. On one hand, discrete diffusion ( 3 ; 31 ; 40 ) reformulates diffusion processes directly in the categorical space based on the continuous-time Markov chain (CTMC) theory. While achieving competitive performance, discrete diffusion sacrifices expressive latent spaces, which limits controllable and few-step generation. On the other hand, simplex diffusion ( 9 ; 49 ; 8 ) applies continuous diffusion over the probability simplex of language data. Although theoretically sound and capable of preserving diffusion tricks, this approach suffers from the extreme sparsity of the simplex space, making score estimation particularly challenging. These shortcomings refocus our attention on embedding-space diffusion ( 29 ; 14 ; 19 ) , an overlooked family of diffusion language models that inherently avoid sparsity and offer editability. Embedding-space diffusion generates token embeddings progressively from Gaussian noise with editable generation paths and denser data spaces. This makes embedding-space diffusion a promising direction to explore for language modeling. However, exploration of embedding-space DLMs has long been blocked by several challenges. First, the theoretical grounding of embedding-space DLMs remains entangled. Training objectives in prior work are either heuristic or cumbersome to implement, for example requiring dynamically sliced batches to optimize different objectives ( 19 ) . Second, a reliable ODE-based estimation of perplexity (PPL) – the primary evaluation metric in language modeling – has yet to be established for embedding-space DLMs, as prior works rely exclusively on SDE-based bounds. Third, the absence of PPL evaluation further obscures the goal of optimizing training techniques, leaving good design choices unclear. Without a solid theoretical foundation for evaluation and improved training techniques, the true potential of embedding-space diffusion language models will always remain unrealized. In this paper, we recognize that the ODE formulation of embedding-space DLMs naturally aligns with Flow Matching ( 30 ) . Building on this, we connect the cross-entropy objective to minimizing Bregman divergence , providing a strong theoretical justification for its use in training LangFlow ( Section 3.1 ). This theoretical connection allows us to derive a novel ODE-based upper bound of negative log-likelihood (NLL), which tackles the challenge of evaluating embedding-space DLMs and improves upon prior SDE-based bounds ( Section 3.2 ). Based on this framework, we further explore the training techniques of LangFlow. Firstly, we elucidate that the noise scheduler of embedding-space diffusion needs to be significantly different from that of image diffusion, which stems from the discrepancy in data modality. We then propose the information-uniform principle to attain the optimal noise schedule ( Section 4.1 ). Secondly, we reveal how the effects of self-conditioning differ between discrete and continuous diffusion and rectify the training protocol of embedding-space DLMs ( Section 4.2 ). Both training techniques contribute greatly to the performance of LangFlow, driving its training to optimality. Contributions: To summarize, our contributions are three-fold: 1) We rethink embedding-space diffusion language models through Bregman Divergence Flow Matching. This unlocks key theoretical insights and empirical gains, including a theoretically grounded cross-entropy loss that streamlines training, and a novel ODE-based negative log-likelihood (NLL) bound that supersedes previous SDE-based ELBOs and provides more accurate evaluation for embedding-space DLMs. The embedding space formulation circumvents the high-dimensionality bottlenecks in diffusion models on the probability simplex and one-hot space. 2) We elucidate the design space of embedding-space DLMs, and propose two key design choices that greatly improve training efficiency and sampling quality. First, we introduce an information-uniform noise schedule with a new time-conditioning based on the logarithmic noise-to-signal ratio \gamma . Our extensive profiling reveals a new optimal noise schedule given by the Gumbel distribution of \gamma , an observation that greatly differs from conclusions in the image generation domain. Second, we reveal the underexplored discrepancy of self-conditioning between its effect on discrete diffusion and that on continuous diffusion, rectifying the training recipe of continuous diffusion language modeling. Both design choices substantially enhance the performance of embedding-space DLMs. 3) Combining everything together, we introduce LangFlow, the first continuous DLM that exceeds discrete DLMs on multiple tasks. Specifically, LangFlow achieves a Perplexity (PPL) of 30.0 on LM1B and 24.6 on OpenWebText, surpassing all uniform-state discrete diffusion and matching the state-of-the-art masked diffusion. On zero-shot transfer, LangFlow beats AR on 4 out of 7 benchmarks and masked diffusion on 3 out of 7 benchmarks. Our unified framework re-establishes baselines for continuous diffusion language modeling, enabling future extensions to be built on a stronger foundation comparable to discrete diffusion. 2 Preliminaries Flow Matching (FM) ( 30 ) is a generative modeling paradigm that learns a velocity field {\bm{u}}_{t}({\bm{z}}_{t}) to transport a simple prior p_{\rm prior} (e.g., standard Gaussian) to the data distribution p_{\rm data} . Starting from {\bm{z}}_{0}\sim p_{\rm prior} , solving the ordinary differential equation (ODE) \mathop{}\!\mathrm{d}{\bm{z}}_{t}={\bm{u}}_{t}({\bm{z}}_{t})\mathop{}\!\mathrm{d}t yields {\bm{z}}_{1}\sim p_{\rm data} . The marginal velocity field {\bm{u}}_{t}({\bm{z}}_{t}) is constructed by marginalizing conditional velocity fields. Under mild regularity conditions, if a conditional velocity field {\bm{u}}_{t}({\bm{z}}_{t}\mid{\bm{z}}) transforms {\bm{z}}_{0}\sim p_{\rm

Read the complete text