Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
Linrui Ma, Yufei Cui, Kai Han & Yunhe Wang · 2026
Read Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion by Linrui Ma, Yufei Cui, Kai Han & Yunhe Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13599.txt.
Introduction
Linrui Ma, Yufei Cui, Kai Han, and Yunhe Wang study how diffusion language models build text in 2026. In these models, text takes shape from noise, with each part able to see all the rest. That two-way view helps the whole piece hang together. Many fast designs split text into blocks and build block by block. That choice adds a left-to-right bias.
The team argues that the block style can break the broad sense of the whole. Their work, called Diffusion In Diffusion, tries to win that broad fit back. It keeps the speed gains of block-wise work where it can. The goal is text that stays true to its full theme from start to end.
Catalog record
| CREATOR | Linrui Ma, Yufei Cui, Kai Han & Yunhe Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion Linrui Ma, Yufei Cui, Kai Han & Yunhe Wang Abstract One of the most compelling features of global discrete diffusion language models is their global bidirectional contextual capability. However, existing block-based diffusion studies tend to introduce autoregressive priors, which, while offering benefits, can cause models to lose this global coherence at the macro level. To regain global contextual understanding while preserving the advantages of the semi-autoregressive paradigm, we propose Diffusion in Diffusion — a ’draft-then-refine’ framework designed to overcome the irreversibility and myopia problems inherent in block diffusion models. Our approach first employs block diffusion to generate rapid drafts using small blocks, then refines these drafts through global bidirectional diffusion with a larger bidirectional receptive field. We utilize snapshot confidence remasking to identify the most critical tokens that require modification, and apply mix-scale training to expand the block diffusion model’s global capabilities. Empirical results demonstrate that our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset. Using only 26% of the fine-tuning budget of baseline models, we reduce generative perplexity from 25.7 to 21.9, significantly narrowing the performance gap with autoregressive models. 1 Introduction Today, research on discrete diffusion models ( Austin et al. 2021 ; Lou et al. 2024 ) is divided into two major paradigms: global diffusion models and block diffusion models (also known as Semi-Autoregressive diffusion models). Global diffusion models ( Sahoo et al. 2024 ; Nie et al. ) possess strong global planning capabilities and complete bidirectional context, resulting in text with high global consistency. However, they also prevent the application of Key-Value Cache—the foundation for inference acceleration in modern language models ( Vaswani et al. 2017 ; Brown et al. 2020 ) —leading to an O(L^{2}) complexity during inference and extremely slow processing. In contrast, block diffusion models ( Arriola et al. 2025 ) represent an emerging semi-AR architecture that incorporates autoregressive priors. They enable parallel diffusion execution within “blocks” while employing inter-block autoregression for sequence generation. This enables rapid inference leveraging KV Cache while benefiting from the performance gains of autoregressive priors. However, its macro-level autoregressive paradigm imposes a “shortsightedness,” sacrificing diffusion’s core essence—the global receptive field—and degenerating into a local model. It also introduces irreversibility similar to AR, preventing modifications to already written content. Current research often seeks a balance between the following trade-offs: stronger global planning capabilities necessitate sacrificing inference speed, while better autoregressive priors and faster inference require sacrificing global perspective. We contend that this dichotomy is not irreconcilable. Future architectures should not be inherently AR models with minor local diffusion biases, but rather novel diffusion frameworks that preserve semi-AR advantages while retaining global vision and planning capabilities. By reintroducing global capabilities into Block Diffusion models, we aim to break the impossible triangle of speed vs. quality vs. global consistency in discrete diffusion, resulting in a unified architecture that combines the strengths of both approaches. In this study, we propose a novel generative framework called Diffusion in Diffusion (also known as Structured Block Diffusion), designing a multi-stage “draft-then-revise” paradigm to bring back the global ability to the aforementioned semi-autoregressive diffusion models. Inspired by the human writing process—rapidly drafting a first version followed by holistic revision to ensure coherence—our approach decomposes generation into progressive stages: During the initial drafting phase, the model employs a small blocksize for fast, efficient generation, focusing first on specific grammar and local content. In subsequent revision phases, the model re-examines and refines the previously generated sequence using a larger blocksize (global receptive field). We also innovatively introduce a Remasking mechanism into Block-Diffusion Models, which selectively transforms tokens within the generated sequence back into noise. This enables the model to resample these positions within a more holistic context for optimization. Our work makes three key contributions: 1. Structural Diffusion Framework : We propose a multi-stage generative framework that progressively increases the block size through multiple stages. This enables the model to first rapidly generate drafts using semi-regressive methods, then fine-tune content with high quality via bidirectional global diffusion, effectively combining the strengths of both paradigms. 2. Snapshot Confidence Remask : We evaluated multiple inter-stage remask token selection strategies and determined that snapshot confidence remask—based on token generation-time confidence—is the most effective approach, outperforming methods like Post-hoc likelihood evaluation. 3. Mix-Scale Training Strategy : We also propose an enhanced Block Diffusion training approach. By mixing block masks of minimal and maximal scales, the model gains multi-granularity information processing capabilities, enabling both generation and revision functions. Empirical results demonstrate that our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset using only one-quarter of the fine-tuning budget of comparable models: generating perplexity drops from 25.7 to 21.9 , significantly narrowing the gap with autoregressive benchmarks. 2 Preliminaries 2.1 Autoregressive Models In a discrete (tokenized) sequence x=[x_{1},\dots,x_{L}] of length L , each token x_{i} is a one-hot vector of model vocabulary \mathcal{V} . The standard autoregressive model decomposes the joint probability distribution p(x) into a product of conditional probabilities: \log p_{\theta}(x)=\sum_{i=1}^{L}\log p_{\theta}(x_{i}|x_{<i}), (1) Here, x_{<i} denotes the prefix token preceding position i . The conditional distribution p_{\theta}(x_{i}|x_{<i}) is typically modeled via a causal attention mechanism. Although AR models can generate highly fluent and natural text and support key-value caching, their inherently sequential nature precludes straightforward parallel generation. 2.2 Block Diffusion Models Block Diffusion Language Models (BD3-LMs) ( Arriola et al. 2025 ) lie between autoregressive and diffusion paradigms. They predefine a fixed block size \mathcal{B} (assuming B=L/\mathcal{B} ) and partition the entire sequence into B blocks (denoted as x^{1},\dots,x^{B} ). Each block is subsequently treated as a ‘metatoken’ at the autoregressive level, \log p_{\theta}(x)=\sum_{b=1}^{B}\log p_{\theta}(x^{b}|x^{<b}), (2) Here, x^{b} denotes the token sequence in the b th block of x , while x^{<b} represents all preceding blocks. Unlike autoregressive models, the conditional distribution within blocks, p_{\theta}(x^{b}|x^{<b}) , is modeled via a discrete diffusion process. Forward Masking Process Following the masked diffusion framework ( Austin et al. 2021 ; Sahoo et al. 2024 ) , in the forward process q(x_{t}^{b}|x^{b}) , each token in b is independently converted to the special masking token [MASK] (denoted as m ) via according to transition probabilities derived from a noisy scheduling parameter \alpha_{t}\in[0,1] : q(x_{t}^{b}|x^{b})=\prod_{j=1}^{\mathcal{B}}\left(\alpha_{t}\mathbb{I}(x_{t,j}^{b}=x_{j}^{b})+(1-\alpha_{t})\mathbb{I}(x_{t,j}^{b}=m)\right), (3) where t goes from 0 to 1 , a larger t denotes a noisier timestep, and t=1 indicates full noise, and x_{j}^{b} represents the j -th token within the b -th block. Reverse Denoising Process & Training The objective of the generation process (i.e., the denoising procedure) is to learn to reverse this noise-adding process. The aim is to train the denoising network x_{\theta} to predict a clean current block x^{b} based on the context provided by the current noisy block x_{t}^{b} (subject to its current level of random noise) and the previously denoised clean blocks x^{<b} . Here, BD3-LM employs a key-value cache to leverage prior blocks x^{<b} , whilst simultaneously enabling parallel denoising within blocks and decoding multiple tokens concurrently. Consistent with conventional MDLM practices, the model’s training objective is to minimise the negative evidence lower bound (NELBO) ( Sahoo et al. 2024 ) : \mathcal{L}_{\text{BD}}(x;\theta)=\sum_{b=1}^{B}\mathbb{E}_{t\sim[0,1]}\mathbb{E}_{q(x_{t}^{b}|x^{b})}\left[\frac{\alpha^{\prime}_{t}}{1-\alpha_{t}}\mathcal{L}_{\text{CE}}(p_{\theta}(\cdot|x_{t}^{b},x^{<b}),x^{b})\right]. (4) This objective effectively trains the model to perform ”infilling” or conditional generation for the