Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
Tongyuan Miao, Gary Huang, Kai Jun Han, Annie Jiang · 2025
Read Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models by Tongyuan Miao, Gary Huang, Kai Jun Han, Annie Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19004.txt.
Introduction
Tongyuan Miao, Gary Huang, Kai Jun Han, and Annie Jiang speed up diffusion language models by starting the race partway down the track. These models generate text in parallel but begin from a fully masked blank, wasting many denoising steps. Their idea, posted to arXiv in December 2025, is context-aware initialization: a small auxiliary model injects prompt-based priors into the starting point, through discrete token injection or embedding-level interpolation.
A confidence-based remasking step keeps the model skeptical of bad priors, since unmask-only decoding can lock in early mistakes. On the GSM8K math benchmark, denoising iterations drop by about 35 percent in function evaluations. The honest caveat: naive warm-starting can also hurt final accuracy, so the authors frame calibration and revision mechanisms as the research agenda. Faster generation only counts if the text stays good.
Catalog record
| CREATOR | Tongyuan Miao, Gary Huang, Kai Jun Han, Annie Jiang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models Tongyuan Miao, Gary Huang, Kai Jun Han, Annie Jiang Abstract . 1. Abstract Diffusion Large Language Models (DLLMs) enable fully parallel token decoding but often remain impractical at inference time due to the many denoising iterations required to refine an information-free, fully masked initialization into coherent text. Most existing acceleration methods focus on traversing this generative trajectory more efficiently via improved solvers or sampling strategies. We advance a complementary perspective: shorten the trajectory itself by starting closer to the target distribution through context-aware initialization . We propose a training-free interface that injects prompt-conditioned priors from a lightweight auxiliary model into the diffusion initialization, and instantiate it with two mechanisms: discrete token injection and representation-level embedding interpolation. Because injected priors can be imperfect and unmask-only decoding can over-commit early, we also introduce a simple confidence-based remasking mechanism as a form of prior skepticism. Preliminary evidence on GSM8K suggests that context-aware initialization can substantially reduce denoising iterations (about 35% fewer function evaluations in our setting), while also exposing a key open challenge: naive warm-starting can degrade final accuracy relative to strong diffusion baselines. We use these findings to motivate a research agenda around calibration, revision mechanisms, and representation alignment for reliable warm-started diffusion decoding. 2. Introduction Consider a simple question-answering prompt: Prompt: "2 + 2 = ?" An autoregressive language model produces the answer “4” in a single forward pass. A Diffusion Language Model (DLLM), despite having access to the same context, must instead begin from a fully masked sequence and iteratively refine each token through dozens or even hundreds of denoising steps before “4” emerges. This inefficiency persists even when the output is highly predictable: basic arithmetic, common facts, or formulaic responses all demand the same costly iterative refinement. Figure 1 . Autoregressive models generate tokens sequentially conditioned on prior outputs, while diffusion language models decode all positions in parallel via iterative denoising. Adapted from ( 5 ) . We argue that a primary cause of this inefficiency is initialization mismatch . DLLMs begin generation from a state that is extremely far from the target distribution, regardless of how informative the input prompt may be. When a user asks “The capital of France is…”, the model effectively “knows” the answer should be Paris , yet the diffusion process forces it to start from a completely masked sequence as if no contextual signal were available. Tokens with high contextual certainty are treated identically to highly uncertain ones. This mismatch fundamentally limits DLLM efficiency. While diffusion models offer fully parallel token decoding and avoid the sequential bottleneck of autoregressive generation, their practical throughput often falls short of autoregressive systems enhanced with KV-caching. Existing acceleration methods focus primarily on traversing the generative trajectory more efficiently, for example through improved numerical solvers or block-wise unmasking strategies. We propose a complementary perspective: rather than accelerating movement along the diffusion trajectory, we seek to shorten the trajectory itself by starting closer to the destination. Why now. Recent progress in diffusion-based language modeling has significantly improved per-step efficiency, but inference latency remains dominated by the number of required denoising iterations. At the same time, modern inference pipelines increasingly rely on auxiliary models—such as autoregressive LLMs, retrievers, or tool-augmented systems—that already produce strong task-conditioned hypotheses. These signals are typically discarded before diffusion begins. This convergence suggests that initialization, rather than solver efficiency alone, represents a natural next bottleneck in diffusion language model inference. In this paper, we advance the vision of context-aware initialization for diffusion language models. Instead of beginning from an information-free masked sequence, we propose conditioning the initial diffusion state on context-dependent signals derived from upstream language models. By injecting semantic structure before denoising begins, the diffusion process can start substantially closer to the final output, reducing the number of required function evaluations while preserving the model’s ability to refine and correct its predictions. Importantly, our goal is not to present a finalized acceleration technique, but to identify initialization as a critical and underexplored dimension of the DLLM design space. Through a set of training-free warm-start mechanisms and preliminary empirical evidence, we expose both the promise of this approach and the failure modes that must be addressed to make it reliable in practice. Contributions. This paper makes the following contributions: • Problem framing. We identify initialization mismatch as a first-order bottleneck in diffusion language model inference and argue that trajectory length itself is a controllable variable. • Design space for context-aware initialization. We introduce a family of training-free warm-start strategies that inject auxiliary model priors into the diffusion initialization, including token-level and embedding-level approaches. • Correction mechanisms. We propose remasking as a general principle for revising unreliable warm-start signals during diffusion, enabling the model to reconsider low-confidence initializations. • Preliminary evidence and insights. Initial experiments suggest that warm-starting can substantially reduce denoising iterations, while also revealing key challenges related to prior miscalibration and representation mismatch. Together, these contributions motivate a research agenda centered on reliable, context-aware initialization as a path toward making fully parallel diffusion language models practical at scale. Figure 2 . Model confidence increases with denoising steps, suggesting that earlier access to reliable structure could reduce the required refinement budget. Adapted from ( 7 ) . 3. System Architecture Setting and notation. Let the input prompt be a token sequence p=(p_{1},\ldots,p_{m}) and let the desired output length be n . We generate an output sequence x_{0}=(x_{0,1},\ldots,x_{0,n})\in\mathcal{V}^{n} over vocabulary \mathcal{V} using a masked diffusion language model (DLLM). The diffusion state at reverse step t is a partially specified sequence x_{t}\in(\mathcal{V}\cup\{\texttt{[MASK]}\})^{n}, where [MASK] denotes the special mask token and indices i\in\{1,\ldots,n\} refer to output positions. Baseline initialization. In masked diffusion, inference begins from an information-free terminal state in which every position is masked: (1) x_{T}\;=\;(\texttt{[MASK]},\ldots,\texttt{[MASK]})\in(\mathcal{V}\cup\{\texttt{[MASK]}\})^{n}. Because x_{T} contains no lexical content, early reverse steps must discover coarse structure from scratch, which can increase the number of denoising iterations required to reach a high-quality x_{0} . Goal and high-level approach. We seek to reduce inference cost by warm-starting the reverse process: we replace Eq. ( 1 ) with a prompt-conditioned initialization that injects semantic structure without retraining the DLLM. Concretely, we compute a prompt-conditioned warm proposal using a smaller auxiliary language model, then inject this signal into the DLLM initialization either as discrete token IDs (token-level prior) or as continuous embeddings (representation-level prior). Design requirements and scope. We focus on a training-free interface that (i) modifies only inference-time initialization, (ii) preserves compatibility with existing Fast-DLLM decoding (e.g., confidence-threshold unmasking), and (iii) retains the ability to revise unreliable priors during generation. We do not claim that the specific instantiations below are optimal; rather, they serve as concrete points in a broader design space for context-aware initialization. Warm proposal generator. We use an auxiliary autoregressive model f_{\phi} to produce prompt-conditioned token proposals (2) \hat{x}=f_{\phi}(p), where \hat{x}=(\hat{x}_{1},\ldots,\hat{x}_{n}) and \hat{x}_{i}\in\mathcal{V} . Diffusion-embedding warm lookup. Let \mathrm{Emb}_{\theta}(\cdot) denote the DLLM token embedding layer. Given warm token IDs \hat{x} , define (3) \hat{e}_{i}=\mathrm{Emb}_{\theta}(\hat{x}_{i}),\qquad i=1,\ldots,n, and collect them into \hat{E}=(\hat{e}_{1},\ldots,\hat{e}_{n})\in\mathbb{R}^{n\times d} . Warm initialization operator. We define an operator \mathcal{W} that combines the baseline all-mask state and warm proposals to produce a warm-started terminal condition: (4) \tilde{x}_{T}\;\;\text{or}\;\;\tilde{E}_{T}\;=\;\mathcal{W}(x_{T},\hat{x},\hat{E}). We instantiate \mathcal{W} in two ways: (i)