Residual Context Diffusion Language Models

Yuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Hooper, Jintao Zhang, Aditya Tomar, Michael W. Mahoney, Sewon Min, Mehrdad Farajtabar, Kurt Keutzer, Amir Gholami, Chenfeng Xu · 2026

Read Residual Context Diffusion Language Models by Yuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Hooper, Jintao Zhang, Aditya Tomar, Michael W. Mahoney, Sewon Min, Mehrdad Farajtabar, Kurt Keutzer, Amir Gholami, Chenfeng Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22954.txt.

Introduction

Yuezhou Hu and nine co-authors improve diffusion language models, which write text by refining many tokens at once. Unlike standard models, they can decode tokens in parallel. But common block-wise versions keep only the most confident tokens and throw the rest away, wasting compute.

Their residual context approach keeps more of that work instead of discarding it each round. Reusing context should make parallel writing faster and less wasteful. Efficient diffusion models could rival standard models for chat and drafting at lower cost. Residual context is the core idea: information the old remasking step discarded is carried forward instead of recomputed.

Catalog record

CREATORYuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Hooper, Jintao Zhang, Aditya Tomar, Michael W. Mahoney, Sewon Min, Mehrdad Farajtabar, Kurt Keutzer, Amir Gholami, Chenfeng Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22954

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Residual Context Diffusion Language Models Yuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Hooper, Jintao Zhang, Aditya Tomar, Michael W. Mahoney, Sewon Min, Mehrdad Farajtabar, Kurt Keutzer, Amir Gholami, Chenfeng Xu Abstract Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a “remasking” mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ∼ \sim 1 billion tokens. RCD consistently improves frontier dLLMs by 5–10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4–5x fewer denoising steps at equivalent accuracy levels. † † affiliationnotice: Equal contribution \icmlJointAdvisement Links: Code | Models | Project Page 1 Introduction Figure 1 : Overview of the residual denoising mechanism in Residual Context Diffusion (RCD). Remasking happens during each denoising step, discarding low-confidence token even though they are already explicitly computed. To recycle computation from previous steps, our proposed method extracts information before it is discarded, and forwards it to the next denoising step. Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to purely autoregressive (AR) models, showing encouraging results on instruction following ( 25 ; 46 ) , code generation ( 10 ) , long-context understanding ( 22 ) , and complex reasoning ( 38 ; 53 ) . While AR models currently dominate large-scale industrial deployment, what makes dLLMs particularly attractive today is their parallel decoding capability: instead of generating tokens strictly one-by-one like AR, they update multiple tokens simultaneously, offering a path to move decoding away from the memory-bandwidth-limited regime that often dominates AR inference ( 39 ) , toward higher compute utilization ( 17 ) . Indeed, recent systems have reported notable speedups over strong AR baselines ( 4 ; 37 ) . However, today’s dLLMs still trail AR models in accuracy. This gap is counterintuitive: AR decoding generates each token in a single pass, whereas dLLMs typically expend significantly more computation per token via multiple sequential denoising iterations. We observe that in state-of-the-art block-wise dLLMs, this extra compute does not reliably translate into accuracy gains. While this gap is partly rooted in the inherent challenges of training diffusion models to capture the strict left-to-right causal dependencies, it is further exacerbated by the inference-time “remasking” strategy, which commits only the most confident tokens in each iteration and discards the rest, as shown in Fig. 1 (left). This effectively “wastes” the intermediate computation performed on tokens that remain undecoded. In Fig. 2 , we show that these intermediate distributions can indeed provide crucial guidance toward the final denoised sequence. In other words, much of the additional compute is wasted, even though it contains structured signals regarding global context. This motivates us to raise a fundamental question: How can we fully exploit the information from every denoising step to unleash the potential of dLLMs? Inspired by the residual learning paradigm ( 12 ) , we propose Residual Context Diffusion (RCD) , a dLLM with a residual denoising mechanism (Fig. 1 (right)) that leverages low-confidence tokens instead of discarding them. By treating their latent representations as a residual update to the input context, the model propagates not only discrete tokens but also continuous embedding vectors enriched with contextual information (Eq. ( 2 )). This allows the model to progressively refine and crystallize its knowledge, transforming previously wasted computation into a guiding signal. Figure 2 : Average token recall across LLaDA GSM8K sampling steps. For each step k (x-axis), recall@5 (y-axis) is the fraction of final decoded tokens that appear in the step- k top-5 predictions at the same position.High early-step recall@5 indicates that intermediate distributions contain semantically informative signals. Although the idea of reusing intermediate computation in block-wise diffusion decoding appears seemingly simple, there are several reasons why making it work is far from trivial. (1) It is unclear where residual contextual signals should come from and how they should be aggregated. One could feed back the token prediction itself, but richer information lives in the full predictive distribution since decoding collapses a distribution into a one-hot decision, and even more semantic cues may be encoded in the model’s hidden states. More importantly, naively injecting and aggregating such residual context can distort the denoising input distribution, leading to instability or degraded decoding. (2) Training with residual feedback is intrinsically hard. The self-referential loop creates a long unrolled computation graph reminiscent of RNNs, making backpropagation-through-time prohibitively expensive under realistic memory budgets. To address these issues, we propose Residual Context Diffusion, which equips diffusion LLMs with the mechanism for recycling computation via residual information. \bullet Entropy-Based Embedding Aggregation: For context selection and aggregation, prior approaches often reuse hidden states (e.g., AR-style such as Coconut ( 11 ) and diffusion-style such as Loopholing ( 16 ) ). However, this is suboptimal: it fails to account for masked-vs-unmasked structure, and it can suffer from a mismatch between embedding magnitudes. RCD instead constructs residual context based on the model’s own embedding codebooks, and it introduces entropy-based aggregation, i.e., using normalized Shannon entropy with temperature-adjusted alignment to extract reliable signals from residual distributions and bridging train–test mismatch. \bullet A two-stage training pipeline: We use a memory- and data-efficient two-stage training pipeline to train our framework. In the first stage, we train a lightweight reference model that generates residual probabilities, and in the second stage we use the reference model to generate model’s input context. The two-stage pipeline decouples the training target and successfully overcomes the memory bottlenecks of backpropagating. We demonstrate that our pipeline can efficiently convert a standard dLLM into the RCD paradigm using as few as \sim 1B tokens. \bullet A new Pareto knob: Residual context allows for a new trade-off between denoising steps and residual transmission. RCD achieves consistently better accuracy–latency trade-offs than throughput-matched baselines, with minimal extra computation compared to the dLLM backbone. Evaluated on the SDAR family and LLaDA, RCD yields 5–10% accuracy gains on benchmarks like GSM8K and MATH500. Notably, on AIME24/25, RCD nearly doubles baseline accuracy while requiring up to 4–5 \times fewer steps at equivalent accuracy levels. Figure 3 : (Left) In vanilla dLLM inference, tokens not selected are reverted to static mask embeddings for the next step, discarding the predicted probability information. (Center) RCD Inference preserves this information by calculating a weighted summation of the unselected probability distributions (residuals) and injects them into the input embeddings of the subsequent step. This allows the model to aggregate more information

Read the complete text