Causal Autoregressive Diffusion Language Model
Junhao Ruan, Bei Li, Yongjing Yin, Pengcheng Huang, Xin Chen, Jingang Wang, Xunliang Cai, Tong Xiao, Jingbo Zhu · 2026
Read Causal Autoregressive Diffusion Language Model by Junhao Ruan, Bei Li, Yongjing Yin, Pengcheng Huang, Xin Chen, Jingang Wang, Xunliang Cai, Tong Xiao, Jingbo Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22031.txt.
Introduction
Junhao Ruan and co-authors merge two rival ways of generating text. Autoregressive models write one token at a time, which trains efficiently but decodes slowly. Diffusion language models generate in parallel and run fast, but their training is less direct.
Their 2026 preprint proposes Causal Autoregressive Diffusion, or CARD. It reshapes the diffusion process inside a strictly causal attention mask, so every token still only sees what came before. That design enables dense, per-token supervision in a single forward pass, combining autoregressive training efficiency with diffusion-style throughput at inference. Faster text generation at the same quality would cut serving costs across every product built on language models.
Catalog record
| CREATOR | Junhao Ruan, Bei Li, Yongjing Yin, Pengcheng Huang, Xin Chen, Jingang Wang, Xunliang Cai, Tong Xiao, Jingbo Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22031
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Autoregressive Diffusion Language Model Junhao Ruan, Bei Li, Yongjing Yin, Pengcheng Huang, Xin Chen, Jingang Wang, Xunliang Cai, Tong Xiao, Jingbo Zhu Abstract In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion models. CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass. To address the optimization instability of causal diffusion, we introduce a soft-tailed masking schema to preserve local context and a context-aware reweighting mechanism derived from signal-to-noise principles. This design enables dynamic parallel decoding, where the model leverages KV-caching to adaptively generate variable-length token sequences based on confidence. Empirically, CARD outperforms existing discrete diffusion baselines while reducing training latency by 3 × \times compared to block diffusion methods. Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation, establishing a robust paradigm for next-generation efficient LLMs. Figure 1: Comparison of training paradigms. Current diffusion methods like MDLM and BD3LM are inefficient compared to ARM; MDLM reaches only 50% of ARM’s expected efficiency, while BD3LM relies on complex masking and sequence duplication. CARD overcomes these issues by using causal diffusion, maintaining the same high efficiency as ARM while achieving better performance. Figure 2: Inference comparison of the four paradigms. CARD achieves high-quality results similar to ARM. With KV cache support, friendly operators, and parallel generation, it offers faster throughput than earlier methods. In particular, our inference parallelism is flexible, unlike BD3LM which is tied to the fixed block size used during training. 1 Introduction Causal Autoregressive Models (ARMs) currently serve as the dominant paradigm for training Large Language Models (LLMs), owing to their stable training dynamics and predictable scaling laws. However, as model parameters and test-time compute requirements grow, the sequential nature of autoregressive decoding has emerged as a critical bottleneck. This inefficiency has sparked renewed interest in Text Diffusion Models, which offer theoretical advantages including parallel inference ( 2 ) , iterative refinement ( 39 ) , and potentially higher data modeling capacity ( 27 ) . Early attempts at discrete diffusion faced significant hurdles due to complex training objectives involving variational bounds and numerical instabilities caused by noise sampling ( 2 ) . A turning point occurred with the introduction of Simplified Masked Discrete Diffusion Models (MDLM) ( 37 ; 35 ) . By simplifying the diffusion process into a subspace assumption analogous to a randomized Masked Language Modeling (MLM) ( 10 ) task, MDLM ushered in the era of scalable text diffusion ( 28 ) , enabling the training of modern LLM-scale diffusion models like LLaDA ( 29 ) and Dream ( 42 ) . Despite these advancements, standard MDLMs face severe architectural constraints. As illustrated in the MDLM panel of Figure 1 , its reliance on bidirectional (“Full”) attention prevents the utilization of Key-Value (KV) caching. Consequently, inference speed often falls behind ARMs in practical scenarios ( 41 ) . Furthermore, the arbitrary dependency order in training can lead to ineffective learning pathways ( 19 ) , and the architecture fundamentally lacks support for variable-length generation. To address these limitations, recent works have proposed hybrid architectures such as Block Diffusion (e.g., BD3LM) ( 1 ) . These models (drawn in Figure 1 ) operate at a coarser granularity, applying causal attention between fixed-size blocks and bidirectional attention within them. However, it introduces significant computational overhead. The vectorization required for block-wise training necessitates complex attention masking and can increase memory consumption and training latency by factors of 2\times and 3 \times , respectively. Moreover, the rigid, fixed block size fails to adapt to the varying information density inherent in natural language, limiting dynamic parallelism. In this work, we propose CARD, a framework that combines the training efficiency of ARMs with the parallel inference of diffusion models through a strictly causal formulation. For training, CARD employs a shifted causal attention mechanism where each position predicts its original token from the preceding noised context. This generates a dense diffusion loss for the entire sequence in a single forward pass, achieving 100% token utilization without the overhead of block vectorization. For inference, CARD’s causal structure enables KV-caching (Figure 2 ), allowing the model to append a variable number of [MASK] tokens to the prefix and decode them in parallel through iterative denoising. This dynamic strategy generates multiple tokens per step when confidence is high while falling back to sequential decoding when necessary. We empirically validate CARD on 1B-parameter models trained on 300B tokens, benchmarking against state-of-the-art autoregressive and diffusion baselines. Our results demonstrate that CARD effectively bridges the gap between efficiency and performance: • Superior Performance: CARD achieves an average zero-shot accuracy of 53.2% , outperforming existing diffusion models (MDLM and BD3LM) by over 5.7 points and matching the generation quality of ARMs. Notably, it achieves the lowest zero-shot perplexity on 6 out of 8 evaluated domains. • Training & Inference Efficiency: By eliminating block-wise overhead, CARD reduces training latency by 3 \times compared to Block Diffusion, matching the throughput of standard ARMs. During inference, our confidence-based decoding achieves 1.7 \times to 4.0 \times wall-clock speedup with negligible quality degradation. • Data Potential: Scaling analysis reveals that CARD possesses higher data efficiency than ARMs in data-constrained settings, continuing to improve performance through repeated training epochs where autoregressive baselines saturate. 2 Background We review the evolution of text diffusion models and the specific discrete objective function that serves as the foundation for our work. 2.1 Evolution of Text Diffusion Models Applying diffusion to the discrete domain of language has followed two primary trajectories: continuous embedding methods and discrete state-space models. Continuous approaches, such as Diffusion-LM ( 20 ) and DiffuSeq ( 14 ) , map discrete tokens to Gaussian latent spaces. The disconnect between the continuous diffusion process and the discrete nature of text leads to rounding errors during decoding, often resulting in lower generation performance compared to autoregressive baselines. Discrete DDPM (D3PM) ( 2 ) addressed this by defining the corruption process directly on the vocabulary via transition matrices. While theoretically rigorous, D3PMs initially suffered from optimization instability and inefficient inference. To mitigate this, SEDD ( 23 ) reformulated the objective using score entropy, aligning discrete diffusion closer to its continuous counterparts. However, SEDD relied on time-dependent probability ratios, which prevented step-skipping and slowed inference. RADD ( 30 ) later demonstrated that the explicit time dependency in the input was not strictly necessary for mathematical validity, enabling flexible sampling strategies. A paradigm shift occurred with the introduction of MDLM ( 35 ) and MD4 ( 37 ) . By isolating the absorbing state (masking) transition, these works reduced the complex variational bound to a simplified, randomized Masked Language Modeling (MLM) objective. This simplification significantly improved numerical stability and allowed for scaling laws to be established ( 28 ) , culminating in large-scale pre-trained models like LLaDA ( 29 ) . Despite these successes, standard MDLMs utilize bidirectional attention, which prevents the use of KV caching and degrades inference speed for long sequences. BD3LM ( 1 ) attempts to bridge this gap by segmenting sequences into fixed-size blocks with causal masking between them. While this restores