Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Caio Vicentino · 2026

Read Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison by Caio Vicentino (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22075.txt.

Introduction

Vicentino runs a controlled comparison of two kinds of language model: autoregressive and masked diffusion. Both train on the same data, budget, and hardware: 50 million tokens of TinyStories. Training speed is nearly equal, with diffusion only 4.7% slower overall. The autoregressive model converges faster but starts overfitting by step 14,000. The diffusion model improves more slowly and is still gaining at step 20,000. Vicentino finds a trade-off: autoregressive text is fluent but repetitive, while diffusion text is more varied but less grammatical.

Catalog record

CREATORCaio Vicentino
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24700
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison Caio Vicentino Abstract We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput ( ∼ {\sim} 50K tokens/second), with MDLM requiring only 4.7% more wall-clock time, countering the perception that diffusion training is substantially more expensive. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity–fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct- n n , lower Self-BLEU), at the cost of occasional grammatical inconsistencies. We note that validation losses across paradigms are not directly comparable, as they measure different objectives (next-token prediction vs. masked token prediction). All code, trained checkpoints, and data pipelines are released for reproducibility. 1 Introduction Language models have been overwhelmingly autoregressive: tokens are generated left-to-right, each conditioned on all preceding tokens. This paradigm underpins GPT ( Radford et al. 2019 ) , LLaMA ( Touvron et al. 2023 ) , and virtually all production language models. Its strengths are well understood—efficient training via teacher forcing, straightforward sampling, and strong perplexity—but so are its limitations: sequential generation that cannot revise earlier tokens, and a strong left-to-right bias that can produce repetitive outputs when the most probable prefix dominates. A fundamentally different paradigm has recently emerged: diffusion-based language models. Adapted from the continuous diffusion framework that revolutionized image generation ( Ho et al. 2020 ; Song et al. 2021 ) , these models generate text by starting from a fully corrupted (masked) sequence and iteratively denoising it. Masked Diffusion Language Models (MDLM; Sahoo et al. 2024 ) use a learned masking schedule to corrupt text during training and reverse the process during generation. SEDD ( Lou et al. 2024 ) achieved ICML 2024 Best Paper with substantially improved generative perplexity over GPT-2. LLaDA ( Nie et al. 2025 ) demonstrated competitive performance at 8B parameters, matching LLaMA3 on downstream benchmarks. Gemini Diffusion ( Google DeepMind 2025 ) demonstrated fast generation at commercial quality. Despite this rapid progress, controlled head-to-head comparisons remain scarce. Existing studies either compare at different scales, use different datasets, or employ different compute budgets, making it difficult to attribute observed differences to the generation paradigm itself rather than confounding variables. This work. We train an autoregressive Transformer and a masked diffusion Transformer on exactly the same data, compute budget, and hardware. The only differences are those inherent to each paradigm: causal vs. bidirectional attention, next-token vs. masked-token objective, and the MDLM’s timestep conditioning module. This controlled setup allows us to isolate the effect of the generation paradigm on training dynamics, convergence behavior, and generation quality. We acknowledge upfront that our experiments are at small scale (123–163M parameters, 50M tokens) and that our generation quality analysis is qualitative rather than quantitative. Our goal is not to declare a winner, but to provide a reproducible baseline comparison that highlights the distinct trade-offs of each paradigm. 1.1 Contributions 1. Controlled experimental setup : AR and MDLM trained with identical data, steps, batch size, sequence length, model dimensions, optimizer, and hardware (Table 1 ). To our knowledge, this is the most tightly controlled small-scale comparison of these paradigms to date. 2. Training dynamics analysis : We provide step-by-step convergence curves (Table 3 ) showing that AR converges faster but overfits earlier, while MDLM converges slowly but steadily, suggesting different compute-optimal training lengths. 3. Throughput parity : We find that MDLM training is only 4.7% slower than AR in wall-clock time (113.0 vs. 107.9 minutes), countering the common assumption that diffusion training is substantially more expensive. 4. Quantitative diversity–fluency trade-off : Over 1,000 generated samples, we show that MDLM achieves 93.4% unique openings vs. AR’s 3.3%, with higher Distinct- n and lower Self-BLEU, confirming a structural diversity advantage at the cost of fluency. 5. Full reproducibility : All code, data pipelines, trained checkpoints (PyTorch), and evaluation scripts released at https://github.com/caiovicentino/arche . 2 Related Work 2.1 Autoregressive Language Models The autoregressive Transformer ( Vaswani et al. 2017 ; Radford et al. 2019 ) remains the dominant language modeling paradigm. Scaling laws ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) have established predictable relationships between model size, data, and performance. Recent architectural innovations include Multi-Head Latent Attention for KV-cache compression ( DeepSeek-AI 2024 ) , Gated Linear Attention for efficient training ( Yang et al. 2024 ) , and State Space Models for linear-time inference ( Gu & Dao 2023 ) . All of these operate within the autoregressive framework. 2.2 Diffusion Language Models Adapting diffusion to discrete text has been an active area. D3PM ( Austin et al. 2021 ) introduced discrete diffusion with transition matrices. MDLM ( Sahoo et al. 2024 ) simplified this to a masking-based framework with a cosine schedule, achieving strong results with standard Transformer backbones. SEDD ( Lou et al. 2024 ) used score-based estimation for discrete distributions, earning ICML 2024 Best Paper. LLaDA ( Nie et al. 2025 ) scaled masked diffusion to 8B parameters, demonstrating competitive downstream performance with LLaMA3. Gemini Diffusion ( Google DeepMind 2025 ) achieved commercial-grade generation with 5 \times speed improvements. The key architectural difference from AR models is that diffusion models use bidirectional attention (each position can attend to all unmasked positions) and condition on a noise level or timestep. This bidirectional context is both a strength (richer conditioning) and a difference that complicates direct comparison. 2.3 Existing Comparisons Prior comparisons between AR and diffusion language models exist but are not tightly controlled. Sahoo et al. 2024 compare against GPT-2 but at different parameter counts and with different training data. Nie et al. 2025 compare against LLaMA3 but with different training corpora and compute budgets. Lou et al. 2024 report perplexity improvements over GPT-2 but using a different model architecture. Our work fills the gap of a same-data, same-budget, same-hardware comparison, albeit at small scale. 3 Experimental Setup 3.1 Controlled Variables Table 1 summarizes the experimental design. All shared hyperparameters—data, training steps, batch size, sequence length, model dimensions, number of layers, number of heads, FFN dimension, optimizer, and hardware—are identical between the two models. Table 1: Controlled experimental setup. All shared variables are identical; differences arise solely from the paradigm. Variable AR MDLM Shared (identical) Data TinyStories ( Eldan & Li 2023 ) , 50M tokens Steps 20,000 Batch size 32 Sequence length 512 d_{\text{model}} 768 Layers 12 Heads 12 FFN dim 2,048 Optimizer AdamW (lr = 3e-4, wd = 0.01) Hardware NVIDIA H100 80GB Paradigm-specific (inherent differences) Attention mask Causal Bidirectional Training objective Next-token prediction Masked token prediction Noise schedule N/A Cosine Timestep conditioning N/A Sinusoidal embedding Parameters 123.6M 162.7M 3.2 Parameter Count Difference The MDLM model has 31.6% more parameters than the AR model (162.7M

Read the complete text