Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Models

Ahmad Fraij, Sam Dauncey · 2025

Read Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Models by Ahmad Fraij, Sam Dauncey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24974.txt.

Introduction

Ahmad Fraij and Sam Dauncey compare two families of language models by how hungrily they consume training data, in work from 2025. When data is scarce, sample efficiency decides which approach is usable.

The lens is double descent, the odd pattern where test error falls, rises, then falls again as models grow. Discrete diffusion models needed more capacity and more training passes to escape their under-sized regime and reach the interpolation point. Deep in the over-sized regime, both families behaved alike, with no strong second descent across many sizes. The verdict: autoregressive models win on small datasets, and diffusion models compete only when given generous capacity and compute.

Catalog record

CREATORAhmad Fraij, Sam Dauncey
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24974

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Models Ahmad Fraij, Sam Dauncey Abstract Data scarcity drives the need for more sample-efficient large language models. In this work, we use the double descent phenomenon to holistically compare the sample efficiency of discrete diffusion and autoregressive models. We show that discrete diffusion models require larger capacity and more training epochs to escape their underparameterized regime and reach the interpolation threshold. In the strongly overparameterized regime, both models exhibit similar behavior, with neither exhibiting a pronounced second descent in test loss across a large range of model sizes. Overall, our results indicate that autoregressive models are more sample-efficient on small-scale datasets, while discrete diffusion models only become competitive when given sufficient capacity and compute. The code is available at: https://github.com/Ahmad1732002/AR-vs-Diffusion-Sample-Efficiency . 1 Introduction Recent progress in large language modeling has been driven primarily by scaling: training increasingly larger models on larger datasets to achieve substantial performance improvements [ 1 , 2 ] . This dependence on massive datasets implies that the demand for human-generated text is growing rapidly. However, such scaling is not sustainable. The stock of public human-generated text may be exhausted within this decade [ 3 ] , highlighting a fundamental limitation for continued scaling. One way to mitigate this data scarcity is to improve the sample efficiency of generative models—that is, to design or adopt architectures that achieve stronger performance with less data. By analyzing and comparing the sampling efficiency of different models, we can identify those that make better use of available data and reduce the dependence on huge datasets. We investigate discrete diffusion models [ 4 ] , whereby the model learns to patch a corrupted passage of text as a potentially more sample-efficient method of pretraining large language models compared to the more common next-token prediction objective of autoregressive models. We propose discrete diffusion models as a promising direction in the search for sample-efficient large-language model pretraining for two reasons: (1) Prior work has shown that autoregressive models overfit to the direction in which facts are presented to them [ 5 , 6 ] but that the masked-language modeling objective can mitigate this [ 7 ] . (2) In the generative image task, despite being introduced in 2015 [ 8 ] , diffusion models only broke through as state-of-the-art in 2020 [ 9 ] , with Ho et al.’s headline result on CelebA-HQ requiring \approx 1000 repetitions of each data sample. We hypothesize that this emergence of diffusion models on images could be the result of increasing compute budgets over time being applied to comparatively small fixed-size datasets such as CelebA-HQ, with 30,000 samples. We propose comparison of double descent curves as a holistic way to measure the sample efficiency of these models in the data constrained setting. Double descent extends the classical bias–variance trade-off by revealing that as model capacity increases, test error does not monotonically decrease but instead follows a double descent curve: decreasing in the classical bias–variance regime, increasing near the interpolation threshold where models just interpolate the training data, and then decreasing again in the over-parameterized regime. It was first formalized in 2019 by Belkin et al. [ 10 ] , who claimed that scaling the number of parameters in the model is the primary driver of double descent behavior. Subsequent work by Nakkiran et al. [ 11 ] generalized this perspective by revealing that double descent arises along multiple axes, including model size, training epochs, and dataset size. By studying double descent behavior through scaling model size, we gain a clearer understanding of how sample-efficient different models are when trained on limited data. A more sample-efficient model generalizes well with fewer parameters. In double descent terms, this means a more sample-efficient model maintains lower test loss in the underparameterized regime, achieves better optimal test performance, reaches the interpolation threshold at a larger model size, and experiences a faster decrease in test loss in the overparameterized regime. Concurrent work by Prabhudesai et al. [ 12 ] compares autoregressive (AR) and diffusion models in data-constrained regimes, analyzing validation loss across model sizes, compute budgets, and repeated data exposure. We contribute to this work by examining the behaviour of these models in the extremely overparameterized regime, and examining the non-monotonic generalization behavior that arises past the interpolation threshold. We leverage the double descent framework to provide a systematic comparison of autoregressive and discrete diffusion models on the extremely small-scale Shakespeare dataset. We empirically show that: • The double descent curve for discrete diffusion models has a broader underparameterised regime compared to autoregressive models. • The double descent curve for diffusion models appears after a much larger number of epochs than autoregressive models (when equating the number of epochs in each model by training flops). • Nonetheless, for our comprehensive sweep of model sizes and epoch counts for a small text dataset, the variational bound on the diffusion model’s test bits-per-token (BPT) is higher than the BPT of the autoregressive model. • For our setup, neither autoregressive nor diffusion models exhibit a pronounced second descent, with test loss remaining high even as the models are scaled past the interpolation thereshold, even with weight decay applied. 2 Experimental Setup 2.1 Model Training Autoregressive and Diffusion models are trained using an exact and an upper bound on the negative log-likelihood, respectively, allowing for their direct comparison. 2.1.1 Autoregressive Language Model Autoregressive models generate sequences by predicting each token conditioned on all preceding tokens. For a training sequence be x=(x_{1},x_{2},\dots,x_{L}) , where L is the sequence length and each token x_{t} belongs to the vocabulary \mathcal{V} . We train the model to minimize the bits-per-token (BPT) loss, which can be computed exactly via the autoregressive factorisation: \text{BPT}(\theta)=-\frac{1}{L}\log_{2}p_{\theta}(x_{1}\dots x_{L})=-\frac{1}{L}\sum_{l=1}^{L}\log_{2}p_{\theta}(x_{l}\mid x_{<l}) 2.1.2 Discrete Diffusion Model In discrete diffusion models, the forward corruption process of discrete data, such as text tokens, is done by applying a categorical transition matrix on the one-hot encoded text tokens [ 4 ] . After a series of T forward diffusion steps the distribution of discrete tokens approaches a stationary distribution over K=|\mathcal{V}|+1 categories, where |\mathcal{V}| is the size of the original vocabulary and the extra category, added at index K-1 , corresponds to the absorbing [MASK] token. In our experiments we choose the transition matrix to use the same absorbing state as in Austin et al. [ 4 ] meaning that each token is either unchanged or transitions to a [MASK] state with probability \beta_{t}=\frac{1}{T-t+1} , hence after T diffusion steps it approaches a non-uniform stationary distribution that has its mass concentrated on the [MASK] token. Specifically, we use a forward process defined by: q(x_{l}^{(t)}|x_{l}^{(t-1)})=\begin{cases}\beta_{t}&\text{if }x_{l}^{(t)}=[MASK]\\ (1-\beta_{t})&\text{if }x_{l}^{(t)}=x_{l}^{(t-1)}\\ 0&\text{otherwise}\end{cases} Whereby each token begins uncorrupted, ie x_{l}=x_{l}^{(0)} . A transformer-based model is trained to denoise the model and predict the true categorical distribution over \mathcal{V} categories. For discrete diffusion models, we have an upper bound on the negative log-likelihood, given by: \begin{gathered}\text{BPT}(\theta)=-\log_{2}p_{\theta}(x_{1}\dots x_{L})\\ \leq\frac{1}{\ln 2}\mathbb{E}_{q}\big[-\log p_{\theta}(x_{:}^{(0)}\mid x_{:}^{(1)})\big]+\frac{1}{\ln 2}\sum_{t=2}^{T}\mathbb{E}_{q}\Big[\mathrm{KL}\big(q(x_{:}^{(t-1)}\mid x_{:}^{(t)},x_{:}^{(0)})\,\|\,p_{\theta}(x_{:}^{(t-1)}\mid x_{:}^{(t)})\big)\Big]\end{gathered} where x_{i} denotes the true value of token i , and x_{:}^{(t)} represents the full sequence after corruption at diffusion timestep t . We defer a complete derivation to Austin et al. [ 4 ] . The first term corresponds to

Read the complete text