Unifying Autoregressive and Diffusion-Based Sequence Generation

Nima Fathi, Torsten Scholak & Pierre-André Noël · 2025

Read Unifying Autoregressive and Diffusion-Based Sequence Generation by Nima Fathi, Torsten Scholak & Pierre-André Noël (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06416.txt.

Introduction

Nima Fathi, Torsten Scholak, and Pierre-André Noël seek to join two ways of writing text with computers, in a 2025 paper. One way writes one word at a time, left to right. The other starts from noise and cleans it up step by step, which lets it fix early slips but can be harder to train and run fast.

The authors give each word place its own noise plan, so both older ways appear as special cases of one frame. They add two mixed noise paths that let a model repair past errors, plus a new way to draw text from the model. Their design also works with a speed trick used by left-to-right models. On standard tests, the method scored well and produced varied, high-grade text.

Catalog record

CREATORNima Fathi, Torsten Scholak & Pierre-André Noël
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06416

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unifying Autoregressive and Diffusion-Based Sequence Generation Nima Fathi, Torsten Scholak & Pierre-André Noël Abstract We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models. We introduce hyperschedules , which assign distinct noise schedules to individual token positions, generalizing both autoregressive models ( e.g. , GPT) and conventional diffusion models ( e.g. , SEDD, MDLM) as special cases. Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes, and we introduce a novel inference algorithm that leverages this new feature in a simplified context inspired from MDLM. To support efficient training and inference, we design attention masks compatible with KV-caching. Our methods achieve state-of-the-art perplexity and generate diverse, high-quality sequences across standard benchmarks, suggesting a promising path for autoregressive diffusion-based sequence generation. See code and resources at https://hdlm-colm.github.io/ . 1 Introduction Generative diffusion models, primarily recognized for their impressive image generation performance in continuous domains ( Yang et al., 2023 ) , are rapidly gaining traction in language modeling, a discrete domain historically dominated by autoregressive (AR) models such as the GPT family ( Radford et al., 2019 ; Brown et al., 2020 ) . Contrary to the apparent separation between AR and diffusion models, and despite their distinct historical development, this work reveals a fundamental connection: AR models are a form of diffusion. The core principle behind diffusion models involves prescribing a “noising” process that gradually destroys information in training data samples, subsequently learning a neural network that progressively generates new samples from “pure noise” with a denoising process. The noising process acts as a form of data augmentation: together with the original training dataset, it specifies the curriculum on which the generator (denoiser) is trained. Part of the attraction for these models arises from their rich theoretical grounding, resulting in concrete practical techniques. In particular, a model’s training and inference environments can be decoupled, allowing for a compute-budget knob at inference time, and guidance techniques adapting a model’s behavior to specific situations. Despite the common use of Gaussian noise in continuous diffusion, the underlying principles can be adapted to discrete state spaces ( Austin et al., 2021 ; Zhou et al., 2023 ; Lou et al., 2024 ) . Common practices for sequence generation have the noising process randomly and independently substituting some original tokens by completely unrelated ones, i.e. , uniformly sampled tokens or a special “mask” absorbing state. A noise schedule determines token replacement probabilities at different points in the curriculum. The resulting sequence at the schedule’s highest noise level retains no mutual information with the original sequence. The generator is trained on this curriculum to enable the production of novel sequences. This work unifies AR and diffusion sequence generation by introducing hyperschedules , allowing different positions in the sequence to be affected by different noise schedules. We establish that autoregressive models, such as GPT, can be understood as diffusion models without data augmentation, utilizing a discrete noise schedule comprising only “full noise” and “no noise” levels. This unification expands model design space and enables a variety of generalized AR-like approaches. With the hindsight, hyperschedule-like concepts can be identified in the “Mask” process of Bansal et al. (2023) as well as in the “Swin-DPM” technique of Feng et al. (2024) . {\lx@inpgf@ignorespaces\mathbf{X}_{T}} {\lx@inpgf@ignorespaces\mathbf{X}_{T-1}} {\lx@inpgf@ignorespaces\cdots} {\lx@inpgf@ignorespaces\mathbf{X}_{1}} {\lx@inpgf@ignorespaces\mathbf{X}_{0}} {\lx@inpgf@ignorespaces\hat{\mathbf{X}}_{T}} {\lx@inpgf@ignorespaces\hat{\mathbf{X}}_{T-1}} {\lx@inpgf@ignorespaces\cdots} {\lx@inpgf@ignorespaces\hat{\mathbf{X}}_{1}} {\lx@inpgf@ignorespaces\hat{\mathbf{X}}_{0}} \scriptstyle{\lx@inpgf@ignorespaces\mathclap{\text{The training curriculum gradually destroys information from data sample (noising).}}} \scriptstyle{\lx@inpgf@ignorespaces q_{T-1|T}} {\lx@inpgf@ignorespaces\scriptstyle\mathrm{data}} \scriptstyle{\lx@inpgf@ignorespaces q_{T-2|T-1}} \scriptstyle{\lx@inpgf@ignorespaces q_{1|2}} \scriptstyle{\lx@inpgf@ignorespaces q_{0|1}} {\lx@inpgf@ignorespaces\scriptstyle\mathrm{noise}} \scriptstyle{\lx@inpgf@ignorespaces p_{T|T-1}^{\boldsymbol{\theta}}} \scriptstyle{\lx@inpgf@ignorespaces p_{T-1|T-2}^{\boldsymbol{\theta}}} \scriptstyle{\lx@inpgf@ignorespaces p_{2|1}^{\boldsymbol{\theta}}} \scriptstyle{\lx@inpgf@ignorespaces p_{1|0}^{\boldsymbol{\theta}}} At inference, new samples are generated in \scriptstyle T generation steps (denoising). Figure 1 : Generative diffusion models prescribe (through q_{\smash{t|t+1}} ) a curriculum process \{\mathbf{X}_{t}\} , then learn (through p_{\smash{t+1|t}}^{\boldsymbol{\theta}} ) a reverse process \{\hat{\mathbf{X}}_{t}\} so that the marginal distributions match at each step t (vertical squiggly lines). \mathbf{X}_{T} is the training dataset and \hat{\mathbf{X}}_{T} is the generated output. This work focuses on discrete diffusion for sequences of discrete tokens. We show that standard autoregressive models ( e.g. , GPT) are an extreme case of this framework, a unification enabling a vast continuum of diffusion models, including autoregressive ones. Recent ( Sahoo et al., 2024 ; Ou et al., 2024 ; Shi et al., 2024 ) and concurrent ( Liu et al., 2024 ; Kim et al., 2025 ; Peng et al., 2025 ; Nie et al., 2025 ; Wang et al., 2025 ; Arriola et al., 2025 ) works have focused on mask diffusion models (MDMs). By specializing on the “absorb” noising mechanism, these MDMs enable great simplifications over a general-purpose treatment: neural networks no longer need an explicit noise-level dependency, and a more standard loss function can be used. However, our work shows that the same feats and simplifications can be accomplished in a general, non-MDM case. Motivated by the same rationale that led concurrent MDMs to conceive “remasking” strategies, we introduce hybrid noising processes, interpolating between the “absorb” and “uniform” processes to combine the benefits of both and achieve state-of-the-art performances, with aspects further improved by our novel adaptive correction sampler (ACS) inference algorithm. Our hyperschedule-equipped approach supports specialized attention matrices enabling KV-caching and efficient training. In summary, our main contributions are: • we unify AR and diffusion sequence generation by introducing hyperschedules ; • we consider hybrid noising processes, reaping benefits from both leading noising processes and achieving state-of-the-art performances, with and without our novel ACS inference algorithm; and • our hyperschedule-powered hybrid processes generalize multiple concurrent developments to non-MDM setting, including efficient training and KV-caching. Figure 2 : We introduce \boldsymbol{\tau} - hyperschedules (top row) – subjecting different token positions i with different noise levels (red high; blue low) at different generation step t – and illustrate how three different noising processes (bottom 3 rows) can be modulated by such hyperschedules. Hyperschedules. (a) Standard AR models ( e.g. , GPT) determine tokens one by one, “quenching” each of them to full determination in a single step. They may thus be construed as an extreme case of a diffusion model. (b) Standard diffusion models ( e.g. , SEDD) gradually anneal all tokens independently of their position. (c) Block-wise application of flat annealing, here for blocks of width \omega=3 . (d) Annealing with a sliding window (“smoothed” AR), here using window width \omega=3 . These last two examples share important features of both AR and diffusion models. While the 4 presented examples all generate \rho=1 token per step in the long-sequence limit (with the caveat that slide experiences an initial overhead of \omega-1 steps), the last 3 patterns are all straightforwardly adapted to \rho>1 (“quick draft”) and \rho<1 (“think hard”) regimes. Noising Processes. (2) The absorb noising process – a.k.a. Masked Diffusion Model (MDM) – overwrites tokens with a special MASK token. These masks are “known unknowns”: it is clear to a denoising model that it must put a non-mask token in its stead. Conversely, unmasked tokens are taken as “absolute truth” during generation: once a token has been unmasked, it remains unaltered until the end. (3) The uniform noising process overwrites with (non-mask) tokens selected uniformly at random from the vocabulary of possible tokens. The orange/light-blue color coding is not part of the state \mathbf{x}_{t}

Read the complete text