Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation
Tianqi Du, Lizhe Fang, Weijie Yang, Chenheng Zhang Zeming Wei Yifei Wang Yisen Wang · 2026
Read Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation by Tianqi Du, Lizhe Fang, Weijie Yang, Chenheng Zhang Zeming Wei Yifei Wang Yisen Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13228.txt.
Introduction
Tianqi Du and co-authors show that step by step text models can match diffusion models at free order writing. Diffusion text models can fill a gap at any spot in a text and can use both sides of the gap. That flex helps tasks such as fill in, rewrite, and self fix.
Yet their form caps depth. They guess one part from a next part in a single step link. Step by step models tend to give text of more grade and calm. The team shows how such models can be taught to write in any order too. If the claim holds, teams may not need a split model class for flex tasks.
Catalog record
| CREATOR | Tianqi Du, Lizhe Fang, Weijie Yang, Chenheng Zhang Zeming Wei Yifei Wang Yisen Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13228
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation Tianqi Du, Lizhe Fang, Weijie Yang, Chenheng Zhang Zeming Wei Yifei Wang Yisen Wang Abstract Diffusion language models enable any-order generation and bidirectional conditioning, offering appealing flexibility for tasks such as infilling, rewriting, and self-correction. However, their formulation—predicting one part of a sequence from another within a single-step dependency—limits modeling depth and often yields lower sample quality and stability than autoregressive (AR) models. To address this, we revisit autoregressive modeling as a foundation and reformulate diffusion-style training into a structured multi-group prediction process. We propose Any-order Any-subset Autoregressive modeling (A3), a generalized framework that extends the standard AR factorization to arbitrary token groups and generation orders. A3 preserves the probabilistic rigor and multi-layer dependency modeling of AR while inheriting diffusion models’ flexibility for parallel and bidirectional generation. We implement A3 through a two-stream attention architecture and a progressive adaptation strategy that transitions pretrained AR models toward any-order prediction. Experiments on question answering, commonsense reasoning, and story infilling demonstrate that A3 outperforms diffusion-based models while maintaining flexible decoding. This work offers a unified approach for a flexible, efficient, and novel language modeling paradigm. 1 Introduction Diffusion language models have recently emerged as a powerful alternative to autoregressive (AR) modeling for text generation ( 29 ; 11 ; 15 ; 37 ) . By iteratively denoising partially masked sequences, they allow any-order generation and bidirectional context conditioning, enabling novel capabilities such as global rewriting, infilling, and self-correction. These properties make diffusion models conceptually appealing for flexible text generation beyond the rigid left-to-right paradigm. However, diffusion modeling also introduces fundamental limitations. In each denoising step, the model predicts one part of the sequence conditioned on another, forming only a single-layer dependency structure. This simplification weakens the model’s ability to capture deep, hierarchical dependencies across tokens. As a result, despite their flexibility, diffusion-based models often underperform AR models in generation quality and training stability ( 25 ; 37 ; 52 ) . These challenges highlight an inherent trade-off between the expressive power of AR factorization and the flexibility of diffusion-style generation. Motivated by this observation, we revisit autoregressive modeling as a foundation for constructing a more expressive and stable framework. AR models naturally capture rich, recursive dependencies through sequential conditioning but suffer from their strict left-to-right constraint. We therefore reformulate diffusion training in a more structured way by extending its two-group prediction to multiple groups, so that the model can preserve AR’s multi-layer dependency structure while supporting arbitrary generation orders. This leads to Any-order Any-subset Autoregressive modeling (A3), a unified framework that generalizes standard AR factorization to allow token groups of arbitrary composition and order. A3 retains the probabilistic rigor and stability of AR training while inheriting diffusion models’ flexibility for parallel generation and bidirectional context use. To realize A3 in practice, we design a two-stream attention architecture that supports arbitrary group orderings, and develop a progressive training strategy that adapts pretrained AR models to any-order prediction. Our framework naturally supports diverse inference strategies, including groupwise AR sampling and dynamic resampling, offering a tunable trade-off between generation speed and quality. Through comprehensive experiments on question answering ( 23 ) , commonsense reasoning ( 51 ; 38 ; 40 ; 2 ) , and story infilling tasks ( 33 ) , we show that A3 achieves strong performance across diverse benchmarks while enabling flexible and efficient generation. Notably, A3 outperforms state-of-the-art diffusion-based models despite using substantially less training data, and demonstrates promising scaling behavior with model size. These results suggest that A3 provides a new direction for bridging the gap between AR and parallel generation paradigms. Our contributions can be summarized as follows: • Conceptually , we propose A3, a new sequence modeling paradigm that reformulates diffusion-style group prediction into a generalized autoregressive framework, preserving dependency depth while enabling any-order , any-subset generation. • Practically , we design a two-stream attention architecture and a progressive training recipe that extend pretrained AR models to support arbitrary group orderings. • Empirically , we demonstrate that A3 surpasses diffusion-based approaches, offering a flexible and scalable foundation for next-generation language modeling. 2 Any-order Any-subset Autoregressive Modeling 2.1 Formulation In this section, we will demonstrate the formulation of our proposed Any-order Any-subset Autoregressive modeling (A3). Masked diffusion language models provide a flexible approach to sequence generation by iteratively predicting masked subsets of tokens conditioned on visible context ( 19 ; 34 ) . Concretely, during training, given a sequence x_{1:N} , the index set {1,2,\dots,N} is partitioned into two disjoint subsets G_{1} and G_{2} . The model predicts the masked tokens x_{G_{2}} conditioned on the visible tokens x_{G_{1}} : P(x_{G_{2}}\mid x_{G_{1}})=\prod_{t\in G_{2}}P(x_{t}\mid x_{G_{1}}). (1) This one-step conditional formulation enables parallel prediction and bidirectional context use, supporting any-order generation through repeated resampling of subsets. However, it models only shallow dependencies —each token in G_{2} depends directly on x_{G_{1}} but not recursively on other predicted tokens. As a result, the global dependency structure is limited to one layer of conditioning, unlike the multi-step compositional nature of autoregressive (AR) factorization. To enrich the dependency structure while retaining the flexibility of arbitrary grouping, we reformulate this process through an autoregressive lens. Starting from the AR formulation, P(x_{1:N})=\prod_{t=1}^{N}P(x_{t}\mid x_{<t}), (2) we generalize the token-wise sequence to a group-wise factorization. The index set \{1,\dots,N\} is partitioned into K disjoint groups \{G_{1},G_{2},\dots,G_{K}\} , and the joint probability is expressed as P(x_{1:N})=\prod_{k=1}^{K}P(x_{G_{k}}\mid x_{G_{<k}}). (3) Each group G_{k} may contain one or more tokens, and the group order is arbitrary. This generalization retains the recursive dependency structure of AR while introducing diffusion-style flexibility. Each factor can be viewed as predicting one subset of tokens from previously generated subsets, but now organized into multiple layers of dependency. Training the model on randomly sampled group partitions and permutations exposes it to diverse factorization orders and improves robustness to different conditional structures. This resembles the permutation LM objective of XLNet ( 49 ) but at the group level, enabling richer structural modeling. This formulation bridges diffusion’s parallel prediction and AR’s hierarchical modeling, forming the foundation of Any-order Any-subset Autoregressive modeling (A3) . 2.2 Discussion with Previous Paradigms Comparison with Masked Diffusion Language Models. Masked diffusion language models (MDLMs) have recently emerged as an alternative to AR decoding, aiming to overcome the sequential bottleneck of one-token-at-a-time generation. By iteratively denoising partially masked sequences, MDLMs can update multiple tokens in parallel and exploit bidirectional context ( 1 ; 29 ; 11 ; 34 ) . This flexibility enables controllable, any-order generation and supports tasks such as infilling and global rewriting. However, diffusion approaches face two major limitations. First, inference speed is constrained by the need for many iterative refinement steps, with generation efficiency depending critically on noise schedules and step counts ( 11 ) . While large-scale efforts such as DiffuGPT and LLaDA demonstrate that diffusion LMs can match or surpass AR models in quality, they still require careful tuning and incur nontrivial decoding costs ( 11 ; 34 ) . Second, training stability is less favorable than AR: discrete diffusion objectives require complex forward–reverse processes and additional pretraining or adaptation, making optimization more resource-intensive and sensitive to hyperparameters ( 19 ) . In contrast, A3 preserves the probabilistic rigor and training simplicity of AR modeling