Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code
Kishan Maharaj, Ashita Saxena, Srikanth Tamilselvam · 2026
Read Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code by Kishan Maharaj, Ashita Saxena, Srikanth Tamilselvam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23690.txt.
Introduction
Maharaj, Saxena, and Tamilselvam survey alternatives to left to right generation for AI code assistants, in a 2026 survey paper on software engineering. Autoregressive models write code one token at a time, never revising earlier choices. That rigidity hurts programming, where a decision late in a file can demand changes near the top, and an early mistake poisons everything after it. Human developers instead draft, sketch, and refine in loops.
The survey covers three emerging paradigms. Diffusion models generate code by denoising a whole draft at once, capturing long range structure that next token prediction misses. Code world models simulate program execution states, grounding reasoning in what code actually does. State space models process very long contexts in linear time, suiting massive codebases. Drawing also on cognitive neuroscience, the authors sketch directions toward deliberative, System 2 style code generation agents that plan before they write.
Catalog record
| CREATOR | Kishan Maharaj, Ashita Saxena, Srikanth Tamilselvam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23690
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Code Kishan Maharaj, Ashita Saxena, Srikanth Tamilselvam Abstract Autoregressive (AR) language models have driven significant progress in automated software engineering, enabling powerful code generation and assistance systems. However, the next-token prediction paradigm introduces structural limitations for code reasoning, including restricted global planning, challenges in maintaining long-range dependencies, and limited grounding in program execution semantics. Noting the heavy skewness of existing literature towards AR models, we discuss emerging paradigms that could potentially overcome the logic and scaling bottlenecks of next-token prediction by unlocking next-generation architectural capabilities for code intelligence. Specifically, we discuss the potential of Diffusion Models, which generate code via holistic denoising that captures long-range syntactic constraints often missed by AR models. We also discuss Code World Models (CWMs), which simulate execution states to support reasoning, and State Space Models (SSMs), which provide linear-time efficiency for massive contexts. By connecting these developments with findings from cognitive neuroscience, we outline directions for developing "System 2" code generation agents. 1 Introduction: The Autoregressive Bottleneck The dominant framework for modern Large Language Models (LLMs) is the autoregressive (AR) objective 63 ; 64 , which models the joint probability of a sequence x=(x_{1},...,x_{n}) as the product of conditional probabilities: P(x)=\prod_{t=1}^{n}P(x_{t}|x_{<t}) . While this formulation is mathematically tractable and highly effective for natural language, which flows linearly, aligning with human speech, it imposes a rigid sequentiality that can limit its expressiveness, which could lead to poor generalizability 16 ; 96 ; 50 . The left-to-right generation process of AR models creates what we term the "Sequential Dependency Trap" . In software engineering, early decisions (e.g., selecting a library version or defining a function/class) constrain code written later, while implementation details in function bodies may determine required imports at the beginning of the file, introducing dependencies that flow backwards relative to AR generation. When an AR model makes an early mistake, such as choosing a suboptimal data structure or misusing a function definition, it creates a "poisoned" context. As the model cannot backtrack, subsequent tokens must condition on this error, which cannot be corrected, leading to error propagation in which the model hallucinates justifications for the initial mistake 97 ; 35 ; 41 . In contrast, human developers code non-linearly: drafting function signatures, sketching implementations, adding missing imports, and iteratively refining the logic . The "write-once" nature of AR inference, therefore, precludes this feedback loop. Programming languages are formal languages governed by strict grammatical rules that often span long distances in code 33 ; 3 . For instance, a closing brace } at line 500 must match an opening brace { at line 10, and a variable used deep within a loop must comply with type definitions established earlier in a header file. AR models rely on attention mechanisms 80 to reference prior tokens, yet they struggle to maintain such long-range dependencies precisely 65 ; 54 , particularly as the context window becomes saturated with irrelevant information 53 ; 70 . Although attention enables access to past tokens, it does not allow the model to revise earlier tokens to satisfy constraints that emerge later . This limitation becomes pronounced in Fill-in-the-Middle (FIM) tasks 9 , where generated code must remain consistent with both a prefix and a suffix. While AR models can be trained with FIM objectives 85 ; 22 , their architecture cannot natively treat the prefix and suffix as a jointly editable representation. Moreover, this training paradigm does not account for execution semantics that arise as interpreters or compilers process code sequentially. Overall, traditional Autoregressive (AR) models face three key bottlenecks that limit their effectiveness in repository-scale software engineering. First, sequential dependency enforces unidirectional, token-by-token generation, preventing multi-token predictions across positions and disallowing modification of previously generated tokens. This irreversibility causes early mistakes to propagate, often resulting in cascading errors and hallucinations. Second, the computational cost of self-attention scales quadratically ( O(N^{2}) ), making the processing of large, repository-level contexts prohibitively expensive. Finally, AR models exhibit a semantic disconnect by treating code purely as text, lacking an inherent understanding of execution semantics. As a result, they may generate syntactically plausible but functionally incorrect logic that ignores the runtime behavior of the programming environment. While existing literature in automated software engineering remains largely centred on Autoregressive (AR) frameworks 82 ; 31 ; 90 ; 39 ; 18 ; 46 ; 55 , this work aims to broaden the discussion by examining emerging non-autoregressive paradigms relevant to code modelling (Figure 1 ). Specifically, we analyse Diffusion Models, Code World Models (CWMs), and State Space Models (SSMs), highlighting how their architectural properties offer alternative mechanisms for code generation and reasoning (Figure 2 ). To the best of our knowledge, this work presents one of the first surveys that systematically discusses these paradigms within the context of code synthesis and software engineering. We hope this survey encourages further exploration of such approaches within the NLP and Software Engineering communities. We would like to further mention that this work is based on our understanding of open-source models, and we do not intend to comment or speculate on anything about closed-source models, which can potentially also have a component of non-autoregressive models (like diffusion, state space, etc.), for all we know. Our Contributions are: • We present a structured survey of emerging non-autoregressive paradigms for code modeling, focusing on Diffusion Models, State Space Models (SSMs), and Code World Models (CWMs) by outlining their architectural principles and relevance to code tasks. • We conduct a comparative analysis of these paradigms, examining their trade-offs in terms of computational efficiency, scalability to long contexts, and structural consistency in generated code. • We connect these paradigms with insights from cognitive neuroscience, framing these paradigms within a perspective inspired by human “System 2” reasoning that emphasises deliberate analysis and error correction. Figure 1 : This diagram categorises non-autoregressive paradigms for different code-related tasks. Here, Blue represents diffusion-based approaches, green denotes World Modelling, and red corresponds to State Space paradigms. Based on this, we note that diffusion models remain the most explored non-autoregressive paradigm when compared to World Modelling and State Space Models. 2 Diffusion: The Paradigm of Iterative Refinement The emergence of Diffusion Models (DMs) for text generation 67 ; 93 ; 59 and code synthesis 87 ; 24 ; 72 represents a fundamental architectural shift from autoregressive next-token prediction toward iterative refinement of the entire sequence. Unlike AR models, which generate outputs sequentially and condition each step on previously generated tokens, diffusion-based approaches inspired by non-equilibrium thermodynamics 76 , formulate generation as a denoising process that progressively transforms a noisy sequence into a coherent output. The growing maturity of this approach is reflected in recent large-scale and commercial diffusion-based systems, including Seed Diffusion 77 , Mercury 43 , and Gemini Diffusion 25 , which demonstrate the viability of diffusion-based generation for long-form and high-fidelity text and code. 2.1 Foundation of Diffusion Models In this section, we discuss the foundations of diffusion models in the context of discrete data. Unlike image diffusion, which operates in continuous Gaussian space 34 ; 66 ; 14 , code generation requires handling discrete tokens 71 ; 67 ; 5 . Recent advancements