From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons

Xiangyu Ma, Teng Xiao, Zuchao Li, Lefei Zhang · 2026

Read From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons by Xiangyu Ma, Teng Xiao, Zuchao Li, Lefei Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27387.txt.

Introduction

Xiangyu Ma, Teng Xiao, Zuchao Li, and Lefei Zhang adapt autoregressive language models to diffusion-style parallel generation. Their FLUID framework aligns strictly causal generation with GPT-style checkpoints, avoiding training from scratch. Elastic horizons then vary denoising strides with local information density, reaching top performance at far lower training cost.

Catalog record

CREATORXiangyu Ma, Teng Xiao, Zuchao Li, Lefei Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41443
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons Xiangyu Ma, Teng Xiao, Zuchao Li, Lefei Zhang Abstract Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models. This incompatibility precludes reusing robust AR priors, necessitating prohibitive pre-training from scratch. To bridge this gap, we propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm. By enforcing Strictly Causal Alignment, FLUID enables seamless initialization from standard GPT-style checkpoints, circumventing the need for massive pre-training. Furthermore, we introduce Elastic Horizons, an entropy-driven mechanism that dynamically modulates denoising strides based on local information density rather than fixed schedules. Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude, effectively reconciling established AR foundations with efficient parallel generation. Our code is available at https://huggingface.co/MYTH-Lab/FLUID . 1 Introduction Autoregressive (AR) language models, trained via the next-token prediction paradigm, constitute the cornerstone of modern natural language processing. By conditioning each token solely on its preceding context, AR models ensure rigorous logical consistency and training stability, powering breakthroughs in reasoning-intensive tasks such as code generation and mathematics ( 12 ) . However, this strictly sequential formulation imposes a bottleneck: inference latency grows linearly with sequence length ( 26 ; 31 ; 40 ) . As model scales and context windows expand, this serial decoding cost increasingly dominates deployment budgets, creating an urgent demand for parallel generation paradigms ( 36 ; 6 ) . To transcend this limitation, Discrete Diffusion Models (DLMs) have emerged as a promising alternative, offering the capability to generate multiple tokens in parallel through iterative denoising ( 16 ) . Approaches such as LLaDA ( 29 ) and Dream ( 37 ) demonstrate that diffusion can model global context effectively. However, standard diffusion models typically rely on bidirectional attention mechanisms. While effective for global coherence, this architecture creates a critical structural mismatch with the pre-trained priors of ubiquitous AR models (e.g., Llama, GPT). This incompatibility precludes the efficient reuse of existing checkpoints, necessitating computationally prohibitive pre-training from scratch. Figure 1: Causal mismatch in fixed-size block diffusion. First-pass accuracy decays significantly faster for high-entropy data (e.g., complex reasoning steps in MATH500) than low-entropy text (e.g., GSM8k) as lookahead increases. Recent “Block Diffusion” strategies ( 1 ; 8 ) attempt to bridge this gap by adopting a “semi-autoregressive” hybrid design: sequences are partitioned into fixed blocks, inheriting AR’s serial dependency between blocks while applying bidirectional diffusion within them. While this mitigates some efficiency bottlenecks, we identify a critical limitation in the fixed-size nature of these strategies. Despite these advancements, we identify a critical limitation in fixed-size block diffusion: the Entropy-Horizon Dilemma . As illustrated in Figure 1 , rigid generation windows fundamentally misalign with the dynamic entropy of natural language. Text exhibits varying information density, oscillating significantly between deterministic functional phrases and high-uncertainty reasoning steps. In high-entropy regions (e.g., MATH500), large blocks cause a rapid decay of the “causal horizon,” necessitating aggressive error correction that negates parallelization gains. Conversely, in low-entropy regions (e.g., GSM8K), conservative blocks fail to fully exploit parallel decoding potential. This static approach decouples generation cadence from the intrinsic semantic rhythm, limiting both inference speed and quality. To resolve these structural and dynamic mismatches, we propose FLUID (Flexible Unidirectional Inference Diffusion), a novel framework designed to efficiently adapt AR models into parallel diffusers. Unlike traditional diffusion models that enforce bidirectional dependencies, FLUID introduces Strictly Causal Alignment by imposing a unidirectional attention mask during the diffusion process. This design realigns the generative mechanism with the inductive biases of pre-trained AR backbones, enabling seamless initialization and efficient fine-tuning without the need for massive pre-training. Furthermore, we replace fixed block boundaries with Elastic Horizon Modeling, a mechanism driven by real-time entropy estimation. This allows FLUID to dynamically modulate the generation window—accelerating through high-confidence segments while allocating dense computation to complex reasoning steps—thereby achieving a seamless balance between inference speed and generation quality. Comprehensive Experiments across general understanding, mathematical reasoning, and code generation demonstrate that FLUID matches the performance of top-tier autoregressive models while significantly outperforming existing bidirectional diffusion baselines. By enforcing a unidirectional attention mask to realign with the inductive biases of pre-trained backbones, our framework validates strictly causal diffusion as a paradigm that superiorly balances training efficiency, inference latency, and logical consistency. This alignment—enabling seamless initialization from standard GPT-style checkpoints to reduce training costs by orders of magnitude—theoretically facilitates native KV cache support, establishing a decisive efficiency advantage over non-linear bidirectional diffusers. Leveraging an entropy-driven Elastic Horizon mechanism, FLUID dynamically modulates its generation window, accelerating through predictable segments while ensuring high-fidelity reasoning in complex transitions to effectively mitigate the semantic fracture inherent in fixed-size strategies. Ultimately, our work establishes a scalable foundation for transforming established AR foundations into efficient parallel diffusers without compromising structural integrity or logical coherence. Figure 2: From Autoregressive to FLUID: Evolution of four language model generation paradigms. The diagram highlights the pivotal transition to FLUID (d), which integrates Unidirectional Attention to maintain strictly causal alignment with AR priors, and Dynamic Mask Blocks to adaptively modulate generation horizons based on sequence entropy, resolving the structural and dynamic mismatches inherent in prior paradigms. 2 Related Work 2.1 Discrete Diffusion for Language Modeling DLMs offer a non-autoregressive alternative to traditional modeling by iteratively denoising discrete tokens ( 2 ; 21 ) . Utilizing masking or absorbing states, DLMs enable parallel generation and mitigate exposure bias ( 42 ; 39 ) . While initially computationally demanding ( 15 ) , recent Masked Diffusion Models have bridged this efficiency gap. 25 matched GPT-2 performance, and subsequent scaling to the billion-parameter level—exemplified by LLaDA ( 29 ) and Mercury Coder ( 19 ) —has achieved parity with strong AR baselines like LLaMA3 ( 32 ) , establishing DLMs as a scalable foundation for language generation. 2.2 Adapting Autoregressive Models to Diffusion To circumvent prohibitive pre-training costs, recent research adapts pre-trained AR backbones to diffusion, leveraging robust priors to reframe sequential generation as parallel denoising. DiffuLLaMA ( 14 ) pioneered this by relaxing causal masking and applying parameter-efficient fine-tuning. Subsequent hybrid models like SDAR ( 8 ) integrate AR consistency with diffusion refinement to ensure scalable quality. Meanwhile, efficiency-focused methods such as Fast-DLLM ( 34 ) and Fast-DLLM-v2 ( 33 ) mitigate inference bottlenecks via training-free acceleration. Collectively, these works validate the computational efficiency of transforming AR foundations into robust diffusion decoders. 2.3 Block-Wise Adaptation and Hybrid Approaches While capturing global context, full-sequence diffusion incurs high computational overhead and misaligns with the causal inductive biases of AR pre-training. Although recent caching mechanisms ( 24 ; 27 ; 34 ) attempt to alleviate these inefficiencies, they do not fundamentally alter the generation paradigm. To enable scalable generation, Semi-Autoregressive (or Block-Wise) approaches ( 17 ; 1 ) partition sequences into segments, denoising the current block given committed priors. While balancing quality and speed, these methods typically rely on rigid boundaries and bidirectional attention, restricting dynamic adaptability. 3 Preliminaries Consider a sequence of tokens \mathbf{x}=[x_{1},\dots,x_{L}] from a vocabulary \mathcal{V} . Let \mathcal{D} denote the data distribution. We aim to learn a model p_{\theta} that approximates \mathcal{D} . Figure 3: The Schematic Framework of FLUID. The

Read the complete text