Evo: Autoregressive–Diffusion Large Language Models with Evolving Balance

Junde Wu, Minhao Hu, Jiayuan Zhu, Yuyuan Liu, Tianyi Zhang, Kang Li, Jingkun Chen, Jiazhen Pan, Min Xu, Yueming Jin · 2026

Read Evo: Autoregressive–Diffusion Large Language Models with Evolving Balance by Junde Wu, Minhao Hu, Jiayuan Zhu, Yuyuan Liu, Tianyi Zhang, Kang Li, Jingkun Chen, Jiazhen Pan, Min Xu, Yueming Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06617.txt.

Introduction

Wu and team bring Evo, a latent path model bridging autoregressive and diffusion-based language generation within a generative plan. Each token carries a code growing over a growth variable showing maturity. Low values give confident AR-like polish, and high values invoke spread-style planning. AR and spread models emerge from slicing a shared probability flow, with Evo's training objective derived from a varied ELBO. Evo 8B posts top results, highly competitive, on 15 diverse benchmarks spanning reasoning, code generation, and general language grasp, with fast inference kept.

Catalog record

CREATORJunde Wu, Minhao Hu, Jiayuan Zhu, Yuyuan Liu, Tianyi Zhang, Kang Li, Jingkun Chen, Jiazhen Pan, Min Xu, Yueming Jin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52183
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06617

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evo: Autoregressive–Diffusion Large Language Models with Evolving Balance Junde Wu, Minhao Hu, Jiayuan Zhu, Yuyuan Liu, Tianyi Zhang, Kang Li, Jingkun Chen, Jiazhen Pan, Min Xu, Yueming Jin Abstract We introduce Evo , a duality latent trajectory model that bridges autoregressive (AR) and diffusion-based language generation within a continuous evolutionary generative framework. Rather than treating AR decoding and diffusion generation as separate paradigms, Evo reconceptualizes text generation as a latent flow: each token is associated with a vector-valued embedding that evolves over a progression variable t i ∈ [ 0 , 1 ] t_{i}\in[0,1] , indicating its semantic maturity. Low t i t_{i} values correspond to confident AR-like refinement, while high values invoke diffusion-style planning, allowing the model to adaptively balance AR and diffusion based on uncertainty. Theoretically, we show that both AR and diffusion models emerge as discretizations of a shared probability flow, and we derive Evo’s training objective from a unified variational ELBO. The model is implemented as a time-conditioned Transformer governed by a shared vector field, trained end-to-end to jointly infer latent codes and their progression times. During decoding, Evo performs efficient, semantics-aware refinement, achieving high-quality outputs without sacrificing speed. Empirically, Evo 8B achieves state-of-the-art or highly competitive results on 15 diverse benchmarks, including reasoning (GSM8K, ARC-C), code generation (HumanEval, MBPP), and general language understanding, while maintaining fast inference speed. Our results demonstrate that Evo delivers a new paradigm for LLM design with strong generation quality, robust symbolic reasoning, and decoding efficiency. 1 Introduction Large Language Models (LLMs) have emerged as a transformative paradigm in natural language processing, demonstrating remarkable capabilities in open-ended generation, reasoning, and multi-turn dialogue 24 ; 20 . From GPT-series to instruction-tuned variants like ChatGPT and Claude, the predominant architecture driving these successes is the autoregressive (AR) Transformer, which models the conditional distribution of a sequence token-by-token in a left-to-right manner 15 ; 16 . This causal decoding framework has enabled scalable pretraining, fine-tuned alignment, and effective sampling. Yet, despite their ubiquity, AR-based LLMs exhibit the limitations that they operate in a strictly unidirectional generation regime and often exhibit compounding errors due to greedy or approximate decoding heuristics. To address these challenges, a growing body of work has explored diffusion-based LLMs, generative models that synthesize text by iteratively denoising corrupted inputs 27 ; 7 . Inspired by their success in image and audio domains, diffusion models for language seek to replace one-pass AR decoding with a multi-step process that allows for non-sequential token generation. These models offer promising benefits, such as iterative self-correction and better global coordination 23 ; 26 ; 22 ; 1 ; 13 ; 21 . However, diffusion models often require extensive inference steps, lack explicit control over high-level semantics, and generally underperform AR models in perplexity due to lossy training objectives and the absence of strategic planning. Figure 1: Evo generates text by evolving a diffusion-based semantic scaffold into token-level autoregressive realizations along a shared latent trajectory, allowing different tokens to reach semantic maturity at different times. Unlike prior autoregressive, diffusion, and AR+diffusion models, which either generate strictly left-to-right or apply diffusion in a fixed autoregressive order, Evo adaptively balances diffusion-style global planning and autoregressive next-token prediction by semantic granularity, selectively refining mature tokens while continuing to finalize the remaining scaffold within the same sentence. In this paper, we introduce Evo, a duality latent flow model that generalizes autoregressive (AR) and diffusion-based language generation within a single, continuous generative framework. Different from existing AR–diffusion hybrids, we take autoregressive and diffusion generation as two manifestations of the same underlying diffusion process at different progression times in latent space. This inherent duality directly motivates Evo’s unique design, allowing it to adaptively modulate the balance between AR-style refinement and diffusion-style planning based on the desired level of generation granularity, thereby jointly improving semantic clarity and detail richness. An illustration is shown in Fig. 1 . Specifically, at the core of our framework lies a theoretical unification of AR and diffusion. We show both are shown to emerge as discretizations of a shared probabilistic flow. We formalize this connection by defining latent trajectories governed by a time-indexed vector field, and demonstrate that AR generation corresponds to deterministic flows near the origin, while diffusion-based decoding arises as stochastic score-following in the limit. This continuous perspective allows Evo to model generation as a structured path over latent states, with token-level progression times {t_{i}} that control how far each representation lies along the semantic continuum from planning to realization. Built upon this foundation, Evo consists of a time-conditioned latent generator and a shared vector field \mathcal{F}_{\theta} that governs semantic evolution. The model is trained via a variational objective that infers both latent embeddings and their associated time-steps, optimizing a tight evidence lower bound (ELBO) over the sequence likelihood. The resulting training regime unifies next-token prediction and score-based denoising within a single differentiable loss, enabling end-to-end learning with theoretical guarantees. As a special case, we show that Evo recovers both standard AR modeling and diffusion objectives, and converges toward maximum likelihood estimation under mild regularity assumptions. Empirically, Evo 8B achieves state-of-the-art or highly competitive results on 15 diverse benchmarks, including reasoning (GSM8K, ARC-C), code generation (HumanEval, MBPP), and general language understanding, while maintaining fast inference speed. 2 Approach 2.1 AR-Diffusion duality: Two Sides of the Same Coin from an evolving perspective We propose a unified framework for explicit generative modeling, in which both autoregressive models and diffusion models are interpreted as structured paths over latent states governed by probabilistic transitions. Formally, let \mathbf{x}\in\mathbb{R}^{d} be a data sample drawn from the unknown distribution p_{\text{data}}(\mathbf{x}) . We define a latent Markov path \mathbf{z}^{(0)},\mathbf{z}^{(1)},\dots,\mathbf{z}^{(L)} such that \mathbf{z}^{(L)}=\mathbf{x} , and posit that the generative model decomposes as p_{\theta}(\mathbf{z}^{(0{:}L)})=p_{0}(\mathbf{z}^{(0)})\prod_{\ell=1}^{L}p_{\theta}(\mathbf{z}^{(\ell)}\mid\mathbf{z}^{(\ell-1)}), (1) where p_{0} is a known initialization distribution (e.g., standard Gaussian), and p_{\theta} parameterizes each transition. The interpretation of the intermediate states \{\mathbf{z}^{(\ell)}\} and the design of transitions determine whether the resulting model behaves like an autoregressive generator or a diffusion sampler. In the case of autoregressive models, the intermediate states correspond to partial completions of the sequence. For a sequence \mathbf{x}=(x_{1},\dots,x_{T}) , we set \mathbf{z}^{(t)}=(x_{1},\dots,x_{t}) and recover the familiar left-to-right factorization p_{\theta}(\mathbf{x})=\prod_{t=1}^{T}p_{\theta}(x_{t}\mid x_{<t}), (2) which corresponds to a deterministic, monotonic path in the space of prefix sequences. This process can be lifted into a continuous-time representation by interpolating the discrete sequence trajectory into a smooth path governed by an ODE. Specifically, we define a function \mathbf{x}_{s} over s\in[0,1] , where \mathbf{x}_{0}=\mathbf{0} and \mathbf{x}_{1}=\mathbf{x} , and assume it evolves according to \frac{d\mathbf{x}_{s}}{ds}=\mathcal{F}_{\theta}(\mathbf{x}_{s},s), (3) with \mathcal{F}_{\theta} being a parameterized vector field that controls how information is revealed along the generation path. This framing renders autoregression as a flow through latent space, aligning with recent formulations of continuous normalizing flows and neural ODEs. In the case of diffusion models, recall that the forward transition is defined as q(\mathbf{z}^{(t)}\mid\mathbf{z}^{(t-1)})=\mathcal{N}(\sqrt{\alpha_{t}}\mathbf{z}^{(t-1)},(1-\alpha_{t})\mathbf{I}), (4) with a fixed variance schedule \beta_{t}=1-\alpha_{t} . This forward process admits an analytic marginal q(\mathbf{z}^{(t)}\mid\mathbf{x}) , enabling training of a reverse model p_{\theta}(\mathbf{z}^{(t-1)}\mid\mathbf{z}^{(t)}) that approximates the denoising transition. When discretized with sufficiently small noise steps, the forward process converges to an SDE d\mathbf{z}_{t}=\mu(t)\,\mathbf{z}_{t}\,dt+\sigma(t)\,d\mathbf{w}_{t}, (5) and the corresponding reverse-time dynamics satisfy d\mathbf{z}_{t}=\left[-\sigma(t)^{2}\nabla_{\mathbf{z}}\log p_{t}(\mathbf{z})\right]dt+\sigma(t)d\bar{\mathbf{w}}_{t}, (6) where p_{t} is the time-marginal density and d\bar{\mathbf{w}}_{t} is a reverse-time Brownian motion.

Read the complete text