Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion

Chien Van Nguyen, Chaitra Hegde, Van Cuong Pham, Ryan A. Rossi, Franck Dernoncourt, Thien Huu Nguyen · 2026

Read Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion by Chien Van Nguyen, Chaitra Hegde, Van Cuong Pham, Ryan A. Rossi, Franck Dernoncourt, Thien Huu Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12825.txt.

Introduction

Chien Van Nguyen and five colleagues introduce Orthrus, joining autoregressive and diffusion generation. Autoregressive decoding is one-by-one, slowing use, while diffusion models generate in parallel but degrade quality. Orthrus adds a lightweight module to a frozen model, creating a second view beside the first view. Both views share the Key-Value cache, with the first head pre-filling and the second head generating. A consensus check ensures lossless use, delivering 7.8x speedup at flat cache cost.

Catalog record

CREATORChien Van Nguyen, Chaitra Hegde, Van Cuong Pham, Ryan A. Rossi, Franck Dernoncourt, Thien Huu Nguyen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41516
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12825

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion Chien Van Nguyen, Chaitra Hegde, Van Cuong Pham, Ryan A. Rossi, Franck Dernoncourt, Thien Huu Nguyen Abstract We introduce Orthrus , a simple and efficient dual-architecture framework that unifies the exact generation fidelity of autoregressive Large Language Models (LLMs) with the high-speed parallel token generation of diffusion models. The sequential nature of standard autoregressive decoding represents a fundamental bottleneck for high-throughput inference. While diffusion language models attempt to break this barrier via parallel generation, they suffer from significant performance degradation, high training costs, and a lack of rigorous convergence guarantees. Orthrus resolves this dichotomy natively. Designed to seamlessly integrate into existing Transformers, the framework augments a frozen LLM with a lightweight, trainable module to create a parallel diffusion view alongside the standard autoregressive view. In this unified system, both views attend to the exact same high-fidelity Key-Value (KV) cache; the autoregressive head executes context pre-filling to construct accurate KV representations, while the diffusion head executes parallel generation. By employing an exact consensus mechanism between the two views, Orthrus guarantees lossless inference, delivering up to a 7.8 × 7.8\times speedup with only an O ⁡ ( 1 ) O(1) memory cache overhead and minimal parameter additions. We release the code at https://github.com/chiennv2000/orthrus . 1 Introduction Autoregressive (AR) Large Language Models (LLMs) are currently the predominant architecture in natural language processing, demonstrating robust performance across a diverse set of complex reasoning and generation tasks ( 28 ; 6 ; 27 ; 30 ; 1 ; 16 ) . However, AR models suffer from a fundamental inefficiency during the decoding phase. While the pre-filling stage processes prompt tokens in parallel by leveraging self-attention, the generation phase computes tokens strictly sequentially. This one-by-one generation creates a memory-bandwidth bottleneck, leading to hardware underutilization and high inference latency. Diffusion Language Models (DLMs) ( 26 ; 3 ; 37 ; 33 ) natively bypass this bottleneck by generating blocks of tokens in parallel. Despite providing significant inference speedups, DLMs consistently underperform AR models of a similar scale and require massive training datasets to achieve baseline coherence. Recent approaches attempt to adapt pre-trained AR models into diffusion models to bridge this quality gap ( 19 ; 31 ) . However, these adaptations remain computationally expensive, often requiring continuous pre-training up to 500B tokens, and still fail to match the exact predictive distribution of the original AR models due to architectural divergence. To overcome this dichotomy, we propose resolving the trade-off at the fundamental architectural level by unifying the strengths of both paradigms within a single Transformer . We introduce Orthrus, a novel dual-architecture framework designed to natively support parallel generation without sacrificing the exact predictive distribution of the base autoregressive model. The core architectural insight of Orthrus is that the AR bottleneck is strictly confined to the generation phase; its self-attention mechanism remains optimal for building context representations. Consequently, Orthrus freezes the pre-trained AR model and utilizes its standard forward pass exclusively during the pre-filling stage to compute a high-fidelity Key-Value (KV) cache. To enable high-speed parallel generation, we structurally augment the network by integrating a lightweight, trainable diffusion module directly alongside the AR attention heads. This structural unification allows both views to operate over the exact same context, inherently resulting in zero redundant cache overhead. During generation, the diffusion head conditions directly on the high-quality KV cache constructed by the AR head to generate multiple future tokens in parallel. To strictly guarantee lossless inference, the framework incorporates an intrinsic two-head consensus mechanism: token trajectories generated by the diffusion view are structurally validated by the frozen AR view, guaranteeing that the final output strictly matches the base model’s exact predictive distribution. By decoupling the parallel generation mechanism from the sequential constraints of the base model, Orthrus achieves exact inference parity at significantly accelerated speeds. In summary, our main contributions are: • A Novel Dual-Architecture Framework: We introduce Orthrus, a structural unification that embeds a parallel diffusion module within a standard AR Transformer, allowing both views to operate over a shared KV cache with zero redundant historical KV cache storage. Using intra-model consensus, it preserves the exact predictive distribution of the base LLM, ensuring strictly lossless generation that outperforms prior diffusion adaptations. • Significant Inference Acceleration: By natively exploiting the diffusion head for parallel token generation, Orthrus successfully breaks the sequential bottleneck, delivering up to a 7.8\times speedup. • Extreme Parameter and Memory Efficiency: The architectural integration is highly lightweight. Parallel capabilities can be injected into strong AR baselines by fine-tuning only 16% of the total model parameters using less than 1B tokens (requiring under 24 hours on a single 8xH200 node). 2 Preliminaries To contextualize the architectural design of our proposed framework, we formalize the distinct probability modeling paradigms of Autoregressive (AR) and Masked Diffusion Language Models (MDMs). This formulation isolates the mathematical trade-off between generation quality and inference speed, establishing the foundation for our structural unification. 2.1 Autoregressive and Diffusion Paradigms Autoregressive Language Modeling. AR models learn the true data distribution by factorizing the joint probability of a sequence \mathbf{x}=(x_{1},x_{2},\dots,x_{N}) using the exact chain rule of probability p_{\text{AR}}(\mathbf{x})=\prod_{i=1}^{N}p_{\theta}(x_{i}\mid\mathbf{x}_{<i}) . The model parameters \theta are typically optimized via the negative log-likelihood over the data distribution \mathcal{D} : \mathcal{L}_{\text{AR}}(\theta)=-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}}\left[\sum_{i=1}^{N}\log p_{\theta}(x_{i}\mid\mathbf{x}_{<i})\right] (1) By imposing no conditional independence assumptions, this formulation ensures each token x_{i} is strictly conditioned on the entire preceding trajectory. While this causal dependency achieves state-of-the-art fidelity, it mandates sequential sampling. During inference, generating N tokens requires N distinct forward passes, repeatedly loading the Key-Value (KV) cache creating a fundamental, memory-bandwidth-bound bottleneck ( 22 ; 2 ; 18 ) . Masked Diffusion Language Models. Diffusion Language Models (DLMs) bypass the sequential bottleneck by framing generation as a parallel denoising process. Given a historical context \mathbf{c}=\mathbf{x}_{\leq t} and a corrupted block of future tokens \mathbf{y}^{t} , the reverse process trains a network parameterized by \phi to predict the original tokens \mathbf{y}^{0} simultaneously: \mathcal{L}_{\text{MDM}}(\phi)=-\mathbb{E}_{\mathbf{x}\sim\mathcal{D},t,\mathbf{y}^{t}}\left[\sum_{k\in\mathcal{M}}\log p_{\phi}(y_{k}^{0}\mid\mathbf{c},\mathbf{y}^{t})\right] (2) where \mathcal{M} is the set of masked indices. For highly accelerated inference (where denoising steps T\ll|\mathcal{M}| ), the model relies on a strong conditional independence assumption: p_{\text{DLM}}(\mathbf{y}^{0}\mid\mathbf{c})\approx\prod_{k\in\mathcal{M}}p_{\phi}(y_{k}^{0}\mid\mathbf{c},\mathbf{y}^{t}) (3) While this formulation heavily amortizes memory-bandwidth costs by computing the entire block in a single forward pass, it inherently violates the strict causal dependency of the autoregressive model. Because the prediction of token y_{k} does not condition on the exact, realized token y_{k-1} , the joint probability distribution modeled by the DLM drifts from the true AR target distribution ( 24 ; 9 ; 31 ) . 2.2 The Limits of Adaptation and Structural Unification To mitigate the high computational costs of training DLMs from scratch, recent works explore adapting pre-trained AR models into diffusion frameworks ( 29 ; 14 ; 31 ; 10 ; 36 ) . These approaches repurpose the robust representations of AR baselines by fine-tuning them on block-wise masked diffusion objectives (Equation 2 ). While these methods transition the model from sequential to parallel generation, adaptation fundamentally alters the base model, introducing severe performance trade-offs. This distributional drift is particularly catastrophic for reasoning-heavy tasks: during long-horizon generation, conditional errors compound rapidly, causing severe performance degradation. For instance, state-of-the-art adaptations like Fast-dLLM-v2 ( 31 ) suffers an 11-point accuracy drop on MATH-500 ( 17 )

Read the complete text