LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding

Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, Zhijie Deng · 2025

Read LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding by Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, Zhijie Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16229.txt.

Introduction

Chenkai Xu and ten co-authors speed up a new kind of language model in a paper posted on arXiv during 2025. Diffusion language models write text by filling in many tokens at once, promising fast output. In practice they manage only one to three tokens per forward pass, because the order in which blanks are filled limits how many can be settled together.

Their algorithm, LoPA, tries several filling orders in parallel branches and keeps the order whose confidence promises the most future parallelism. It needs no training and plugs into existing models. Applied to the D2F model, LoPA raised output to 10.1 tokens per pass on the GSM8K math benchmark while beating the base model's quality. A multi-device system built around branch parallelism reached 1,073.9 tokens per second for a single sample.

Catalog record

CREATORChenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, Zhijie Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, Zhijie Deng Abstract Diffusion Large Language Models (dLLMs) have demonstrated significant potential for high-speed inference. However, current confidence-driven decoding strategies are constrained by limited parallelism, typically achieving only 1–3 tokens per forward pass (TPF). In this work, we identify that the degree of parallelism during dLLM inference is highly sensitive to the Token Filling Order (TFO). Then, we introduce Lo okahead PA rallel Decoding ( LoPA ), a training-free, plug-and-play algorithm, to identify a superior TFO and hence accelerate inference. LoPA concurrently explores distinct candidate TFOs via parallel branches, and selects the one with the highest potential for future parallelism based on branch confidence. We apply LoPA to the state-of-the-art D2F model and observe a substantial enhancement in decoding efficiency. Notably, LoPA increases the TPF of D2F-Dream to 10.1 on the GSM8K while maintaining performance superior to the Dream baseline. Furthermore, to facilitate this unprecedented degree of parallelism, we develop a specialized multi-device inference system featuring Branch Parallelism (BP), which achieves a single-sample throughput of 1073.9 tokens per second under multi-GPU deployment. Github: https://github.com/zhijie-group/LoPA Figure 1 : Throughput performance of LoPA. LoPA accelerates the single-sample throughput for D2F-Dream to up to 1073.9 and 856.5 tokens/s on MBPP and GSM8K respectively, significantly outperforming baselines. More details are provided in Table 7 . 1 Introduction Figure 2 : Overview of Lookahead Parallel Decoding (LoPA) . In each iteration, LoPA generates a anchor branch alongside multiple lookahead branches (e.g., B_{1},\dots,B_{k} ) by independently sampling high-confidence positions from the baseline’s unfilled set. A branch confidence verification mechanism then evaluates all branches in parallel within a single forward pass, selecting the optimal path to maximize future parallelism. Diffusion Large Language Models (dLLMs) ( 27 ; 35 ; 13 ; 32 ; 8 ) have emerged as a highly promising paradigm for text generation. By iteratively refining a full-mask sequence into text tokens, dLLMs decouple generation depth from sequence length, theoretically offering superior potential for inference speed. Recent studies ( 32 ; 8 ) have demonstrated speed surpassing autoregressive (AR) models, thereby providing higher throughput for latency-sensitive applications. Despite this potential, the practical parallelism of state-of-the-art dLLMs remains constrained. Leading models such as Fast-dLLM ( 33 ) , D2F ( 32 ) , and SDAR ( 8 ) employ confidence-driven sampling, which fills tokens exceeding a confidence threshold \tau in each iteration. However, this strategy typically yields only 1–3 tokens per forward pass (TPF) on common tasks such as mathematics and coding, failing to fully unleash the parallel potential of dLLMs. Our investigation traces this limitation to a fundamental sensitivity: parallelism is bounded by prediction confidence and the confidence is heavily influenced by the Token Filling Order (TFO). As observed in LLaDA ( 27 ) , varying TFOs significantly shifts the generative distribution and confidence landscape. Consequently, the standard strategy of greedily prioritizing positions with the highest current confidence may lead to suboptimal trajectories, raising the question: Can we actively explore superior TFOs to maximize future confidence and unlock higher parallelism? To this end, we introduce Lo okahead PA rallel Decoding ( LoPA ), a training-free, plug-and-play algorithm designed to search for TFOs with high future parallelization potential. LoPA operates in three phases per iteration: (1) advancing decoding by sampling an anchor branch B_{0} via standard confidence-driven strategies; (2) exploring distinct TFOs beyond B_{0} by generating k lookahead branches, each sampling from the top- k high-confidence candidate positions to ensure reliable exploration coverage ( 33 ) ; and (3) identifying the optimal path by verifying all k+1 branches in a single forward pass to retain the one with the highest future parallelization potential. By iteratively selecting optimal branches, LoPA can boost overall TPF. We integrate LoPA with D2F ( 32 ) , scaling the TPF of D2F-Dream to 10.1 on GSM8K ( 9 ) while maintaining performance scores surpassing the original Dream baseline, and scaling D2F-DiffuCoder to 8.3 on HumanEval+ ( 6 ; 22 ) with marginal performance degradation. To support LoPA, we co-design a multi-device inference system that distributes branches across devices on multiple platforms, achieving a single-sample throughput of 1073.86 tokens/s. We further validate LoPA’s generalizability by integrating it with Vanilla Dream ( 35 ) . Evaluations demonstrate that LoPA effectively scales dLLM parallelism, establishing a clear, controllable speed-accuracy trade-off. Our contributions are summarized as follows: • We identify TFO as a key factor influencing dLLM parallelism and propose LoPA, a training-free algorithm that looks ahead to optimize TFO. • We demonstrate that LoPA scales the TPF of D2F-Dream to 10.1 on GSM8K and D2F-DiffuCoder to 8.3 on HumanEval+, while maintaining comparable performance. • We develop a specialized Branch Parallel inference system, achieving near-linear scalability and a throughput of 1073.86 tokens/s. 2 Related Work Diffusion Large Language Models (dLLMs). Autoregressive (AR) models ( 1 ; 30 ; 16 ; 21 ) have long dominated text generation, yet their sequential decoding imposes an inherent latency bottleneck. To address this, dLLMs ( 27 ; 35 ; 13 ; 24 ) have emerged as a non-autoregressive paradigm. By iteratively denoising fully masked sequences, dLLMs enable parallel token prediction and leverage bidirectional attention for holistic context modeling. Recent scaling efforts, whether training from scratch ( 27 ) or initializing from pre-trained AR weights ( 35 ) , have yielded dLLMs with performance competitive to state-of-the-art AR models, validating their potential for high-quality, parallel generation. Acceleration of dLLMs. Despite their parallel nature, dLLM inference remains computationally expensive due to multi-step denoising and incompatibility with standard KV caching. Acceleration strategies fall into two primary paradigms. Training-based methods compress sampling steps or restructure generation; for instance, dParallel ( 7 ) applies consistency distillation, while D2F ( 32 ) employs asymmetric distillation to enable block-autoregressive pipelining. Training-free methods optimize inference without weight updates. One avenue adapts KV caching to bidirectional attention via approximate schemes ( 23 ; 33 ; 25 ) . Another focuses on heuristic decoding optimizations, where works like Fast-dLLM ( 33 ) , Prophet ( 19 ) , and Credit Decoding ( 31 ) exploit confidence patterns or early-layer determinism to skip redundant steps. Speculative Decoding. Speculative decoding, a standard for AR acceleration ( 18 ; 5 ; 29 ) , employs efficient draft models for parallel verification. Innovations include tree-based drafting ( 26 ; 20 ) , multi-head structures ( 4 ) , and draft-free fixed-point iterations ( 11 ; 36 ) . In the dLLM domain, speculative concepts have evolved from relying on external AR guidance ( 15 ) to self-verification methods like Spiffy ( 2 ) and Free Draft-and-Verification ( 34 ) . While the latter achieve lossless acceleration by maximizing token acceptance within a fixed generation distribution, our approach fundamentally diverges. Instead of passively verifying a static sequence, LoPA actively explores TFO to discover trajectories with superior future confidence, effectively optimizing the output distribution to unlock parallelism beyond standard greedy limits. 3 Methodology This section first explains the foundational Confidence-Driven Sampling used in regular dLLM inference ( 33 ) and then elaborates on LoPA. 3.1 Preliminary: Confidence-Driven Sampling for dLLMs Confidence-driven sampling is a prevalent paradigm for

Read the complete text