LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection
Haohui Zhang, Zhiye Wang, Xiaoying Gan, Xinbing Wang, Bo Jiang · 2026
Read LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection by Haohui Zhang, Zhiye Wang, Xiaoying Gan, Xinbing Wang, Bo Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10980.txt.
Introduction
Haohui Zhang and four colleagues unlock more parallel work in diffusion language models, which write text by cleaning up many tokens at once. Current models assume tokens turn independent at high confidence, but the strict confidence cutoffs needed for accuracy choke how much work runs side by side.
Token counts show many tokens settle on their right predictions early yet never reach the cutoff, proving the criteria too cautious. LEAP, a training-free plug-in, spots those early-settling tokens with future-context filtering and multi-sequence superposition, then decodes them early with checked reliability. Tests show latency and denoising steps falling well below confidence-based decoding. Inference engineers get faster diffusion decoding without retraining.
Catalog record
| CREATOR | Haohui Zhang, Zhiye Wang, Xiaoying Gan, Xinbing Wang, Bo Jiang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30962 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10980
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection Haohui Zhang, Zhiye Wang, Xiaoying Gan, Xinbing Wang, Bo Jiang Abstract Diffusion Language Models (dLLMs) have garnered significant attention for their potential in highly parallel processing. The parallel capabilities of existing dLLMs stem from the assumption of conditional independence at high confidence levels, which ensures negligible discrepancy between the marginal and joint distributions. However, the stringent confidence thresholds required to preserve accuracy severely constrain the scalability of parallelism. Through systematic token-level statistical analysis, we reveal that a substantial proportion of tokens converge to their correct predictions early in the denoising process yet fail to reach standard confidence thresholds, confirming that current confidence-based criteria are overly conservative. In response, we introduce LEAP (Lookahead Early-Convergence Token Detection for Accelerated Parallel Decoding). LEAP is a training-free, plug-and-play method that leverages future context filtering and multi-sequence superposition to detect early-converging tokens. By validating the alignment between early convergence and correctness, we enable reliable early decoding of these tokens. Benchmarking across diverse domains demonstrates that LEAP significantly lowers inference latency and decoding steps. Compared to confidence-based decoding, the average number of denoising steps is reduced by about 30%. On the GSM8K dataset, combining LEAP with dParallel accelerates decoding to 7.2 tokens per step while preserving model precision. LEAP effectively breaks the reliance on high-confidence priors, offering a novel paradigm for parallel decoding. Figure 1: Illustration of the ‘early convergence’ phenomenon in diffusion language models. The figure displays the denoising generation process from T=0 to T=N . In confidence-based decoding strategy, only tokens with high confidence are decoded (marked in green). The red box highlights the token "generate," which is correctly predicted early at T=1 and remains stable throughout subsequent steps. However, it fails to be decoded until the final stages because its low confidence score, demonstrating the limitation of confidence-based decoding. 1 Introduction Autoregressive large language models (AR-LLMs) have long dominated the field of language modeling ( 1 ; 24 ; 15 ) . However, their inherent sequential generation process restricts further improvements in inference speed. Consequently, diffusion large language models (dLLMs) have emerged as a promising new paradigm ( 18 ; 4 ; 25 ; 14 ) , garnering significant attention for their potential to highly parallel generation. Recent works have achieved inference speeds exceeding those of AR-LLMs by mitigating per-step denoising costs and scaling the number of tokens processed simultaneously ( 21 ; 16 ; 14 ; 22 ) . While the potential for high-speed generation is well-established, the actual parallelism of the current model is still relatively low. Existing parallel decoding methods generally select multiple high-confidence tokens using marginal probabilities ( 22 ) . Yet, such parallelization assumes token independence. Decoupling interdependent tokens often violates their semantic dependencies, resulting in performance drops, especially in reasoning scenarios requiring strict logical coherence. As a result, state-of-the-art models are restricted to decoding fewer tokens per step, limiting the extensibility of parallel generation. The fundamental reason for the precision degradation in high parallelism scenarios is the discrepancy between independent sampling from marginal probabilities and sampling from the true joint distribution during parallel decoding. Parallel sampling that ignores dependencies can introduce significant bias. Existing methods mitigate this by leveraging high-confidence tokens ( 22 ) —which satisfy independence assumptions. While confidence-based sampling targets high-confidence tokens to minimize inter-dependency issues, the practical scarcity of such tokens limits the effective degree of parallelism. Relaxing the confidence threshold induces sampling bias due to dependencies, resulting in significant accuracy degradation. We identify the limitations as the reliance on confidence metrics, which manifest in two primary aspects. First, confidence-based decoding methods maintain model accuracy by limiting the parallel candidates to only high-confidence tokens. However, our token-level statistical analysis reveals that numerous medium-confidence tokens already converge to their correct predictions early in the denoising process, indicating that high confidence is not a necessary condition for safe decoding. Second, high-confidence tokens inherently contribute less information, leading to a greater number of total decoding steps ( 9 ) . Consequently, high-confidence decoding yields a low-information context for subsequent steps, further inhibiting the potential for parallelism. Therefore, a key challenge remains: how to simultaneously expand parallelism and increase information contribution per step to minimize iteration steps, without sacrificing accuracy? We introduce L ookahead E arly-Convergence Detection for A ccelerated P arallelism Decoding (LEAP), a training-free, plug-and-play parallel decoding strategy. We find empirically that a high proportion of medium-confidence tokens exhibit early correctness and convergence, which implies that a large number of forward steps are performing repeated predictions on them. These tokens show low sensitivity—their predictions stabilize early and demonstrate robustness to future contextual changes. Capitalizing on this, we propose a convergence detection strategy based on future context perturbation. By contrasting the predictions of the current context with those of a superposed context containing potential future information, we identify tokens that exhibit low sensitivity and high robustness to future updates, thereby enabling their early decoding. The feasibility of this strategy stems from the novel future context candidate pruning and multi-sequence superimposed consistency detection strategy we propose. Decode these medium-confidence tokens in advance not only enhances the parallelism in current step, but also triggers further token generation and amplifies parallelism of future due to their higher entropy and information contribution. We conduct extensive evaluations on two popular open-source dLLMs, LLaDA and Dream, covering mathematics, code generation, and multi-disciplinary QA. Empirical results show that LEAP improves generation parallelism across all benchmarks, decreasing latency by around 30% against confidence-based decoding strategy. Meanwhile, LEAP slightly improves average accuracy on LLaDA. Further analysis confirms that LEAP establishes a better Pareto frontier between speed and accuracy compared to confidence-based decoding strategy. Figure 2: (a) Confidence distribution of early decodable tokens for GSM8K with LLaDA-8B-Instruct. The red line denotes Early Correct, and the blue line denotes Early Correct & Converged. (b) Confidence distribution of ground-truth tokens at the preceding time step. The histogram and red curve represent the probability density and CDF, respectively. The annotation (Cum. P=0.1,x\approx 0.32 ) indicates that only 10% of tokens have confidence below 0.32. Figure 3: Overview of LEAP. Given a partially denoised sequence at step t-1 , LEAP first performs future-context candidate pruning: for each masked position, only plausible future tokens whose confidence exceeds a loose threshold \eta are retained. These candidates, together with copied mask tokens, are appended to the original sequence while preserving their original position IDs, forming a superposed context x_{t}^{sup} . This allows LEAP to compare predictions under the original context and lookahead-perturbed context within a single forward pass. At step t , a token is decoded early only if its prediction remains consistent under the two contexts and its confidence exceeds \tau ; otherwise, it stays masked for later refinement. In the example, (B) and (D) satisfy the consistency check and are unmasked early, whereas (A) is kept masked. 2 Related Work Diffusion Language Models. Recently, diffusion-based Large Language Models (dLLMs) have evolved into a distinct paradigm of high-performance foundation models, diverging from the standard autoregressive framework. Open-source efforts, notably the LLaDA series, have validated pure diffusion architectures trained from scratch; LLaDA ( 18 ) and LLaDA 2.0 ( 4 ) leverage masked prediction to match autoregressive baselines (e.g., LLaMA 3 ( 8 ) )