Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast
Jinyuan Feng, Xin Yu, Yiqun Chen, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu · 2026
Read Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast by Jinyuan Feng, Xin Yu, Yiqun Chen, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01373.txt.
Introduction
Jinyuan Feng and seven co-authors study diffusion language models, which write by cleaning up noisy text step by step. These models see the whole passage at once, yet common decoding habits favor local cues and waste that wide view, which lowers quality.
The team finds that high-information tokens, the words that carry the most content, help a lot when fed back as context, and they settle early in decoding. Their method, FoCore, needs no training: it briefly hides those core tokens and uses them as negative samples to steer writing by contrast. A faster variant decodes stable nearby tokens in parallel once core tokens settle. On math, code, and logic tests, and on two model families, FoCore lifts quality, and the fast variant cuts decoding steps by more than half.
Catalog record
| CREATOR | Jinyuan Feng, Xin Yu, Yiqun Chen, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01373
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast Jinyuan Feng, Xin Yu, Yiqun Chen, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, Zhiqiang Pu Abstract The iterative denoising paradigm of Diffusion Large Language Models (DLMs) endows them with a distinct advantage in global context modeling. However, current decoding strategies fail to leverage this capability, typically exhibiting a local preference that overlooks the heterogeneous information density within the context, ultimately degrading generation quality. To address this limitation, we systematically investigate high-information-density (HD) tokens and present two key findings: (1) explicitly conditioning on HD tokens substantially improves output quality; and (2) HD tokens exhibit an early-decoding tendency, converging earlier than surrounding tokens. Motivated by these findings, we propose Focus on the Core (FoCore) , a training-free decoding strategy that utilizes HD tokens in a self-contrast manner, wherein HD tokens are temporarily remasked as negative samples, to guide generation. We further introduce FoCore_Accelerate (FoCore_A) , an efficient variant that, upon detecting HD token convergence, performs parallel decoding over stable candidates within a local context window, substantially accelerating generation. Extensive experiments on math, code and logical reasoning benchmarks demonstrate that FoCore consistently improves generation quality and efficiency across both LLaDA and Dream backbones. For instance, on HumanEval, FoCore improves pass@1 from 39.02 to 42.68 over standard Classifier-Free Guidance, while FoCore-A reduces the number of decoding steps by 2.07× and per-sample latency from 20.76s to 8.64s (-58.4%). 1 Introduction By replacing sequential generation with iterative denoising, diffusion large language models (DLMs) present an alternative to autoregressive (AR) architectures, enabling global context modeling and flexible decoding ( 33 ; 41 ) . However, this advantage is frequently bottlenecked by locally-biased decoding strategies: candidate tokens in closer proximity to already-denoised positions tend to receive disproportionately higher confidence scores ( 14 ; 5 ) . This locally-biased paradigm fundamentally neglects the heterogeneous information density inherent in natural language, as it confines the model’s attention to a narrow positional neighborhood, failing to capture core reasoning tokens within the global context ( 31 ) . Therefore, rather than passively sampling high-confidence tokens locally, proactively prioritizing core reasoning units is essential to fully unleash the contextual capabilities of DLMs. In this paper, we explore how core reasoning tokens influence generation quality through a novel perspective of "Heterogeneous Information Density" within DLMs. Our analysis reveals a significant heterogeneity in the information density of contextual tokens, where the majority of ordinary tokens function merely as redundant background padding, designated as "low-information-density (LD)" tokens , whereas a minority of core tokens act as logical anchors with exceptionally "high-information-density", termed HD tokens . During the denoising processes of DLMs, HD tokens serve as pivotal anchors that steer the reasoning toward correct trajectories. As illustrated by the qualitative analysis in Figure 1 (a), HD tokens effectively constrain the denoising process when the model encounters divergent reasoning paths. For instance, in the depicted age calculation problem, the temporal preposition "in" ("Sarah will be 20 years old in 4 years") functions as a quintessential HD token. This HD token dictates the core logical branch for deriving the current age, such that correctly capturing it steers the model toward the correct path (identifying the current age as 16 to yield the final answer of 22) , whereas neglecting it risks misguiding the reasoning into an erroneous addition (e.g., 20+4=24) . To quantitatively validate the positive impact of HD tokens, we designed comparative experiments as depicted in Figure 1 (b). Specifically, we evaluated the performance disparity between a decoding strategy prioritizing HD tokens (Mask HD) and an alter prioritizing LD tokens (Mask LD). Here, the prioritization of tokens is achieved via Classifier-Free Guidance ( 19 ) , which amplifies the model’s sensitivity by contrasting conditional and unconditional denoising trajectories. The results indicate that the Mask HD consistently outperformed Mask LD in terms of accuracy (Acc@1) across all mathematical sub-domains. Notably, this strategy achieved a maximum accuracy increase of +2.07% (from 0.579 to 0.599) in the Prealgebra task and a +1.48% improvement in Counting & Probability, while maintaining stable positive gains even in the more challenging Algebra (+0.76%) and Precalculus (+0.55%) tasks. In essence, it is HD tokens, rather than LD counterparts, that constitute the informational backbone governing complex reasoning trajectories, underscoring that selectively prioritizing high-density information is effective for optimizing generation quality. Figure 1 : (a) A qualitative example illustrating that focusing on HD tokens effectively guides correct reasoning generation. (b) Quantitative results on Minerva_Math show that conditioning on HD tokens ( Mask HD ) consistently outperforms LD tokens ( Mask LD ). Motivated by the aforementioned validations, we propose FoCore ( Fo cus on the Core ), a training-free decoding strategy that leverages HD tokens in a self-contrast manner. Specifically, FoCore dynamically identifies and temporarily remasks converged HD tokens throughout the denoising trajectory to construct contrastive negative samples, and steers the generation by contrasting the output logits between the original and masked conditions, thereby compelling the model to strictly anchor its attention on core HD tokens during subsequent denoising steps. We further dive deep into the decoding dynamics of HD tokens throughout the denoising process, revealing an intriguing property: HD tokens exhibit an early-decoding tendency, converging to their final states earlier than surrounding tokens . Motivated by this finding, we introduce FoCore_A , an efficient variant that, upon detecting HD token convergence, performs parallel decoding over stable candidates within a local context window, substantially accelerating generation. Our contributions are threefold: • Empirical analysis on HD tokens: We systematically investigate the behavioral properties of HD tokens and derive three critical insights that collectively demonstrate their dual benefits in generation quality and decoding efficiency, serving as the motivation and supporting evidence for the design of FoCore and FoCore_A. • A decoding strategy leveraging HD tokens: We propose FoCore, a training-free decoding strategy that dynamically remasks converged HD tokens as negative samples to enhance generation quality, and FoCore_A, an efficient variant that exploits their early-decoding tendency for parallel decoding to further accelerate generation. • Comprehensive experimental evaluation: Extensive experiments on various benchmarks demonstrate that FoCore consistently enhances generation quality across both LLaDA and Dream backbones, while FoCore_A reduces per-sample latency from 20.76s to 8.64s ( - 58.4%) with a 2.07 \times decoding step reduction over standard Classifier-Free Guidance. 2 An Exploratory Analysis on DLMs 2.1 Preliminary on DLMs DLMs formulate text generation as an iterative denoising process, centered on two probabilistic processes applied to a clean sequence x_{0}\sim p_{\text{data}}(x_{0}) of length L : a forward noising process that progressively corrupts x_{0} into noise, and a reverse denoising process that learns to reconstruct x_{0} from the corrupted input. Due to space constraints, additional related work is provided in Appendix A . Forward Noising Process. In the discrete space, the forward process is a Markov chain progressively introducing an absorbing state, [MASK] (denoted as m ) ( 1 ) . By unrolling the step-wise transitions q(x_{t}|x_{t-1}) , the marginal probability at any step t\in[0,T] is analytically condensed into a categorical distribution governed by a monotonically decreasing schedule \alpha_{t} : q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1})\quad\Longrightarrow\quad q(x_{t}|x_{0})=\text{Cat}(x_{t};\alpha_{t}x_{0}+(1-\alpha_{t})m). (1) This indicates that at step t , each token retains its original identity x_{0} with probability \alpha_{t} , and is corrupted into m with probability 1-\alpha_{t} . As t\to