Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models
Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiaoge Zhang, Tianyi Li, Kaiyu Tang, Xiao Li, Jing Li · 2026
Read Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models by Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiaoge Zhang, Tianyi Li, Kaiyu Tang, Xiao Li, Jing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13413.txt.
Introduction
Fang and seven colleagues show that dataset-level metrics hide non-determinism in diffusion language models. Averaging sample quality across runs makes setups with similar total scores look alike, even when single inputs behave very differently. Their fine-grained evaluation tracks sample differences across guidance scale, diffusion steps, Monte Carlo sampling, batch size, and hardware. That spread proves widespread and structured, with code generation far more sensitive than question answering. Their Factor Variance Attribution metric splits observed spread across factor settings.
Catalog record
| CREATOR | Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiaoge Zhang, Tianyi Li, Kaiyu Tang, Xiao Li, Jing Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55445 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13413
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiaoge Zhang, Tianyi Li, Kaiyu Tang, Xiao Li, Jing Li Abstract Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remains poorly understood. The existing non-determinism evaluations for LLMs predominantly rely on dataset-level metrics under fixed inference configurations, providing limited insight into how model behavior varies across runs and evaluation conditions. In this work, we show that dataset-level metrics systematically attenuate non-determinism in diffusion language models by aggregating sample-level prediction quality across different runs. As a result, configurations with similar aggregate performance can exhibit substantially different behaviors on individual inputs, leaving fine-grained instability and distinct error patterns uncharacterized. To address this limitation, we conduct a fine-grained evaluation of non-determinism based on sample-level prediction differences across a range of model-related factors—including guidance scale, diffusion steps, and Monte Carlo sampling—as well as system-related factors such as batch size, hardware, and numerical precision. Our analysis reveals that non-determinism in DLMs is pervasive and structured, with code generation exhibiting markedly higher sensitivity to factor-level choices than question answering. To attribute sources of non-determinism evaluation, we introduce Factor Variance Attribution (FVA), a cross-factor analysis metric that decomposes observed non-determinism into variance attributable to different evaluation factor settings. Our findings highlight the need for fine-grained, factor-aware evaluation to enable reliable non-determinism assessment of diffusion language models. 1 Introduction Non-determinism has become an increasingly important concern in the evaluation of large language models ( 55 ; 20 ) : repeated evaluations under the same conditions can yield inconsistent predictions, raising questions about reliability, robustness, and reproducibility. While prior work has examined non-determinism primarily through variability in dataset-level performance metrics, such analyses offer only a coarse view of model non-determinism behavior and provide limited insight into non-determinism measurement. This challenge is particularly acute for diffusion language models (DLMs) ( 2 ; 36 ; 59 ; 50 ) . Unlike autoregressive decoding, diffusion-based generation is inherently stochastic and proceeds through multiple inference steps, introducing additional degrees of freedom at inference time, including sampling strategies, guidance scales, diffusion steps, and numerical execution details ( 26 ) . As a result, DLMs can exhibit pronounced non-determinism even when evaluated under seemingly minor configuration changes. Crucially, different inference configurations can achieve nearly identical dataset-level metrics—such as accuracy or pass@k ( 10 ; 28 ; 45 ) —while producing substantially different predictions for the same individual inputs. We argue that this phenomenon exposes a fundamental limitation of existing evaluation practices. Dataset-level metrics aggregate sample-level prediction quality across inputs in different runs, thereby attenuating non-determinism by reducing sensitivity to input-conditional variability. When correctness systematically flips for individual samples across configurations (e.g., one wrong-to right flip and one right-to-wrong), these changes are averaged out, obscuring which inputs are affected, how error modes differ, and whether observed variability arises from specific inference factors or from instability within fixed settings ( 41 ) . Consequently, evaluation based solely on aggregate quality metrics can give a misleading impression of stability and reproducibility, particularly for diffusion-based generation. In this work, we propose a fine-grained, cross-factor evaluation framework for characterizing non-determinism in diffusion language models beyond dataset-level metrics. First, we adopt a sample-level evaluation perspective that explicitly measures how prediction outcomes for the same input vary across inference-time configurations, enabling the identification of correctness flips that are invisible to dataset-level metrics. Second, we systematically study non-determinism induced by both model-related factors , which directly affect diffusion dynamics, and system-related factors , which arise from execution, hardware, and numerical choices, without any performance-driven tuning or hyperparameter optimization. Third, we introduce Factor Variance Attribution (FVA) , a variance-based decomposition that disentangles non-determinism arising from the choice of evaluation factors from sensitivity to specific settings within each factor, enabling a concrete and reproducible analysis of task-dependent evaluation non-determinism beyond dataset-level metrics. Our empirical analysis reveals that non-determinism in diffusion language models is pervasive and highly structured. In particular, code generation tasks exhibit substantially greater sensitivity to factor-level choices than question answering, and the dominant sources of variability differ across tasks and evaluation regimes. Together, these findings demonstrate that dataset-level metrics alone provide an incomplete picture of non-determinism and highlight the need for fine-grained, cross-factor evaluation to support reliable non-determinism assessment of diffusion language models. Table 1 : Examples of sample-level prediction disagreements induced by numerical precision (LLaDA). All configurations contribute identically at the dataset level, yet produce different answers for the same inputs, exposing distinct error modes that are obscured by aggregate metrics. Additional qualitative examples are provided in Appendix D . Question Answer Options INT8 BF16 FP16 Ground Truth In regions where prevailing winds blow from the oceans to the shore, which of these is most likely to occur? A: heavier rainfall B: dryer conditions C: cold temperatures D: frequent hurricanes D C C A Matt is a tall, eleven-year-old boy. He has a scar on his right cheek. He is intelligent and an excellent drummer. Which of his traits did he most likely inherit? A: his height B: his scar on his right cheek C: his intelligence D: his ability to play the drums B C C A Two bicyclists were riding their bikes at a rate of 20 kilometers per hour (km/h). During the next half hour, they observed that they had increased their speed to 26 km/h. What was their average acceleration? A: 0.2 km/h 2 B: 3.0 km/h 2 C: 6.0 km/h 2 D: 12.0 km/h 2 A C C D (a) Batch size. (b) CFG scale. (c) Monte Carlo samples. Figure 1 : Sample-level prediction flip rates across inference-time configurations. Each panel varies one inference factor while holding all other factors fixed: (a) batch size, (b) classifier-free guidance (CFG) scale, and (c) number of Monte Carlo (MC) samples. For a given sample s , the prediction flip rate is defined as 1-\max_{a}\lvert\{c\in\mathcal{C}:\hat{y}_{s,c}=a\}\rvert/\lvert\mathcal{C}\rvert , where \hat{y}_{s,c} denotes the predicted label for sample s under configuration c , a ranges over all possible predictions, and \mathcal{C} is the set of configurations obtained by varying the selected factor across its settings, with all other factors held fixed. Reported values are averages of the per-sample flip rates over the dataset, with error bars indicating the standard error across samples. Larger flip rates indicate stronger configuration-induced non-determinism at the sample level, even when dataset-level accuracy remains nearly unchanged. 2 Related Work 2.1 Diffusion Language Models and Evaluation Diffusion language models (DLMs) have recently emerged as an alternative to autoregressive generation, extending diffusion and denoising paradigms to language modeling and a broad range of generative tasks ( 26 ) . This line of work now spans continuous-space models ( 27 ; 17 ; 13 ; 54 ; 19 ; 33 ; 34 ; 44 ) , discrete masked diffusion models ( 2 ; 21 ; 39 ; 16 ; 50 ; 36 ; 31 ; 18 ; 24 ; 59 ; 11 ; 30 ; 32 ) , and multimodal diffusion language models ( 51 ; 49 ; 25 ; 53 ; 47 ) . Recent large-scale DLMs have demonstrated competitive performance on tasks such as question answering, reasoning, and code generation, while