Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models

Jucheng Shen, Yeonju Ro · 2025

Read Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models by Jucheng Shen, Yeonju Ro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02077.txt.

Introduction

Jucheng Shen and Yeonju Ro work on speeding up a newer kind of language model, the masked diffusion model, which reveals text in blocks rather than word by word. Popular fast decoders unmask many tokens at once when confidence passes a fixed global threshold. The authors show confidence actually swings strongly from block to block and step to step, while different inputs in one dataset follow nearly identical confidence paths.

Their method, One-Shot Dynamic Thresholding, calibrates thresholds on a single sequence and reuses them for later inputs at almost no extra cost. On question answering, maths, and coding benchmarks it improved the balance between accuracy and speed, raising throughput by roughly a quarter to a half depending on the task.

Catalog record

CREATORJucheng Shen, Yeonju Ro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02077

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models Jucheng Shen, Yeonju Ro Abstract Masked Diffusion Language Models (MDLM) are becoming competitive with their autoregressive counterparts but commonly decode with fixed steps and sequential unmasking. To accelerate decoding, recent works like Fast-dLLM enables parallel decoding via a static global confidence threshold, yet we observe strong block/step-wise confidence fluctuations and, within a dataset, near-identical confidence trajectories across inputs indicated by cosine similarity. Inspired by these two observations, we introduce One-Shot Dynamic Thresholding (OSDT) , which calibrates thresholds on a single sequence and applies them to subsequent inputs with negligible overhead. On GPQA, GSM8K, and HumanEval, OSDT attains superior accuracy–throughput trade-offs ( +24% tokens/s on GSM8K at the best accuracy, +45% on GPQA with comparable accuracy, and +50% on HumanEval with a modest accuracy gap). Beyond these results, our findings suggest broader opportunities to leverage reusable task-level confidence signatures for more general-purpose algorithmic and systems innovations in diffusion decoding. Our code is available at https://github.com/jackshen-1215/osdt . \workshoptitle Efficient Reasoning 1 Introduction Masked diffusion language models (MDLM) have recently advanced rapidly, with pre-trained models such as LLaDA-8B ( 1 ) and Dream-7B ( 3 ) scaling discrete diffusion and demonstrating strong performance across language, math, and code. Inference in these models is semi-autoregressive: the sequence is partitioned into contiguous blocks B_{1},\dots,B_{T} that are generated left-to-right (autoregressive across blocks), while within each block decoding proceeds by diffusion-style denoising steps s=1,\dots,S that can unmask any subset of still-masked positions in a non-left-to-right order. At each step, a mask predictor proposes token distributions and a fixed per - step quota of token positions is filled with topk by confidence or randomly, while the remaining positions stay masked for later steps. To push throughput beyond fixed number of tokens per step, recent works like Fast-dLLM ( 2 ) enables parallel decoding by unmasking all tokens above a static global confidence threshold. However, our empirical observations show that confidence is highly non-uniform: it varies across blocks and denoising steps, yet confidence trajectories are remarkably stable across inputs within the same dataset (or, the same task). The first observation calls for adaptivity over the course of generation; the second indicates that a small amount of computation can capture a task-level “confidence signature.” Together, they reveal substantial space for more efficient parallel decoding in MDLM than static schedules can offer. We therefore study dynamic thresholding for parallel decoding in masked diffusion models . Formally, given a remasking decoder and per-step confidence scores, the goal is to design an inference-time policy \pi that selects thresholds \{\tau_{b}\} or \{\tau_{b,s}\} to decide which masked tokens to unmask, maximizing throughput while maintaining task accuracy (or, equivalently, improving the accuracy–throughput Pareto frontier). The policy must be task-aware, training-free, and add negligible overhead . Our contributions can be summarized as follows: • We empirically establish that within a dataset the block/step-wise confidence vectors are highly correlated across inputs (cosine similarity near 1), revealing a stable task-level signature. • We introduce One-Shot Dynamic Thresholding (OSDT) , a two-phase decoder that calibrates thresholds on a single sequence and reuses them for subsequent inputs at either block or step-block granularity. • We provide a comprehensive evaluation on GPQA, GSM8K, and HumanEval, showing that OSDT consistently improves tokens/s at comparable accuracy and often sets a better Pareto frontier than static thresholds as in Fast-dLLM. 2 Observation Prior work 2 assumes confidence follows a generalizable pattern and applies a static threshold across tasks. In practice, as shown in Figure 1 , benchmarks as different as GPQA (expert-level Q&A), GSM8K (grade-school math), and HumanEval (code generation) exhibit distinct signatures. This task-dependent variability undermines any “one-size-fits-all” thresholding strategy, highlighting the need for adaptation at inference time. Figure 1 : Step-block mean token confidence. Across GPQA, GSM8K, and HumanEval, confidence starts low, peaks mid-process, and drops near the final steps. These structured U-shaped dynamics highlight the limits of static thresholding. A natural question is whether these dynamics remain stable within a dataset. If confidence patterns are consistent across inputs, the profile of one sequence could proxy for the rest. To test this, we measured cosine similarity between average step-blockwise confidence vectors for all input pairs. Results in Figure 2 show striking consistency: cosine values are near 1.0 across GPQA, GSM8K, and HumanEval, producing nearly uniform bright heatmaps. This indicates that while confidence levels differ across tasks, the relative trajectories are highly stable within a dataset—a task-level rather than instance-level property. Figure 2 : Pairwise cosine similarity of step-block mean token confidence. Confidence trajectories are nearly identical across inputs of the same dataset, suggesting a single calibration run can generalize to the entire benchmark. This stability implies strong predictive power: the behavior on one sequence forecasts that of others in the same dataset. It provides the basis for adaptive thresholding methods that calibrate once and generalize broadly, enabling more efficient inference without sacrificing accuracy. 3 Dynamic Threshold These findings expose the limits of static cutoffs: they fail to adapt to structured, task-dependent confidence dynamics, yet within-dataset patterns are highly consistent. This motivates a method that is both dynamic and sample-efficient—able to adapt thresholds to a dataset’s confidence profile with minimal overhead. We propose One-Shot Dynamic Thresholding (OSDT) , a two-phase approach: • Phase 1 (Calibration): The first sequence is decoded with standard static-thresholding as in Fast-dLLM, and block- or step-block-wise confidence vectors are collected. • Phase 2 (Dynamic Inference): Subsequent sequences use thresholds derived from the calibration profile, with safeguards such as a cap \kappa and slack ratio \epsilon to balance efficiency and robustness. This mechanism dynamically aligns decoding decisions to the task’s confidence landscape. Unlike static thresholding, OSDT tailors thresholds to dataset-level dynamics while incurring negligible extra cost. Algorithm 1 (see Appendix) outlines the procedure. Its reliance on a single calibration run makes OSDT practical, efficient, and well-suited to real-world scenarios where both accuracy and latency matter. 4 Evaluation We evaluate One-Shot Dynamic Thresholding (OSDT) on three benchmarks: GPQA (expert-level Q&A), GSM8K (grade-school math), and HumanEval (code generation). OSDT calibrates a confidence profile on the first sequence of a dataset and applies dynamically adjusted thresholds to subsequent inputs. We study both its hyperparameter behavior and its performance against Fast-dLLM ( 2 ) . All experiments use batch size 1 on a single NVIDIA H100 GPU. 4.1 Hyperparameters OSDT exposes four hyperparameters that trade accuracy for throughput: • Dynamic Mode ( M ): Threshold per block ( block ) or per denoising step within each specific block ( step-block ). • Threshold Metric ( \mu ): Statistic over calibration confidences (mean, Q1, median, Q3, min-whisker). • Threshold Cap ( \kappa ): Upper bound to avoid overly strict thresholds. • Slack Ratio ( \epsilon ): Downscales thresholds to increase parallelism. We perform a grid search over \mu,\ \kappa\!\in\!\{0.75,0.8,0.85,0.9,0.95\},\ \epsilon\!\in\!\{0.01,0.05,0.1,0.15,0.2\} . Results (see Appendix for more details) show task-dependent optima: GPQA benefits from fine-grained step-block thresholds, whereas GSM8K and HumanEval prefer the simpler block mode. We use the following configurations in all following comparisons: • GPQA: step-block , q2 , \kappa=0.75 , \epsilon=0.20 . • GSM8K: block , q1 , \kappa=0.75 , \epsilon=0.20 . • HumanEval: block , q1 , \kappa=0.80 , \epsilon=0.10 . 4.2 Comparative Results Table 1 compares OSDT to Fast-dLLM’s fixed-threshold ( \tau=0.9

Read the complete text