Measuring Temporal Linguistic Emergence in Diffusion Language Models
Harry Lu · 2026
Read Measuring Temporal Linguistic Emergence in Diffusion Language Models by Harry Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23235.txt.
Introduction
Lu tracks when kinds of information show up during generation in diffusion language models, using three 32-step runs of LLaDA-8B-Base on WikiText-103. Four timed measures cover token commitment, linear readout of POS, semantic category, and identity, plus confidence shifts and remasking sensitivity. Across seeds the same order recurs: content groups settle before function groups, and coarse labels stay far more recoverable than word identity. Uncertainty runs higher for tokens that end up wrong, confidence grows less calibrated, and remasking sensitivity peaks mid-run, at the changed spots.
Catalog record
| CREATOR | Harry Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22414 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23235
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring Temporal Linguistic Emergence in Diffusion Language Models Harry Lu Abstract Diffusion language models expose an explicit denoising trajectory, making it possible to ask when different kinds of information become measurable during generation. We study three independent 32-step runs of GSAI-ML/LLaDA-8B-Base on masked WikiText-103 text, each with 1,000 probe-training sequences and 200 held-out evaluation sequences. From saved trajectories, we derive four temporal measurements: token commitment; linear recoverability of part-of-speech (POS), coarse semantic category, and token identity; confidence and entropy dynamics; and sensitivity under mid-trajectory re-masking. Across seeds, the same ordering recurs: content categories stabilize earlier than function-heavy categories, POS and coarse semantic labels remain substantially more linearly recoverable than exact lexical identity under our probe setup, uncertainty remains higher for tokens that ultimately resolve incorrectly even though late confidence becomes less calibrated, and perturbation sensitivity peaks in the middle of the trajectory. A direct/collateral decomposition shows that this peak is overwhelmingly local to the perturbed positions themselves. In this LLaDA+WikiText setting, denoising time is therefore a useful analysis axis: under our measurements, coarse labels are recovered earlier and more robustly than lexical identity, trajectory-level uncertainty tracks eventual correctness, and mid-trajectory states are the most intervention-sensitive. 1 Introduction Autoregressive language models reveal little about when a prediction becomes stable: generation advances left-to-right, and intermediate states are not naturally aligned to a fixed denoising clock. Diffusion language models, by contrast, iteratively refine a masked sequence and therefore expose a temporal axis that can be analyzed directly ( 13 ; 16 ) . That temporal structure is especially attractive for interpretability. If representations and predictions evolve over denoising steps, we can ask whether coarse linguistic labels, lexical identity, certainty, and intervention sensitivity follow the same schedule or distinct ones. We analyze three independent full 32-step runs of GSAI-ML/LLaDA-8B-Base on masked WikiText-103 text ( 15 ) . Our analysis is intentionally lightweight: it operates only on saved trajectories, uses simple linear probes in the spirit of prior probing work ( 11 ; 14 ; 17 ) , and pairs descriptive summaries with a single intervention-sensitivity curve. Even under these constraints, the temporal structure is far from uniform. We do not claim a universal law of diffusion-language generation from three runs. Instead, we ask whether four temporal measurements follow a consistent ordering across seeds: prediction stability, linearly recoverable linguistic information, self-reported certainty, and sensitivity to mid-trajectory re-masking. They do, but only in a measurement-aware sense. Under our probes, coarse labels are more recoverable than exact lexical identity throughout denoising, uncertainty remains informative about eventual correctness even late in the trajectory, and a non-initial re-masking-sensitive window becomes disproportionately important for the final outcome. Because the lexical probe uses a compact, partially unseen label space, we treat these comparisons ordinally rather than as normalized effect sizes. Our contributions are threefold: a lightweight trajectory-first framework for commitment, recoverability, certainty, and intervention sensitivity; empirical evidence, in one LLaDA+WikiText setting across three seeds, for a stable temporal ordering in which coarse labels are more recoverable than lexical identity under our probe setup and trajectory-level uncertainty tracks eventual correctness despite calibration drift; and a direct/collateral perturbation analysis showing that the strongest re-masking-sensitive window is non-initial and mostly local to the perturbed positions. 2 Related Work Diffusion language modeling has been driven mainly by controllable generation, masked-LM initialization, scaling, and noise-schedule design ( 13 ; 9 ; 23 ; 16 ; 21 ) . Recent work also pushes DLMs toward AR-scale adaptation and hybrid causal decoding, including AR-to-diffusion conversion in DiffuLLaMA, non-Markovian trajectory conditioning in CaDDi, and causal/masked hybrids such as ARMD and CARD ( 8 ; 24 ; 12 ; 18 ) . Trajectory-level work has also begun using denoising dynamics for downstream signals such as hallucination detection and generative stability ( 10 ; 7 ) , but these studies do not directly ask when linguistic structure becomes available during denoising. We focus on that temporal question. Our commitment analysis is also related to stabilization and early-exit work outside diffusion, where intermediate-layer confidence is used to decide when further computation is unnecessary ( 22 ; 6 ) . That literature treats stability mainly as an efficiency signal; we instead use a fixed denoising clock to compare when different kinds of linguistic information become linearly decodable, prediction-stable, and causally important. The analysis also connects to probing and representation analysis ( 1 ; 4 ; 2 ; 11 ; 14 ; 19 ; 5 ; 17 ) . Prior work uses simple probes to study what contextual encoders represent, often along network depth; we keep that probe family lightweight and move the analysis axis to denoising time. In this sense, our paper is closer in spirit to diffusion-trajectory interpretability in vision and planning-style diffusion, where temporal stages already serve as meaningful units of analysis ( 20 ; 3 ) . 3 Methods We treat each full denoising run as a token-by-step table defined over masked positions. For every masked token, the logger stores the predicted token, top-1 confidence, entropy, and final-layer hidden state at every denoising step. This representation lets us analyze the same trajectory from four complementary views: stabilization, recoverability, certainty, and intervention sensitivity. Measurements. Let M_{r} denote the masked positions in record r , let \hat{y}^{(t)}_{r,i} be the prediction at step t for masked position i\in M_{r} , let h^{(t)}_{r,i} be the corresponding hidden state, and let q^{(t)}_{r,i} and H^{(t)}_{r,i} be the stored top-1 confidence and entropy. We introduce the four measurements one by one. First, token commitment is the earliest step after which a prediction no longer changes: \displaystyle c_{r,i} \displaystyle=\min\{t:\hat{y}^{(t^{\prime})}_{r,i}=\hat{y}^{(t)}_{r,i},\forall t^{\prime}\geq t\}. (1) Second, for a probe target g\in\{\text{POS},\text{semantic},\text{token}\} , temporal recoverability is the average linear-probe accuracy over masked positions: \displaystyle A_{g}(t) \displaystyle=\frac{1}{N}\sum_{r,i}\mathbf{1}\!\left[\arg\max W_{g}h^{(t)}_{r,i}=\ell^{g}_{r,i}\right], (2) where N=\sum_{r}|M_{r}| . Third, we summarize certainty at step t by the mean top-1 confidence and entropy: \displaystyle\bar{q}(t) \displaystyle=\frac{1}{N}\sum_{r,i}q^{(t)}_{r,i},\qquad\bar{H}(t)=\frac{1}{N}\sum_{r,i}H^{(t)}_{r,i}. (3) Finally, intervention sensitivity is the drop in final accuracy after perturbing the trajectory at step t : \displaystyle\Delta(t) \displaystyle=\mathrm{Acc}_{\text{base}}-\mathrm{Acc}_{\text{perturb at }t}. (4) Throughout, we interpret this as sensitivity under a specific re-masking intervention rather than as a fully identified causal mechanism. For the direct/collateral follow-up, we additionally evaluate this drop separately on the re-masked positions themselves and on the untouched masked positions, yielding \Delta_{\text{direct}}(t) and \Delta_{\text{collateral}}(t) . For the probe plots, we reuse per-record outputs to estimate bootstrap uncertainty. For grouped commitment curves, let z_{r,i} be the coarse POS group of masked position (r,i) and let N_{k}=\sum_{r,i}\mathbf{1}[z_{r,i}=k] . We then report the empirical commitment CDF \displaystyle F_{k}(t)=\frac{1}{N_{k}}\sum_{r,i}\mathbf{1}[z_{r,i}=k]\mathbf{1}[c_{r,i}\leq t], (5) which makes Figure 1 (a) interpretable as a family of group-conditioned stabilization curves rather than only as mean commitment summaries. Experimental setup and aggregation. We evaluate GSAI-ML/LLaDA-8B-Base on masked WikiText-103 trajectories over three independent 32-step runs and report either cross-seed mean \pm standard deviation or a representative seed-42 trajectory when the qualitative pattern is visually stable. Full sampling, masking, and logging details are deferred to Appendix A. (a) Commitment accumulates earlier for numbers and content words than for function-heavy categories. (b) Across three seeds, coarse labels remain much more linearly recoverable than exact lexical identity. (c) Trajectory-level uncertainty separates eventually correct from eventually wrong tokens late in denoising. (d) Across three seeds, perturbation sensitivity peaks in the middle of the trajectory rather than at step