Mixing Times of Glauber Dynamics on Masked Language Models

Suvadip Sana, Sami Wolf, Neer Mehta, Alina Shah, Aitzaz Shaikh, Janna Goodman, Lionel Levine · 2026

Read Mixing Times of Glauber Dynamics on Masked Language Models by Suvadip Sana, Sami Wolf, Neer Mehta, Alina Shah, Aitzaz Shaikh, Janna Goodman, Lionel Levine (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16378.txt.

Introduction

Sana and six others model iterative masked-token redraws in masked language models as Glauber dynamics chains. Masked models define local conditional distributions but usually lack one shared joint pattern, raising the question of what global outcome looped redraws induce. A rectangle test proves that model conditionals are intrinsically incompatible, a pattern found across modern models. Under bounded cross-token influence, high-temperature contraction gives O(n log n) mixing time, while a uniform local margin rule yields metastability with slow escape from semantic basins at low temperatures.

Catalog record

CREATORSuvadip Sana, Sami Wolf, Neer Mehta, Alina Shah, Aitzaz Shaikh, Janna Goodman, Lionel Levine
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52772
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mixing Times of Glauber Dynamics on Masked Language Models Suvadip Sana, Sami Wolf, Neer Mehta, Alina Shah, Aitzaz Shaikh, Janna Goodman, Lionel Levine Abstract Masked language models (MLMs) define local conditional distributions over tokens but do not, in general, correspond to any consistent joint distribution over sequences. This raises a fundamental question: what global distributional behavior is induced when such conditionals are used iteratively for generation? We address this question by modeling iterative masked-token resampling as a Glauber dynamics Markov chain on the discrete space of token sequences. We first show that MLM conditionals are intrinsically incompatible: we introduce a rectangle test that certifies this incompatibility and empirically verify its prevalence across modern MLMs. We then provide a theoretical analysis of the induced Markov chain. Under bounded cross-token influence, we establish a high-temperature contraction result implying O ⁡ ( n ​ log ⁡ n ) O(n\log n) mixing time where n n is the sequence length. In contrast, we prove that under a uniform local margin condition, the chain exhibits metastability, with exponentially slow escape from semantic basins at low temperatures. Empirically, we demonstrate a phase transition in mixing behavior as a function of temperature and sequence length, consistent with the theoretical predictions. We further characterize the induced stationary behavior through semantic trajectories, identifying persistent structures such as long-lived traps and recurrent semantic basins, with political content serving as a measurable case study. 1 Introduction Masked language models (MLMs) such as BERT [ 4 ] are commonly used for natural language understanding (NLU) tasks such as generating semantically meaningful embeddings [ 4 ] , Machine translation [ 3 ] sentiment analysis [ 19 ] , and named entity recognition [ 4 ] . However, the global properties of MLMs, such as model biases and structural attractors, have not been completely characterized. Existing methods for interpreting MLMs fall short of capturing global structure. Single-token masked predictions ignore multi-token dependencies. Pseudo-Log-Likelihood scoring [ 17 ] is widely used as if conditionals define a joint distribution; we show this is unjustified. Linear probes on static embeddings [ 10 , 20 ] miss temporal phenomena entirely, such as the metastable semantic traps we document in this work. Figure 1: Glauber dynamics on BERT exhibits metastable semantic basins. PCA projection of sentence-embedding trajectories over 10,000 resampling steps, colored from warm (early) to cool (late). Tight clusters correspond to traps — configurations where the chain remains for hundreds to thousands of steps before escaping (§ B.6 ). Initial: “The overnight train rattled through the mountains as thunder echoed across the empty valley.” Final: “Greenville police say 2 sexual predators creeped on patients at Trinity Medical Center” Mapping this landscape is necessary for reasons beyond interpretability. While MLMs are not historically used for standalone text generation, [ 21 ] showed that Gibbs sampling from these models can produce fluent text, and many modern discrete diffusion models involve MLMs during denoising [ 11 ] . The energy landscape of the employed MLM is partially inherited by the generation process, raising the question of whether semantic biases in the MLM propagate to the outputs of these systems. As development of text-generating diffusion models continues to accelerate, understanding the hidden structure of these models will become increasingly important for fluent generation. Glauber dynamics, a single-site Markov Chain Monte-Carlo (MCMC) method introduced to study the Ising model [ 6 ] , provides the natural framework for this investigation: at each step, a random token position is masked and resampled from the MLM’s conditional distribution. Iterative resampling reveals two striking phenomena (Figs. 1 , 2 ): chains become trapped in metastable semantic basins for thousands of steps, and the time to forget the initial state undergoes a temperature-dependent transition. We make four contributions: 1. Incompatibility. We develop a “rectangle test” to measure the incompatibility of MLM conditionals by comparing the path-dependence of multi-token swaps. We demonstrate that incompatibility is an intrinsic structural artifact of the pseudo-likelihood training objective: it persists across model sizes and is amplified by inter-token semantic influence. 2. Mixing theory. We establish provable, sufficient conditions for both fast and slow mixing parameterized by token influence functions. Specifically, when cross-site influence is bounded at high temperatures, we prove an O(n\log n) upper bound on mixing time. Conversely, at low temperatures, we establish an exponentially slow lower bound in temperature, driven by uniform local score margins that trap the chain in metastable basins. We provide an empirically certified slow-mixing basin using a drift condition. 3. Phase transitions. We empirically characterize how the boundary between fast and slow mixing depends on temperature and sequence length, validating the theoretical predictions on BERT. 4. Semantic landscape. We probe the landscape induced by long-term dynamics to reveal semantic traps and recurrent basins. Using sentence embeddings, we track semantic trajectories to visualize metastable behavior. We analyze political content as a measurable case study of semantic recurrence across diverse chain initializations. We validate our approach by comparing the Glauber dynamics induced by several MLMs, including BERT [ 4 ] , RoBERTa [ 12 ] , and ModernBERT [ 22 ] . Figure 2: A temperature-length phase transition in mixing time. Two chains initialized from independent MS MARCO passages on RoBERTa-base are evolved under maximal coupling. Color: median steps to coupling within a 10^{4} -step budget. Black: no coupling within budget. The slow-to-fast boundary near \tau\approx 1.5-2 matches the regimes characterized in § 5 , § 6.1 . 2 Related Work Glauber dynamics and mixing in high-dimensional systems. Glauber dynamics was originally introduced in statistical physics as a stochastic single-site update process for sampling from Gibbs distributions, particularly in the Ising model [ 6 ] . At each step, one coordinate is selected and resampled conditioned on the remaining variables, producing a Markov chain whose stationary distribution corresponds to the target Gibbs measure. Theoretical analysis of Glauber dynamics and mixing times has played a central role in probability theory, statistical physics, and Markov Chain Monte Carlo methods [ 9 ] . Discrete diffusion and iterative denoising language models. Recent years have seen growing interest in diffusion and denoising approaches for discrete text generation. Structured discrete diffusion models [ 1 ] generalized denoising diffusion probabilistic models to discrete state spaces through categorical corruption processes. Diffusion-LM [ 11 ] demonstrated controllable text generation using diffusion dynamics in continuous latent spaces, while DiffuSeq [ 7 ] extended diffusion methods to sequence-to-sequence generation tasks. Several recent works established stronger connections between diffusion modeling and masked language modeling. DiffusionBERT [ 8 ] integrated diffusion-style denoising with MLM objectives, while SEDD [ 13 ] proposed ratio-based discrete diffusion modeling for text generation. Most closely related to our setting, masked diffusion language models [ 16 ] showed that iterative masked denoising objectives can serve as effective generative mechanisms for language modeling. Collectively, these works suggest that iterative local token refinement may serve as a viable alternative to autoregressive generation. Our work differs from prior diffusion-based language modeling research in that we focus not on generation quality or controllability, but on the global stochastic geometry induced by MLM conditionals. In particular, we study how repeated masked-token updates give rise to metastable semantic regions, topic persistence, and transition dynamics across sequence space. Biases, probing, and representation analysis

Read the complete text