Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k k -Parity

Jianhao Huang, Baharan Mirzasoleiman · 2026

Read Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k k -Parity by Jianhao Huang, Baharan Mirzasoleiman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22450.txt.

Introduction

Jianhao Huang and Baharan Mirzasoleiman tune a hidden brake inside masked diffusion language models. These models build text by filling masked slots, not word by word. Their 2026 preprint notes that their learning traits are little studied next to the standard left-to-right kind. The team study a toy task: the parity, or XOR sum, of a few key bits.

That task is famed for grokking, where a model stalls, then grasps all at once. Tuning the brake aims to make that grasp come sooner and hold. The work guides how this new model class should be trained.

Catalog record

CREATORJianhao Huang, Baharan Mirzasoleiman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22450

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k k -Parity Jianhao Huang, Baharan Mirzasoleiman Abstract Masked Diffusion Language Models have recently emerged as a powerful generative paradigm, yet their generalization properties remain understudied compared to their auto-regressive counterparts. In this work, we investigate these properties within the setting of the k k -parity problem (computing the XOR sum of k k relevant bits), where neural networks typically exhibit grokking—a prolonged plateau of chance-level performance followed by sudden generalization. We theoretically decompose the Masked Diffusion (MD) objective into a Signal regime which drives feature learning, and a Noise regime which serves as an implicit regularizer. By training nanoGPT using MD objective on the k k -parity problem, we demonstrate that MD objective fundamentally alters the learning landscape, enabling rapid and simultaneous generalization without experiencing grokking. Furthermore, we leverage our theoretical insights to optimize the distribution of the mask probability in the MD objective. Our method significantly improves perplexity for 50M-parameter models and achieves superior results across both pre-training from scratch and supervised fine-tuning. Specifically, we observe performance gains peaking at 8.8 % 8.8\% and 5.8 % 5.8\% , respectively, on 8B-parameter models, confirming the scalability and effectiveness of our framework in large-scale masked diffusion language model regimes. † † footnotetext: Preprint under review. Emails: {jhhuang2025, baharan}@cs.ucla.edu . † † footnotetext: Code will be released at https://github.com/JHHuang2021/MDLM . 1 Introduction Masked Diffusion Language Models (MDLMs) Lou et al. 2024 , Gong et al. 2025a , Nie et al. 2025a , Nie et al. 2025b , Ye et al. 2025 have recently emerged as a powerful alternative to the dominant autoregressive paradigm, achieving performance levels that rival standard Autoregressive Models (ARMs) OpenAI 2023 , Liu et al. 2024 , Comanici et al. 2025 , Yang et al. 2025 . Unlike ARMs, which are guided by next-token prediction, MDLMs utilize a training framework where tokens are independently masked at a ratio t\sim\mathcal{U}[0,1] , requiring the model to reconstruct the original data from its corrupted version. Interestingly, recent work demonstrates that MDLMs can outperform ARMs in low-data regimes Ni et al. 2025a and in the absence of weight decay Ni et al. 2025b , suggesting a natural superiority in generalization (see section 2 ). However, the role of the MDLM objective in steering optimization toward generalizable solutions has yet to be rigorously characterized. To answer this quesiton, we study a simplified setting—a Transformer trained with a masked diffusion loss on the k -parity problem—to uncover the fundamental mechanisms that drive MDLM learning. The k -parity problem typically presents a formidable challenge for standard training paradigms, which often fall into a “grokking” trap Power et al. 2022 : a prolonged plateau of chance-level performance followed by sudden, late-stage generalization. We demonstrate that the masked diffusion (MD) objective inherently circumvents this challenge. By analytically decomposing the objective, we reveal that the loss naturally partitions into a Signal Regime and a Noise Regime : \mathcal{L}_{\mathrm{eff}}(\theta)\approx P_{S}\underbrace{\mathbb{E}_{S}[\norm{f_\theta(\tilde{\mathbf{z}}) - f^*}^{2}]}_{\text{Task Signal}}+P_{N}\underbrace{\mathbb{E}_{N}[\norm{f_\theta(\tilde{\mathbf{z}})}^{2}]}_{\text{Implicit Regularization}}. Intuitively, the Noise Regime arises when the masking process is either too aggressive, obscuring the bits necessary to compute the parity, or when it only masks irrelevant positions, leaving the target bit untouched. In either case, the resulting sample becomes information-theoretically unidentifiable. Because these unidentifiable examples do not provide sufficient information to reconstruct the target, the objective forces the model to minimize its output norm on these inputs. This mechanism serves as an implicit regularizer that prevents the model from falling into pure memorization, thereby promoting robust algorithmic generalization. Building on this structural insight, we hypothesize that the standard practice of uniform mask sampling ( t\in[0,1] ) is suboptimal for natural language since the extreme ends of the process ( t\approx 0 and t\approx 1 ) may provide vanishingly small gradient signals or redundant regularization. We propose a Signal-Optimal Mask Sampling strategy that concentrates training on the most informative masking intervals. We validate our framework across multiple scales from toy level to pretraining and fine-tuning 8B-parameter models. We thus summarize our primary contributions as follows: • Theoretical Discovery of Implicit Regularization. Using the k -parity problem as an analytical framework, we provide a formal decomposition of the MD objective into a Signal Regime and a Noise Regime . We prove that the latter functions as a built-in implicit regularizer by penalizing model predictions on information-theoretically obscured inputs, a mechanism that eliminates the generalization delay typical of standard supervised objectives. • Empirical Verification on the Parity Case. We demonstrate that the MD objective fundamentally alters the learning dynamics of discrete algorithmic tasks, enabling near-instant generalization on k -parity benchmarks. Notably, MD escapes the “grokking” phenomenon entirely. To the best of our knowledge, we are the first to identify that masked diffusion can achieve such rapid generalization on parity. • Generalization of Theoretical Insights to Large-Scale Models. We demonstrate that the principles derived from the k -parity case translate effectively to real-world language modeling and reasoning. We validate this transfer across two distinct scales. (i) At 50M scale, we show that tuning the masking range to concentrate on high-signal regimes significantly improves perplexity on the WikiText dataset. (ii) We demonstrate that our signal-optimal distribution remains robust at the 8B-parameter scale. Our approach achieves up to 8.8\% improvement in pre-training. In supervised fine-tuning (SFT), optimizing noise intervals yields gains of up to 5.8\% on discriminative tasks and 3.4\% on complex generative reasoning. 2 Related Work Masked Diffusion Language Models. Masked Diffusion Language Models (MDLM) Lou et al. 2024 , Gong et al. 2025a , Nie et al. 2025a , Nie et al. 2025b , Ye et al. 2025 have recently emerged as a powerful alternative to the dominant autoregressive paradigm, achieving performance levels that rival standard Autoregressive Models (ARMs). Recent scaling-law analysis Ni et al. 2025a , Ni et al. 2025b demonstrate a unique compute-data tradeoff: while MDLMs are more data-intensive under fixed compute budgets, they outperform ARMs when training longer on small data. A particularly striking property of MDLM is its intrinsic robustness to overfitting, which manifests across two distinct settings. First, in low data-budget settings, MDLMs remain largely unaffected by data repetition, whereas ARMs exhibit severe degradation in validation loss and benchmark performance Ni et al. 2025a . Second, MDLMs maintain stable performance even without weight decay Ni et al. 2025b , while removing weight decay typically compromises the generalization of ARMs. Theoretically, Shi et al. 2024 simplified the MDLM objective into a weighted integral of cross-entropy losses. Sahoo et al. 2024 introduced a simplified, Rao-Blackwellized objective that takes the form of a weighted average of masked language modeling losses Devlin et al. 2019 . Ou et al. 2025 established its equivalence to the expected negative log-likelihood of any-order autoregressive models (AO-ARMs). While these studies clarify the structural form of the objective, they do not explain the superior generalization capabilities of the MDLM objective. We bridge this gap by analytically decomposing the MD objective, demonstrating that it contains an inherent implicit regularizer that penalizes the model’s output on unidentifiable inputs, thereby preventing memorization and facilitating rapid generalization. Noteably, our contribution is orthogonal to recent MDLM improvements on token-wise importance reweighting etc., as none of these works modify the fundamental distribution of the

Read the complete text