Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
Injin Kong, Hyoungjoon Lee, Yohan Jo · 2026
Read Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models by Injin Kong, Hyoungjoon Lee, Yohan Jo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14758.txt.
Introduction
Injin Kong, Hyoungjoon Lee, and Yohan Jo look inside a model conversion in this 2026 paper. Post-training pretrained autoregressive models into masked diffusion models has emerged as a cost-effective strategy. It aims to overcome the limitations of sequential generation. Autoregressive models write one token at a time. Masked diffusion models can fill many blanks in parallel.
However, the internal algorithmic transformations induced by this paradigm shift remain unexplored. It is unclear whether post-trained masked diffusion models acquire genuine bidirectional reasoning capabilities. They might merely repackage autoregressive heuristics. The title calls the question a mechanism shift. The test is strict. It asks what the net does, not what score it gets, so a fake gain can be caught. Looks can fool. A smooth text may hide a plain trick, and this work digs for that gap. Readers get a probe of what truly changes inside a converted model.
Catalog record
| CREATOR | Injin Kong, Hyoungjoon Lee, Yohan Jo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models Injin Kong, Hyoungjoon Lee, Yohan Jo Abstract Post-training pretrained Autoregressive models (ARMs) into Masked Diffusion models (MDMs) has emerged as a cost-effective strategy to overcome the limitations of sequential generation. However, the internal algorithmic transformations induced by this paradigm shift remain unexplored, leaving it unclear whether post-trained MDMs acquire genuine bidirectional reasoning capabilities or merely repackage autoregressive heuristics. In this work, we address this question by conducting a comparative circuit analysis of ARMs and their MDM counterparts. Our analysis reveals a systematic “mechanism shift” dependent on the structural nature of the task . Structurally, we observe a distinct divergence: while MDMs largely retain autoregressive circuitry for tasks dominated by local causal dependencies, they abandon initialized pathways for global planning tasks, exhibiting distinct rewiring characterized by increased early-layer processing. Semantically, we identify a transition from sharp , localized specialization in ARMs to distributed integration in MDMs. Through these findings, we conclude that diffusion post-training does not merely adapt model parameters but fundamentally reorganizes internal computation to support non-sequential global planning . 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author. 1 Introduction Large language models have achieved near-human performance across diverse linguistic tasks 16 ; 19 ; 26 . Despite these advances, the prevalent autoregressive framework 27 ; 20 imposes structural limitations on generation 30 ; 2 . Specifically, sequential generation under causal masking prevents correcting past tokens 8 ; 30 ; 27 so that early errors propagate and amplify throughout the sequence, as the model cannot correct past inaccuracies 2 ; 21 . Moreover, many reasoning and planning tasks require global reasoning, where early decisions must account for constraints that apply to the entire sequence 8 ; 31 . Masked diffusion models (MDMs) have gained increasing interest as a non-autoregressive paradigm, with structural properties well-suited to overcoming these limitations 1 ; 22 . However, training diffusion-based language models from scratch remains computationally expensive due to slower convergence 7 . To mitigate this cost, recent work proposes post-training pretrained autoregressive models (ARMs) to the diffusion paradigm 7 ; 32 . Models such as Dream 32 demonstrate that this strategy can achieve strong performance while requiring only a fraction of the compute needed for training from scratch. Despite the empirical success of post-training ARMs with diffusion objectives, the specific algorithm changes induced by this post-training process are not yet understood 7 . It remains unclear whether post-trained MDMs genuinely learn new bidirectional reasoning mechanisms, as intended by the diffusion framework, or instead still rely heavily on autoregressive mechanisms at their core. Although mechanistic interpretability has been applied to understand diffusion models for image generation 23 ; 13 , this level of analysis has not yet been extended to text diffusion. Without such analysis, it is difficult to determine if post-trained MDMs genuinely perform global reasoning. If models continue to rely on local, left-to-right heuristics, the theoretical benefits of diffusion-based architectures fail to materialize. In this work, we address this question by analyzing language models from a circuit-level perspective 3 , which allows us to directly examine whether diffusion post-training induces new computational pathways or primarily reuses existing autoregressive ones. We investigate where algorithmic changes occur by comparing circuit structures between ARMs and MDMs post-trained from the same autoregressive backbones, and then examine how these changes are realized through detailed analysis using logit lens techniques and neuron-level visualizations. Through this analysis, we demonstrate that post-training using MDM objectives does not merely alter the training loss but instead induces a systematic reorganization of internal computation—shifting semantic roles across components and revealing a mechanism shift in how language models process and refine linguistic information. 2 Related Works 2.1 Masked Diffusion Models MDMs generate text by reversing a corruption process that stochastically replaces tokens with a [MASK] symbol 5 ; 1 . This approach allows for non-autoregressive generation using full bidirectional context. While training such models from scratch 12 is computationally expensive, recent work has shown that pretrained ARMs can be effectively post-trained into MDMs 7 . Rather than learning diffusion dynamics from scratch, these approaches initialize from a pretrained ARM. The model is then post-trained to iteratively denoise partially masked inputs, instead of predicting the next token autoregressively. From DiffuLLaMA 7 to Dream 32 , this line of work shows that post-training pretrained ARMs to MDM objectives can retain many practical advantages of diffusion—such as parallel decoding, iterative refinement, and bidirectional attention—while substantially reducing training cost. Post-trained MDMs also achieve strong performance on directionality-sensitive tasks. However, while these works establish the effectiveness of post-trained MDMs, they largely focus on performance and efficiency, leaving open the question of how diffusion objectives reshape the underlying computational mechanisms. 2.2 Mechanistic Interpretability and Circuits Mechanistic interpretability aims to identify the internal components and algorithms responsible for specific model behaviors 15 ; 6 . A central concept is the circuit , defined as a subgraph of the computational graph connecting inputs to the unembedding projection that is sufficient to produce a target behavior 15 ; 3 . Nodes correspond to components such as attention heads and MLPs, while directed edges represent causal dependencies between components, where the output of one node contributes to the input of another 17 ; 10 . Empirical studies show that many behaviors can be explained by sparse circuits involving only a small fraction of model connections 3 ; 28 . Methods such as Edge Attribution Patching identify these subgraphs via gradient-based attribution, enabling circuit discovery for tasks including indirect object identification and numerical comparison 9 ; 11 ; 3 . Across models and scales, similar circuits—such as induction heads—recur consistently, suggesting that they implement stable algorithmic functions rather than incidental patterns 18 ; 25 ; 17 ; 29 . Recent automated approaches, including ACDC and EAP, further enable scalable circuit discovery without manual inspection 24 ; 3 . While mechanistic analyses have begun to probe diffusion models in the vision domain 23 ; 13 , comparable studies for text diffusion models remain limited. As a result, it is unclear whether diffusion objectives induce distinct computational strategies in language models or primarily reorganize existing autoregressive circuitry. 3 Method Figure 1: Overview of the mechanism shift analysis pipeline. We extract task-specific circuits for both the Autoregressive Model (ARM) baseline and the post-trained Masked Diffusion Model (MDM) . We then identify the Top- K components that exhibit the highest topological divergence between the two architectures. Finally, we interpret the algorithmic nature of these shifts using Logit Lens and Neuron Explanation . To investigate the mechanistic shift from ARMs to MDMs, we adopt a comparative framework. Our primary objective is to distinguish the impact of shifting the learning objective from causal modeling to masked diffusion on circuit topology. We control for architectural confounding by analyzing ARMs alongside their directly post-trained MDM counterparts. 3.1 Models and Configuration We conduct experiments across two distinct model families to verify the generalizability of our findings. Specifically, we utilize the Qwen2.5-7B 19 and LLaMA-2-7B 26 architectures. To enable a direct comparison, we pair each standard autoregressive checkpoint with an MDM post-trained from that specific ARM base: • Qwen Series: We compare the autoregressive Qwen2.5-7B against Dream-Base-7B 32 , an MDM post-trained from