Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models

Bo Wang, Junzhuo Li, Hong Chen, Yuanlin Chu, Yuxuan Fan, Xuming Hu · 2026

Read Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models by Bo Wang, Junzhuo Li, Hong Chen, Yuanlin Chu, Yuxuan Fan, Xuming Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08383.txt.

Introduction

Bo Wang and five co-authors wrote a 2026 arXiv paper on where a model gains its facts in pre-training. The paper sets dense designs, where all parts work on each token, next to Mixture-of-Experts designs, where a few parts work on each token. That split lets size grow past dense caps, yet how it shapes the gain of facts was not known. The team sets out Gated-LPI, a neuron-level score based on log-probability increase, which splits gains across neurons.

Catalog record

CREATORBo Wang, Junzhuo Li, Hong Chen, Yuanlin Chu, Yuxuan Fan, Xuming Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08383

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models Bo Wang, Junzhuo Li, Hong Chen, Yuanlin Chu, Yuxuan Fan, Xuming Hu Abstract Mixture-of-Experts (MoE) architectures decouple model capacity from per-token computation, enabling scaling beyond the computational limits imposed by dense scaling laws. Yet how MoE architectures shape knowledge acquisition during pre-training—and how this process differs from dense architectures—remains unknown. To address this issue, we introduce Gated-LPI (Log-Probability Increase), a neuron-level attribution metric that decomposes log-probability increase across neurons. We present a time-resolved comparison of knowledge acquisition dynamics in MoE and dense architectures, tracking checkpoints over 1.2M ( ≈ 5.0 ​ T \approx 5.0\text{T} tokens) and 600K ( ≈ 2.5 ​ T \approx 2.5\text{T} tokens) training steps, respectively. Our experiments uncover three patterns: (1) Low-entropy backbone. The top approximately 1% of MoE neurons capture over 45% of positive updates, forming a high-utility core, which is absent in the dense baseline. (2) Early consolidation. The MoE model locks into a stable importance profile within < < 100K steps, whereas the dense model remains volatile throughout training. (3) Functional robustness. Masking the ten most important MoE attention heads reduces relational HIT@10 by < < 10%, compared with > > 50% for the dense model, showing that sparsity fosters distributed—rather than brittle—knowledge storage. These patterns collectively demonstrate that sparsity fosters an intrinsically stable and distributed computational backbone from early in training, helping bridge the gap between sparse architectures and training-time interpretability. Introduction Large language models (LLMs) continue to scale in parameter count, but dense architectures make such scaling computationally inefficient, given the near-linear growth of per-token FLOPs with model size ( 21 ; 17 ) . To mitigate this, Mixture-of-Experts (MoE) architectures leverage conditional computation, expanding total capacity while activating only a sparse subset of experts per token, thus decoupling parameter growth from per-token compute ( 36 ; 22 ; 1 ) . Building on this idea, recent large-scale LLMs increasingly adopt MoE architectures ( 28 ; 9 ; 6 ; 30 ) , achieving strong downstream performance while scaling to trillion-parameter regimes under manageable per-token compute ( 42 ; 40 ; 41 ; 23 ) . Despite the impressive performance and efficiency of MoE models, our understanding of their internal working mechanisms remains limited. Recent interpretability research has developed a suite of neuron attribution techniques—such as Integrated Gradients (IG) ( 39 ; 12 ) , Layer-wise Relevance Propagation (LRP) ( 7 ; 3 ) , Log-Probability Increase (LPI) ( 43 ; 44 ) , and Neuron-Grad (NEG) ( 47 ) . However, these studies are constrained by two key limitations. First, they have focused on traditional dense Transformers. Second, existing analyses are almost post-hoc, dissecting models only after they have been fully trained. Recent studies have begun to trace representation during learning—for instance, linear probes that chart the rise of trustworthiness dimensions across 360 checkpoints, and circuit-tracing work that follows knowledge sub-graphs through finetuning ( 34 ; 33 ) . However, these efforts again focus on dense models, leaving the question: how architectural differences influence the dynamic knowledge acquisition process. In this paper, we present a neuron-level, time-resolved comparison of MoE and dense Transformers throughout pre-training to fill this gap. Building on LPI, we introduce a gated attribution measure that decomposes log-probability increase across experts (feed-forward network), and attention heads. We apply this method to OLMoE-1B-7B (MoE) and OLMo-7B (dense), two open-source Transformers, with 1.2M training steps ( \approx 5.0\text{T} tokens) and 600K steps ( \approx 2.5\text{T} tokens), respectively. By tracking pre-training checkpoints, we quantify and compare the evolution of importance for individual neurons in both the feed-forward network (FFN) and attention (ATTN) layers, and analyze how their importance evolves from three complementary perspectives: (i) neuron-level stability, (ii) layer-level consolidation dynamics, and (iii) functional robustness. We uncover three findings: • Low-entropy MoE backbone. Top approximately 1% of MoE neurons receive > 45% of positive updates, and this set is preserved—and reinforced—throughout training, contrasting with the churn observed in the dense model. • Early consolidation in MoE. Less than 100K steps suffice for MoE FFN and ATTN layers to lock into a stable importance profile, whereas the dense model exhibits volatility. • Functional robustness. Ablating the ten most important MoE attention heads reduces relational HIT@10 by <\!10\% , and the top 1\% of MoE FFN neurons by \approx 35\% , whereas the dense baseline suffers \approx 50\% and \approx 96\% drops, respectively—indicating that sparsity fosters distributed rather than brittle knowledge storage. By coupling scaling-friendly architectures with pre-training interpretability, our study bridges two previously disjoint threads and reveals that MoE models rapidly consolidate a stable and distributed backbone, from neurons to layers, that supports robust relational reasoning, in contrast to the dense model’s volatile and brittle organization. We hope this work will catalyze deeper dialogue between the MoE and interpretability communities and pave the way for training diagnostics that scale alongside model capacity. Method MoE Architecture Mixture-of-Experts (MoE) models extend the standard Transformer architecture by introducing sparsely activated experts within certain layers. Each MoE layer contains E independent feed-forward networks, known as experts, which share the same input dimension and structure but maintain separate parameters. During inference, a gating network selects the top- k experts for each input token based on their routing. The final output of an MoE layer is computed as a weighted sum of the selected expert outputs: \text{MoE}(x)=\sum_{\mathcal{E}\in\text{Top-}k}\text{gate}_{\mathcal{E}}(x)\cdot\text{FFN}_{\mathcal{E}}(x), (1) where x is the residual stream input, \text{gate}_{\mathcal{E}}(x) is the gating score for expert \mathcal{E} , and each expert FFN follows a two-layer structure with a nonlinearity: \text{FFN}_{\mathcal{E}}(x)=W^{(2)}_{\mathcal{E}}\cdot\sigma(W^{(1)}_{\mathcal{E}}x). (2) While dense Transformers process all tokens through the same parameters, MoE introduces conditional computation—only a small subset of experts is activated per token. This sparsity enables parameter-efficient scaling but also alters how information and knowledge are distributed across the network. Knowledge attribution seeks to identify which internal components of a language model contribute to specific predictions or knowledge representations. The sparsity of MoE architectures presents challenges for such analysis, as activations are no longer uniform across layers or inputs. In this work, we study how knowledge is stored and routed within MoE models, introducing a neuron-level attribution method to trace the predictive contributions of individual components over time. Neuron Attribution in MoE Models We adopt a neuron attribution method originally proposed for dense models ( 43 ) , and extend it to MoE models ( 27 ) . This method allows us to attribute the model’s prediction for a given token to individual neurons based on their effect on the output log-probability. Expert Neuron ( v^{\mathcal{E}} ) Definition. In each selected expert, we define a neuron as a column vector v in the output projection matrix W^{(2)}_{\mathcal{E}} (also referred to as a subvalue). Its activation is controlled by an input-dependent scalar coefficient m=\sigma(w^{(1)}\cdot x) , where w^{(1)} is a row in W^{(1)}_{\mathcal{E}} (the corresponding subkey) and x is the residual stream input. In MoE models with Top- k routing, which is the standard approach where each token activates multiple experts ( 19 ; 11 ; 35 ) , the final output added to the residual stream is the weighted sum over all selected experts: \text{output}_{v}=\sum_{\mathcal{E}\in\text{Top-}k}\text{gate}_{\mathcal{E}}(x)\cdot m_{\mathcal{E}}\cdot v_{\mathcal{E}}, (3) where each (m_{\mathcal{E}},v_{\mathcal{E}}) pair corresponds to a value neuron in expert e selected for

Read the complete text