XPERT: Expert Knowledge Transfer for Effective Training of Language Models

Chang Liu, Boyu Shi, Xu Yang, Xin Geng · 2026

Read XPERT: Expert Knowledge Transfer for Effective Training of Language Models by Chang Liu, Boyu Shi, Xu Yang, Xin Geng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08842.txt.

Introduction

Chang Liu, Boyu Shi, Xu Yang, and Xin Geng recycle the knowledge stored in mixture of experts language models to train new models faster. In such models, routed expert modules make knowledge traceable, and analysis shows a subset of experts fires across nearly all domains, holding generalizable knowledge tied to broad competence. Their XPERT framework finds those cross domain experts using inference alone, refines their representations through tensor decomposition, and adapts the extracted knowledge for reuse in other models.

On language understanding and dialogue benchmarks, models seeded with reused expert knowledge perform stronger and converge faster than strong baselines. Trained expert models become structured knowledge mines, not just end products.

Catalog record

CREATORChang Liu, Boyu Shi, Xu Yang, Xin Geng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08842

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

XPERT: Expert Knowledge Transfer for Effective Training of Language Models Chang Liu, Boyu Shi, Xu Yang, Xin Geng Abstract Mixture-of-Experts (MoE) language models organize knowledge into explicitly routed expert modules, making expert-level representations traceable and analyzable. By analyzing expert activation patterns in MoE large language models (LLMs), we find that a subset of experts is consistently activated across diverse knowledge domains. These common experts encode cross-domain, generalizable knowledge that is closely related to model generalization, naturally raising the question of how such identifiable expert knowledge can be practically reused. Motivated by this observation, we propose XPERT , a framework that extracts, consolidates, and reuses expert knowledge from pre-trained MoE LLMs to support more effective training of language models across different model scales. XPERT identifies cross-domain experts via inference-only analysis, refines their representations through tensor decomposition, and adapts the extracted knowledge to reuse in downstream models. Experiments on language understanding and dialogue generation benchmarks show that models benefiting from reused expert knowledge achieve consistently stronger performance and faster convergence compared to strong baselines. These results highlight MoE LLMs as structured and reusable knowledge sources, and demonstrate the value of expert-level knowledge reuse for improving model training. 1 Introduction With the rapid advancement of large language models (LLMs) ( 42 ; 5 ; 1 ; 13 ; 48 ) , many models have demonstrated strong performance across a wide range of tasks and domains ( 46 ; 34 ; 44 ) . While early LLMs were predominantly based on dense architectures ( 36 ; 23 ; 2 ) , recent research has increasingly shifted towards sparse Mixture-of-Experts (MoE) models ( 31 ; 7 ; 27 ; 35 ; 22 ) , which scale model capacity by activating only a small subset of specialized experts per input. This conditional computation paradigm enables MoE models to achieve strong performance while reducing training and inference costs. Beyond computational efficiency, a key property of MoE architectures is that knowledge is explicitly modularized at the level of experts. Through routing mechanisms, each expert is selectively activated by specific inputs, making the relationship between expert representations and task characteristics both explicit and traceable. Motivated by this property, we analyze expert activation patterns in MoE LLMs using data from different knowledge domains. We observe that while many experts are domain-specific, a subset of experts remains consistently active across diverse domains. As shown in Figure 1 , certain experts (e.g., experts 8, 17, and 30 in OLMoE-7B) exhibit high activation across multiple domains, indicating that they encode shared cross-domain knowledge. We refer to such experts as common experts . The existence of common experts suggests a close connection between expert-level representations and model generalization. This intuition is further supported by recent MoE designs such as DeepSeekMoE ( 7 ) , which explicitly introduce shared experts activated across all inputs, highlighting the importance of general-purpose expert knowledge. These observations naturally raise a key question: what practical use can be made of expert knowledge that is both identifiable and closely tied to generalization? In the vision domain, prior work has shown that generalizable knowledge extracted from pre-trained convolutional neural networks or vision Transformers can be reused to improve training efficiency and stability across tasks and model configurations ( 39 ; 33 ) . Inspired by this line of work, we explore whether expert knowledge in MoE-based LLMs can be similarly extracted and transferred to improve the efficiency and effectiveness of model training. We propose XPERT , a framework that extracts cross-domain common expert knowledge from MoE LLMs and reuses it to support more effective training of models across different scales. Figure 1 : Experts activation frequencies of layer 15 in OLMoE-7B across different domains. Additional examples are provided in Appendix A . Figure 2 : The framework of XPERT. Z is the tensor formed by stacking the selected experts parameter matrices, and (\mathcal{G},U) represents the refined knowledge corresponding to a specific parameter matrix. After parameter-scale adaptation in Step 3, the extracted expert knowledge is used to initialize the FFN layers of language models with different scales. Under the XPERT framework, expert knowledge is identified through a purely inference-based analysis. As illustrated in Step 1 of Figure 2 , we perform forward passes on data drawn from multiple knowledge domains and record expert activation patterns. Experts that are consistently activated across domains are selected, as such activation patterns indicate cross-domain universal knowledge closely tied to model generalization. To transform the selected experts into a compact and reusable form, XPERT further introduces a knowledge consolidation step. As shown in Step 2 of Figure 2 , experts from each transformer block are aggregated into high-order tensors, upon which tensor decomposition is applied to extract shared components that capture domain-agnostic expert knowledge. A practical challenge arises when transferring the extracted knowledge to target models whose parameter dimensions differ from those of the source MoE LLMs. To address this mismatch, we propose a parameter-scale adaptation method that adjusts the extracted tensor representations to align with the dimensionality of the target model. As illustrated in Step 3 of Figure 2 , the reconstructed representations are adapted into Feed-Forward Network (FFN) parameter matrices that match the target model’s hidden and intermediate dimensions, and are used to initialize its FFN layers prior to training. All remaining parameters are initialized using standard random initialization. We evaluate XPERT using OLMoE-7B ( 27 ) and DeepSeekMoE-16B ( 7 ) as source MoE LLMs on supervised fine-tuning (SFT) benchmarks spanning language understanding and dialogue generation across multiple domains. Across all settings, XPERT extracts a compact subset of expert-derived representations (approximately 1.25% of the parameters in source LLMs). Notably, expert selection, knowledge consolidation, and parameter-scale adaptation are entirely training-free. When used as an initialization prior, XPERT-initialized models consistently outperform strong baselines, including training from scratch and knowledge distillation, while converging faster and requiring up to 5 \times less pre-training data. Overall, these results demonstrate that expert knowledge in MoE LLMs can be systematically reused as an effective and generalizable initialization prior. XPERT highlights the potential of MoE LLMs as structured knowledge sources beyond their original deployment. Our contributions can be summarized as follows: • We systematically study the reuse of expert knowledge in MoE LLMs, and show that a subset of experts consistently encodes cross-domain universal knowledge closely tied to model generalization. • We propose XPERT, a framework that extracts, consolidates, and adapts expert knowledge from pre-trained MoE LLMs to support more effective model training, enabling parameter-level reuse in models of diverse dimensions and sizes. • Through extensive experiments across multiple model scales and diverse downstream tasks, we demonstrate that XPERT-initialized models exhibit improved training dynamics and stronger downstream performance compared to other baselines. 2 Related Work Mixture-of-Experts Language Model MoE language models have been widely studied as an efficient architecture for scaling model capacity through conditional computation, where only a subset of experts is activated for each input ( 32 ) . Subsequent work has explored large-scale MoE pre-training and deployment, focusing on efficiency, scalability, and expert specialization, including Switch Transformer ( 9 ) , GShard ( 20 ) , and more recent MoE-based LLMs such as DeepSeekMoE ( 7 ) , OLMoE ( 27 ) , and Gemini ( 35 ) . Several

Read the complete text