MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
Weisen Jiang, Shuhao Chen, Sinno Jialin Pan · 2026
Read MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification by Weisen Jiang, Shuhao Chen, Sinno Jialin Pan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14289.txt.
Introduction
Jiang, Chen and Pan propose MetaMoE, a privacy-preserving framework unifying independently trained, domain-specialized experts into one mixture-of-experts model using public proxy data. Client data stay spread out and unshareable under privacy rules, making unified training hard. Diversity-aware proxy selection picks relevant and diverse public samples to stand in for private data and guide router learning. Proxies also align expert training, improving coordination at unification, while a context-aware router improves expert choice across varied inputs.
Catalog record
| CREATOR | Weisen Jiang, Shuhao Chen, Sinno Jialin Pan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68693 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification Weisen Jiang, Shuhao Chen, Sinno Jialin Pan Abstract Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are distributed across clients and cannot be shared due to privacy constraints, making unified MoE training challenging. We propose MetaMoE , a privacy-preserving framework that unifies independently trained, domain-specialized experts into a single MoE using public proxy data as surrogates for inaccessible private data. Central to MetaMoE is diversity-aware proxy selection, which selects client-domain–relevant and diverse samples from public data to effectively approximate private data distributions and supervise router learning. These proxies are further used to align expert training, improving expert coordination at unification time, while a context-aware router enhances expert selection across heterogeneous inputs. Experiments on computer vision and natural language processing benchmarks demonstrate that MetaMoE consistently outperforms recent privacy-preserving MoE unification methods. Code is available at https://github.com/ws-jiang/MetaMoE . 1 Introduction Large foundation models ( 33 ; 34 ; 38 ) have become indispensable across domains such as computer vision (CV) ( 39 ; 53 ) and natural language processing (NLP) ( 19 ; 18 ; 1 ; 28 ; 17 ) . In practice, organizations and users often finetune a shared seed model on their own private data, resulting in a collection of specialized experts. While these experts could in principle be unified by aggregating private data to train a single Mixture-of-Experts (MoE) model ( 16 ; 7 ) , data sharing is often infeasible due to confidentiality, regulatory, and ethical constraints. This raises a fundamental question (called Privacy-Preserving Mixture-of-Experts Unification): How can we unify independently trained experts into one deployable MoE model while strictly preserving data privacy? Although federated learning ( 25 ; 20 ; 57 ) enables collaborative training without data sharing, it requires costly synchronized optimization across many clients and often suffers from performance degradation under heterogeneous client data distributions ( 52 ) . Several approaches have been proposed to unify independently finetuned experts. BTM ( 23 ) ensembles expert predictions, allowing embarrassingly parallel training but failing to produce a single deployable model for downstream fine-tuning or RLHF ( 36 ) . Model averaging methods such as Model Soup ( 54 ) merge parameters directly, which is computationally efficient but fragile when experts are diverse. BTX ( 47 ) extends this line by transplanting expert feed-forward network (FFN) sublayers into a shared MoE architecture with a learned router. However, its requirement for client-specific data to train this router limits its use in privacy-sensitive scenarios. When private data cannot be shared, public data can be used as proxies to approximate unavailable private distributions and provide supervision for router learning. Most recently, FlexOlmo ( 46 ) adopts this strategy by training a router on proxy data while anchoring experts to a public model. However, its reliance on similarity-based proxy selection often produces redundant and narrowly concentrated proxies, limiting coverage of domain-relevant modes and weakening router supervision. Moreover, because experts are trained exclusively on private data and never exposed to proxies, they remain domain-isolated, further exacerbating the mismatch between expert behavior and proxy-based routing. These limitations motivate the need for a more principled proxy selection and alignment strategy for expert unification under privacy constraints. We propose MetaMoE , a privacy-preserving framework for unifying independently trained experts into a single MoE model via diversity-aware proxy selection . Specifically, MetaMoE selects proxy samples using a relevance-weighted determinantal point process (DPP), an extension of standard DPPs ( 31 ; 21 ) that augments diversity with client-specific relevance scores, ensuring that proxy samples are both diverse and representative of each client domain. Each client then performs proxy-aligned expert training by finetuning FFN sublayers on private data alongside proxy samples. Because the same proxy data are later used for router training, this exposure aligns expert behavior with the supervision available at unification time, mitigating domain isolation and enabling effective coordination across experts. A context-aware router that incorporates both token-level and sequence-level context further improves expert assignment, after which experts’ FFN sublayers are merged into MoE layers and jointly finetuned on the union of proxy data. Experiments on both CV and NLP benchmarks demonstrate that MetaMoE consistently outperforms recent state-of-the-art baselines. Our contributions are three-fold: 1. We study privacy-preserving MoE unification and propose MetaMoE, a framework that unifies independently finetuned experts without sharing private data, and provide formal privacy guarantees. 2. We propose diversity-aware proxy selection via a relevance-weighted DPP, addressing the limitations of similarity-only proxy sampling for router learning. 3. We design a proxy-aligned expert training strategy and a context-aware router, and demonstrate through extensive CV and NLP experiments that MetaMoE achieves superior performance over recent methods. 2 Related Works 2.1 Federated Learning Federated Learning (FL) ( 25 ; 20 ; 57 ) enables collaborative training without centralizing raw data. Classical methods such as FedAvg ( 32 ) aggregate local updates, with extensions for parameter-efficient tuning ( 13 ) or differential privacy ( 5 ) . Yet scaling FL to large models ( 33 ; 34 ; 39 ) is difficult due to costly synchronization, degraded generalization ( 58 ) , and privacy leakage from gradients ( 51 ; 3 ) . In contrast, MetaMoE trains experts independently and asynchronously, then merges them via proxy-data-driven routing, avoiding FL’s communication bottlenecks while preserving privacy. 2.2 Model Merging and Mixture-of-Experts Model Merging ( 56 ; 54 ; 15 ; 55 ; 40 ; 24 ) explores how to combine multiple independently trained models into a single, stronger model without requiring costly joint training. Branch-Train-Merge (BTM) ( 23 ) trains domain experts independently and ensembles outputs at inference time, but does not yield a single unified model (hindering downstream SFT/RLHF ( 36 ) and incurring inference overhead). Model Soup ( 54 ) shows that averaging the weights of finetuned models often improves both accuracy and robustness with no additional inference cost, yet it is fragile when experts diverge in function space, leading to degraded performance in heterogeneous settings. Branch-Train-MiX (BTX) ( 47 ) inserts experts into MoE layers and learns a post-hoc router via additional fine-tuning on private data. More recently, FlexOlmo ( 46 ) advances this direction in federated settings by anchoring experts to a shared public model and aligning them with router embeddings, where proxy samples are selected for each client based solely on similarity. In contrast, our MetaMoE leverages a relevance-and-diversity criterion via a relevance-weighted DPP to select proxies, and employs a context-aware router, thereby combining heterogeneous experts into a unified MoE model under privacy constraints. The Mixture-of-Experts (MoE) ( 16 ) framework enhances model flexibility by combining specialized experts via a gating mechanism. Modern MoE architectures ( 41 ; 7 ) scale Transformers by activating only a sparse subset of experts per token, expanding capacity without proportional compute cost ( 45 ) . Key designs include the Switch Transformer ( 7 ) with top-1 routing and variants ( 45 ; 41 ; 42 ; 2 ) exploring top- k gating, stochastic routing, and random assignment for better accuracy, efficiency, and load balance. While MoE is effective for scaling, these methods rely on centralized access to all training data and synchronized training. To overcome these