Training-Free Dynamic Upcycling of Expert Language Models
Eros Fanì, Oğuzhan Ersoy · 2026
Read Training-Free Dynamic Upcycling of Expert Language Models by Eros Fanì, Oğuzhan Ersoy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29765.txt.
Introduction
Fanì and Ersoy present DUME, a way to merge expert language models without new training. Expert tuning is costly and can overspecialize a model. Multi-domain training also suffers clashes and forgetting. DUME places dense experts inside a mixture-of-experts shell. A closed-form ridge step sets the mix, so no extra optimization runs. Experts can be added later on demand. Tests beat baselines in language modeling and reasoning. DUME keeps up to 97.6 percent of one expert's skill.
Catalog record
| CREATOR | Eros Fanì, Oğuzhan Ersoy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37378 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29765
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training-Free Dynamic Upcycling of Expert Language Models Eros Fanì, Oğuzhan Ersoy Abstract Large Language Models (LLMs) have achieved remarkable performance on a wide range of specialized tasks, exhibiting strong problem-solving capabilities. However, training these models is prohibitively expensive, and they often lack domain-specific expertise because they rely on general knowledge datasets. Expertise finetuning can address this issue; however, it often leads to overspecialization, and developing a single multi-domain expert remains difficult due to diverging objectives. Furthermore, multitask training is challenging due to interference and catastrophic forgetting. Existing work proposes combining the expertise of dense models within a Mixture of Experts (MoE) architecture, although this approach still requires multitask finetuning. To address these issues, we introduce Dynamic Upcycling MoE ( DUME ), a novel approach that reuses dense experts trained on different domains to construct a unified MoE model. Our method builds a single multitask model that preserves the capabilities of the original dense experts without requiring additional training. DUME is both cost-efficient and scalable: by leveraging the closed-form solution of ridge regression, it eliminates the need for further optimization and enables experts to be added dynamically while maintaining the model’s original performance. We demonstrate that DUME consistently outperforms baseline approaches in both causal language modeling and reasoning settings. Finally, we also show that the DUME model can be fine-tuned to further improve performance. We show that, in the causal language modeling setting, DUME can retain up to 97.6% of a dense expert model specialized in one particular domain, and that it can also surpass it in the reasoning setting, where it can achieve 102.1% of the dense expert performance. Our code is available at: github.com/gensyn-ai/dume . \reportnumber 1 Introduction Since the introduction of the transformer architecture ( 41 ) , Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks ( 30 ) , including specialized applications such as code generation ( 5 ) , mathematical problem-solving ( 7 ) , and instruction following ( 49 ) . Their adaptability and scalability have established them as powerful tools in both research and practical applications. In principle, training a single expert model that effectively generalizes across multiple domains would be desirable; however, in practice, this can be challenging due to optimization difficulties ( 46 ) , like negative transfer among heterogeneous tasks, catastrophic forgetting ( 28 ) , or because of the prohibitive computational cost of scaling such models ( 17 ) , and the communication costs or potential privacy challenges ( 47 ) associated with aggregating large, multi-domain datasets. Recent works on model merging ( 16 ) seek to construct unified models with multi-domain and multi-task capabilities by combining multiple specialized experts (that are trained independently in parallel). These experts share the same architecture but differ in their fine-tuned parameters, and are merged into a single model typically through linear operations in parameter space ( 19 ; 27 ; 42 ; 45 ) , such as simple weight averaging or task arithmetic. This approach is appealing because it avoids retraining from scratch, instead leveraging existing specialized models to produce a multitask expert. However, naïve aggregation can lead to destructive interference ( 45 ) , as experts trained on different domains may correspond to distinct loss landscapes; consequently, their parameters may be optimized for incompatible objectives, resulting in merged models that are suboptimal or detrimental across all tasks. To mitigate interference, another approach is to ensemble these domain experts rather than merge them into a single model; see Branch-Train-Merge (BTM) ( 26 ) . BTM ensembling demonstrates strong performance; however, the inference cost increases linearly with the number of experts relative to a single dense model. Branch-Train-Mix (BTX) ( 38 ) addresses this issue by merging the experts into a single Mixture of Experts (MoE) with sparse routing. MoE architectures ( 34 ; 20 ; 13 ; 8 ; 21 ; 4 ) allow sparse parameter activation, enabling conditional computation by activating only a subset of model parameters for each input, jointly reducing the effective computational cost during inference. In BTX, the expert models are merged as follows: the non-MLP parameters of the expert models are averaged, whereas the MLP parameters are used as separate MoE experts, which are sparsely routed by a gating network (or router). After constructing the combined MoE model, BTX requires finetuning it to calibrate the randomly initialized router. This could constitute a limitation in certain settings where such finetuning is unfeasible, for instance, because it requires to aggregate domain data into a single centralized node to train the final MoE model, which could raise concerns about privacy, or simply because of computational costs. To address issues about additional computational cost and domain data privacy, we propose Dynamic Upcycling MoE ( DUME ). Similar to BTX, DUME combines multiple dense experts into a single MoE architecture; however, it differs fundamentally in how the routing mechanism is initialized. Specifically, inspired by the applications of ridge regression in the horizontal federated learning ( 1 ; 12 ; 11 ; 10 ) and vertical federated learning ( 3 ; 18 ) literature, DUME initializes the gating networks of the MoE blocks using the closed-form solution of the ridge regression problem ( 37 ) . With this initialization, we demonstrate that no further training is necessary to approach, or even surpass, the performance of the best dense experts for each domain. Moreover, even if router fine-tuning is possible, our method remains superior because it provides a substantially better router initialization. Our main contributions can be summarized as follows: • We propose DUME , an embarrassingly parallel method that aggregates existing dense experts into a single, multi-domain MoE model, requiring no further training. Our method also allows the addition of more experts later in an incremental learning setting without performance degradation. • We demonstrate, with experiments in various domains and with 115M and 3B LLama models ( 40 ) , that DUME outperforms the baselines in both the causal language modeling and the reasoning scenarios. • We demonstrate that, when a unified multi-domain dataset is available, our model can be further trained to outperform the baselines by a larger margin. 2 Method In this section we present our method. First, in Section 2.1 , we briefly recall what ridge regression is and how can it be used for classification purposes, as our DUME relies on the ridge regression solution to correctly route tokens to the best domain experts. Then, in Section 2.2 , we explain the concept of experts MoErging , a procedure introduced previously and used in our method. Finally, Section 2.3 describes how DUME initializes the router parameters exploiting the closed-form solution of ridge regression. 2.1 Background on Ridge Regression Let f(x;W)=W^{\top}x be a linear predictor parameterized by W\in\mathbb{R}^{H\times C} , which maps inputs x\in\mathbb{R}^{H} to targets y\in\mathbb{R}^{C} . For a given dataset \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n} , the following ridge regression problem ( 37 ) : \argmin_{W\in\mathbb{R}^{H\times C}}\sum_{i=1}^{n}\left\lVert f(x_{i};W)-y_{i}\right\rVert^{2}+\lambda\left\lVert W\right\rVert^{2}, (1) controlled by a Tikhonov hyperparameter \lambda\in\mathbb{R}^{+} , admits the closed-form solution: W^{*}=(X^{\top}X+\lambda I_{H})^{-1}X^{\top}Y, (2) where X\in\mathbb{R}^{n\times H} and Y\in\mathbb{R}^{n\times C} are constructed by stacking the input and target vectors, respectively, and I_{H} is the identity matrix of size H . The term \lambda\left\lVert W\right\rVert^{2} makes the optimization