BTS: Harmonizing Specialized Experts into a Generalist LLM

Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis · 2025

Read BTS: Harmonizing Specialized Experts into a Generalist LLM by Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00075.txt.

Introduction

Qizhen Zhang and a large team of co-authors described a way in 2025 to merge expert language models into one general model. Their method, called Branch-Train-Stitch, starts from one seed model. Copies of the seed are trained apart on fields such as coding or math, then joined back together.

Small stitch layers, trained on a mix of the expert fields, link the frozen experts to the seed model at run time. Because no expert is changed, experts can be added or removed with little new training. In the team's tests, the stitched model beat other merging methods on many tasks while keeping each expert's special skills. The design offers a modular path to broad models.

Catalog record

CREATORQizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BTS: Harmonizing Specialized Experts into a Generalist LLM Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis Abstract We present Branch-Train-Stitch (BTS), an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. Following Li et al. 2022 , we start with a single “seed” language model which is branched into domain-specific (e.g., coding or math) experts with continual pretraining. BTS combines experts into a generalist model using lightweight stitch layers, which are inserted between frozen experts and the seed LLM, and trained on a small datamix of the expert domains. Stitch layers enable the seed LLM to integrate representations from any number of experts during the forward pass, allowing it to generalize to new domains, despite remaining frozen. Because BTS does not alter the constituent LLMs, BTS provides a modular and flexible approach: experts can be easily removed and new experts can be added with only a small amount of training. Compared to alternative model merging approaches, BTS yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts. † † affiliation: GenAI at Meta † † affiliation: University of Oxford † † contribution: First author, work done at Meta † † contribution: Ordered alphabetically † † contribution: Joint last author † † date: August 9, 2026 † † correspondence: Qizhen Zhang at [email protected] 1 Introduction Figure 1 : Overview of the BTS algorithm. BTS operates in three phases. Different colors correspond to different expert domains. 1) Branch : Following Li et al. 2022 , we begin with a pretrained seed model and create N copies of it. 2) Train Experts : Each copy is independently pretrained on its respective data mixture, resulting in specialized expert models, as described in Li et al. 2022 . 2) Stitching : Stitch layers are inserted throughout the layers, alternating between the Experts-into-Hub stitch layer and the Hub-into-Experts stitch layer . Only the stitch layers are updated during this training phase. The BTS model always have a Experts-into-Hub stitch layer as the last layer, as the hub output is returned as the final BTS output. To achieve strong performance across diverse domains, large language models (LLMs) are often densely trained on trillions of tokens using thousands of GPUs ( Dubey et al. 2024 ) . Dense training requires substantial resources and significant infrastructure challenges, often requiring massive synchronization across distant compute clusters. Dense training also poses difficult datamix tradeoffs ( Xie et al. 2023 ; Ye et al. 2024 ) ; for example, it can be challenging to improve performance on a new domain without forgetting the original data ( McCloskey and Cohen 1989 ; Aghajanyan et al. 2021 ) or debug and correct unwanted behaviors without impacting others ( Tuan et al. 2024 ) . Expert merging techniques like Branch-Train-Merge ( Li et al. 2022 ; Gururangan et al. 2023 , BTM;) address these challenges by asynchronously training distinct expert models, specialized to different domains, and merging them back into a single generalist language model by ensembling them at inference time. Experts can be removed from the mix or added as needed. However, BTM is limited because there are no learned connections between expert layers; this restricts the model’s overall expressivity, especially in distant test domains. On the other hand, approaches like Branch-Train-MiX ( Sukhbaatar et al. 2024 , BTX;) , which upcycles experts into an Mixture-of-Experts (MoE) model ( Shazeer et al. 2017 ) , show strong downstream task performance, but lose the flexibility and interpretability inherent in a modular approach where experts remain distinct and intact. We present Branch-Train-Stitch (BTS), a new algorithm for building a generalist LLM from a collection of smaller expert models which achieves the best generalist model performance. Like other merging techniques ( Li et al. 2022 ; Gururangan et al. 2023 ) , BTS begins with a training phase in which experts are created via independent continued pretraining on domains of interest (starting from a shared “seed” checkpoint; Li et al. 2022 ). After expert training, the experts are adapted into a unified, generalist model by inserting and training stitch layers between models, while keeping the experts themselves frozen. This stitching architecture adds connections between experts via a gating mechanism on top of the language model layer outputs which determine how hidden states from one expert flow into another. One can imagine several ways to combine representations produced by different experts: all experts can directly connect to all other experts, only certain experts can connect to certain others, and everything in between. We opt for a hub-and-spoke model, in which a central “hub” model (the seed LLM) can update its own representations via the spokes (specialized experts), and vice versa, but the experts have no direct connection to each other. This design choice balances efficiency and performance. Since the seed model is trained on a variety of data, it is a natural choice for the hub, so all of our experiments adopt this set-up. For each layer in the forward pass, the stitching architecture alternates between hub-to-expert merging , where the hidden representations of the experts are updated with a projected hub LLM representation, and hub-to-expert merging , where the hub’s hidden representation is updated with a combined hidden representation of all experts. The final output provided by the merged LLM is the output of the seed model. These design choices are further motivated and validated empirically with ablations in section 4 . In experiments ( section 3 ), we find that BTS achieves the best generalist model performance compared to both expert merging and expert upcycling baselines and can even perform better than some individual experts on their target tasks. Notably, this is achieved with training only the small set of stitching parameters. The modular design of BTS, in which individual experts remain unchanged in the merging process, offers flexibility and interpretability. Targeted performance improvements for specific domains can be achieved completely asynchronously. Furthermore, downstream behaviors can be easily understood by analyzing which experts are ‘active’ at any given token, providing transparency into the model’s decision-making process. Our contributions are summarized as follows: • Branch-Train-Stitch , section 2 : We propose Branch-Train-Stitch , an efficient and flexible approach for stitching distinct expert models into a more powerful, generalist LLM. • Experiments, section 3 : We validate this approach through experiments on seed language models of 2.7B parameters. Our results demonstrate that BTS outperforms competitive baselines in downstream task performance, achieving the best average performance across benchmarks. • Analysis, section 4 : We motivate the BTS architectural choices with ablations and investigate the impact on “cross capability” tasks, i.e. tasks at the intersection of expert domains, and show that, in certain settings, BTS can achieve cross capability performance greater than any expert. Finally, we provide detailed analysis of the behavior of stitch layers at inference time, showing that BTS can dynamically adjust its expert utilization even within the same prompt. 2 Branch-Train-Stitch This section provides an overview of the BTS algorithm, beginning with a brief background on language model

Read the complete text