Bayesian Model Merging

Kaiyang Li Shaobo Han Qing Su Shihao Ji · 2026

Read Bayesian Model Merging by Kaiyang Li Shaobo Han Qing Su Shihao Ji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12843.txt.

Introduction

Li and three colleagues propose ReForge, a method for merging several task-specific expert models into one without retraining them together. It refines the merged model module by module, using Bayesian regression anchored on the strong merged model, and tunes its settings by Bayesian optimisation. A data-free variant swaps in task-vector statistics instead of calibration examples. Across benchmarks, ReForge beats prior plug-and-play merging baselines.

Catalog record

CREATORKaiyang Li Shaobo Han Qing Su Shihao Ji
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74099
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bayesian Model Merging Kaiyang Li Shaobo Han Qing Su Shihao Ji Abstract Model merging aims to combine multiple task-specific expert models into a single model without joint retraining, offering a practical alternative to multi-task learning when data access or computational budget is limited. Existing methods, however, face two key limitations: (1) they overlook the valuable inductive bias of strong anchor models and estimate the merged weights from scratch, and (2) they rely on a shared hyperparameter setting across different modules of the network, lacking a global optimization strategy. This paper introduces Bayesian Model Merging (BMM) , a plug-and-play bi-level optimization framework, where the inner level formulates the model merging as an activation-based Bayesian regression under a strong prior induced by an anchor model, yielding an efficient closed-form solution; and the outer level leverages a Bayesian optimization procedure to search module-specific hyperparameters globally based on a small validation set. Furthermore, we reveal a key alignment between activation statistics and task vectors, enabling us to derive a data-free variant of BMM that estimates the Gram matrix for regression without any auxiliary data. Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, BMM consistently outperforms all plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). In particular, on the ViT-L/14 benchmark for 8-task merging, a single merged model reaches 95.1 , closely matching the average performance of eight task-specific experts ( 95.8 ). 1 Introduction Adapting foundation models to downstream tasks via fine-tuning has become standard practice, but maintaining a separate expert model for each task introduces substantial storage, deployment, and operational overhead. While multi-task learning offers a unified alternative [ 1 ] , it necessitates joint data access and incurs prohibitive computational costs, making it often infeasible under data silos, privacy constraints, or limited computing budgets. Model merging [ 2 , 3 , 4 ] offers a practical solution. By directly combining multiple expert models into a single architecture, it enables unified inference without revisiting the original training data or incurring the joint retraining costs. This paradigm has become increasingly relevant with the flourishing ecosystem of open-source models on platforms such as Hugging Face [ 5 ] , which offers an abundant supply of task-specific experts for integration. A central challenge in model merging is how to effectively combine expert models when auxiliary data is limited or unavailable. Existing methods fall into two broad regimes based on their use of auxiliary data. Data-assisted methods, such as Fisher Merging [ 2 ] and RegMean [ 6 ] , rely on a small calibration set to estimate empirical statistics for merging. Data-free methods, including Task Arithmetic (TA) [ 3 ] , TIES [ 4 ] , WUDI-Merging [ 7 ] , TSV [ 8 ] , and ISO-CTS [ 9 ] , avoid auxiliary data for estimating merged parameters, though some still use a held-out validation set for lightweight hyperparameter selection. Despite their differences, both regimes suffer from two important limitations. First, they typically estimate merged weights from scratch without exploiting prior knowledge from strong anchor models, leaving valuable inductive bias underused. Second, many advanced methods operate primarily through module-wise merging in isolation, and adopt a shared hyperparameter setting across different modules for tuning convenience, overlooking module heterogeneity and lacking a global optimization strategy for hyperparameter search. To address these limitations, we propose Bayesian Model Merging (BMM) , a plug-and-play bi-level optimization framework that is built on two key ideas. First, instead of estimating merged weights from scratch, BMM leverages prior knowledge from strong anchor models (e.g., TA [ 3 ] , RegMean [ 6 ] , or TSV [ 8 ] ), leading to an activation-based Bayesian regression with an efficient closed-form solution for module-wise merging. Second, rather than relying on a shared hyperparameter setting across all modules, BMM performs globally coordinated architecture-aware Bayesian optimization to adapt regularization strengths across heterogeneous modules of the network. Moreover, we theoretically and empirically reveal a key alignment between activation statistics and module-level task vectors, enabling us to derive a data-free variant of BMM while preserving the closed-form solution. Empirically, BMM is effective across both vision and language model merging. Across four backbones from two model families and seven plug-and-play anchor models, BMM consistently improves over every anchor in both data-assisted and data-free settings, with relative gains of up to 27% on weaker anchors while still improving the strongest ones. In particular, on the standard ViT-L/14 benchmark, a single merged model reaches 95.1% , closely matching the average performance of eight task-specific experts ( 95.8% ). Extensive ablation studies are conducted to further validate the main components and design choices of BMM. 2 Related Work Training-free model merging efficiently combines expert checkpoints without full data access or costly joint retraining. Existing methods generally fall into two broad regimes based on their reliance on auxiliary data: data-assisted and data-free . Data-assisted methods leverage a small calibration set or activation statistics to guide the merge. For example, Fisher Merging [ 2 ] performs Fisher-weighted average of task-specific models, while RegMean [ 6 ] casts linear-layer merging as regression with a closed-form solution. Data-free methods eliminate the need of auxiliary data and rely entirely on the weights of the expert models. In this regime, Task Arithmetic (TA) [ 3 ] serves as the seminal baseline. Subsequent works mainly mitigate interference among models through sparsification, including TIES [ 4 ] , DARE [ 10 ] , PCB-Merging [ 11 ] , and Localize-and-Stitch [ 12 ] . Another line of work (e.g., TSV [ 8 ] and ISO-CTS [ 9 ] ) leverages structured low-rank subspaces to isolate task-specific features or align task-relevant subspaces. More recently, methods such as WUDI-Merging [ 7 ] and DOGE [ 13 ] optimize explicit data-free merging objectives, but do not exploit prior knowledge from strong anchor models and learn a uniform hyperparameter setting for different modules of the network. Our BMM is complementary to these approaches: it leverages existing merged solutions as anchors, admits an efficient closed-form solution for module-level merging, and exploits bi-level optimization for hyperparameter search across different module groups. Representation geometry, neural collapse, and alignment after fine-tuning. A growing literature investigates the emergence of structured geometry in trained neural networks. Neural collapse [ 14 ] shows that during the terminal phase of training, the last-layer features collapse to their class means, and classifier weights align with the same simplex geometry. Neural Feature Ansatz [ 15 ] and Deep RFM [ 16 ] broaden this perspective to network-wise feature learning and connect the layer-wise weight structure to the average gradient outer products. A recent analysis by Liu et al. [ 17 ] further shows that latent representations, network weights, and gradients become mutually aligned across hidden layers. Several recent works [ 18 , 19 , 20 ] also connect neural collapse to transfer learning and fine-tuning, including collapse-inspired fine-tuning, transferability estimation, and analyses of downstream geometric complexity, but these studies mainly focus on last-layer collapse, transferability, or downstream fine-tuning behavior. In contrast, we study model merging from task-specific checkpoints, each of which is fine-tuned on a specific downstream task until convergence. We establish a module-wise alignment relation between the Gram matrix of the task vectors and the corresponding

Read the complete text