Surrogate Benchmarks for Model Merging Optimization

Rio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa · 2025

Read Surrogate Benchmarks for Model Merging Optimization by Rio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02555.txt.

Introduction

Rio Akizuki and six co-authors present a 2025 machine learning paper on cheaper ways to tune model merging, a technique that blends several trained models into one. Merging works well only with good hyperparameter settings, but testing settings on large models is expensive.

The authors build surrogate benchmarks: fast predictive models trained on collected samples that estimate a merged model's score from its settings alone. Two search spaces are defined for the settings. Tests show the surrogates predict merged performance well and reproduce how optimisation algorithms behave over time. Researchers can therefore develop and compare tuning algorithms at low cost before spending on full merges. The approach targets a practical bottleneck as merged language models grow larger.

Catalog record

CREATORRio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02555

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Surrogate Benchmarks for Model Merging Optimization Rio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa Abstract Model merging techniques aim to integrate the abilities of multiple models into a single model. Most model merging techniques have hyperparameters, and their setting affects the performance of the merged model. Because several existing works show that tuning hyperparameters in model merging can enhance the merging outcome, developing hyperparameter optimization algorithms for model merging is a promising direction. However, its optimization process is computationally expensive, particularly in merging LLMs. In this work, we develop surrogate benchmarks for optimization of the merging hyperparameters to realize algorithm development and performance comparison at low cost. We define two search spaces and collect data samples to construct surrogate models to predict the performance of a merged model from a hyperparameter. We demonstrate that our benchmarks can predict the performance of merged models well and simulate optimization algorithm behaviors. 1 Introduction Model merging ( Yang et al., 2024a ) is a promising approach to build a powerful single model from multiple separate models without accessing large datasets and requiring expensive computations. Model merging techniques have succeeded in enhancing the ability of large language models (LLMs) by merging multiple models fine-tuned by different downstream tasks. Most model merging techniques have hyperparameters to set before model merging. Because these hyperparameters affect the performance of merged models, tuning such hyperparameters can maximize the model merging capabilities. Akiba et al., 2025 proposed the evolutionary model merging that optimizes hyperparameters in model merging using an evolutionary algorithm. They used the covariance matrix adaptation evolution strategy (CMA-ES) ( Hansen, 2006 ; Hansen, 2023 ) and succeeded in finding the superior merging hyperparameters and building high-performance models. We call the approach for optimizing hyperparameters in model merging techniques model merging optimization . Model merging optimization is a kind of automated machine learning (AutoML) task. The literature on evolutionary model merging ( Akiba et al., 2025 ) shows the effectiveness and potential of the model merging optimization approach using the standard CMA-ES. Therefore, the development of sophisticated model merging optimization algorithms is a promising direction in the AutoML community. However, the computational cost of model merging optimization is relatively high, as with hyperparameter optimization and neural architecture search (NAS). 1 1 1 For example, the evaluation of a merging configuration for DARE-TIES ( Akiba et al., 2025 ) took about five minutes on an NVIDIA A100 (40GB) in our implementation. Although there is work ( Mencattini et al., 2025 ) to reduce the computational cost of evolutionary model merging by reducing the evaluation dataset and using a performance estimator, it still requires about 3.5 minutes on a single NVIDIA 4090 with 24GB of VRAM. Therefore, when yielding 1,000 merge evaluations, one run of the optimization algorithm requires more than 58 hours. This large computational load will burden the development of new model merging optimization algorithms and increase the cost of algorithm comparison. The benchmarks for hyperparameter optimization and NAS ( Eggensperger et al., 2015 ; Ying et al., 2019 ; Dong and Yang, 2020 ; Hirose et al., 2021 ; Zela et al., 2022 ) greatly contribute to algorithm development and evaluation. There are two types of benchmarks: tabular and surrogate benchmarks, where tabular benchmarks provide table lookup for hyperparameter settings and their evaluations through prior exhaustive search, and surrogate benchmarks construct a regression model that returns the performance values from a hyperparameter setting using a sampled actual evaluation dataset. While tabular benchmarks can provide exact hyperparameter evaluations, creating them in continuous or large search spaces is intractable. Surrogate benchmarks can be constructed even in a continuous and large search space, while the provided hyperparameter evaluation values are predicted by a certain model. Referring to the success of benchmarks for hyperparameter optimization and NAS, the benchmark for model merging optimization will also be essential for further algorithm development, which will enable us to compare model merging optimization methods at low cost and realize a fair and reproducible comparison. He et al., 2025 have proposed a benchmark suite for evaluating model merging techniques, which provides standardized fine-tuning models and evaluation protocols. However, evaluating the performance of merged models requires model merging computation and LLM inferences. Therefore, it cannot reduce the cost of model merging optimization. In this work, we construct a surrogate benchmark to significantly reduce the evaluation cost of model merging optimization. To our knowledge, this is the first surrogate benchmark for model merging optimization. We collect the paired data of the hyperparameter for model merging and its evaluation values, and construct the surrogate model that predicts the evaluation values from a given hyperparameter. We evaluate the proposed surrogate benchmark for model merging optimization, termed SMM-Bench , and demonstrate the use of our surrogate benchmark. The code of SMM-Bench will be made available at https://github.com/shiralab/SMM-Bench . 2 Surrogate Model Merging Benchmark (SMM-Bench) Akiba et al., 2025 optimized model merging configurations in two model merging settings: parameter space (PS) and data flow space (DFS) merging. For PS merging, where the parameters of the multiple source models are aggregated, the continuous hyperparameters in DARE-TIES ( Yadav et al., 2024 ; Yu et al., 2024 ) were optimized. For DFS merging, the merged model is constructed by stacking source models’ layers, and binary variables for the choice of layers and continuous variables for input scaling were optimized. Because the hyperparameters in model merging contain continuous variables and can be high-dimension, a surrogate benchmark is a reasonable choice. We introduce surrogate benchmarks for PS and DFS merging as SMM-Bench-PS and SMM-Bench-DFS . We use Japanese mathematics as a task to evaluate merged models as in Akiba et al., 2025 . The datasets, gsm8k-ja-test_250_1319 (denoting gsm8k-ja in short) ( Cobbe et al., 2021 ; Akiba et al., 2025 ) and the Japanese test set of MGSM ( Shi et al., 2023 ) , are used to calculate objective values in optimization and final test score, respectively. To evaluate the ability to solve mathematical tasks and provide answers in Japanese, we calculate accuracy, defined as the ratio of correct answers and reasoning texts in Japanese. This evaluation protocol is the same as in Akiba et al., 2025 . 2.1 SMM-Bench-PS Search Space Design Tuning the hyperparameters for each layer, called layer-wise merging, has the potential for performance improvement ( Yang et al., 2024b ) , while it increases the number of hyperparameters to be tuned. We focus on layer-wise merging using two source models and the simple merging method of task arithmetic ( Ilharco et al., 2023 ) . We use shisa-gamma-7b-v1 2 2 2 https://huggingface.co/augmxnt/shisa-gamma-7b-v1 and WizardMath-7B-V1.1 ( Luo et al., 2025 ) as source LLMs for PS merging. These LLMs are fine-tuned from Mistral-7B-v0.1 ( Jiang et al., 2023 ) and consist of 32 layers. Task arithmetic has a hyperparameter weight , the weighting factor of a task vector. Considering layer-wise merging, different hyperparameters can be specified for each source model’s layer, resulting in 64 design variables. We restrict the search space to [0,1]^{64} . Data Collection We created a dataset of merging hyperparameters and their evaluation values on gsm8k-ja and MGSM based on three strategies: random sampling, CMA-ES, and the tree-structured Parzen estimator (TPE) (

Read the complete text