Navigating the Accuracy-Size Trade-Off with Flexible Model Merging

Akash Dhasade, Divyansh Jhunjhunwala, Milos Vujasinovic, Gauri Joshi, Anne-Marie Kermarrec · 2025

Read Navigating the Accuracy-Size Trade-Off with Flexible Model Merging by Akash Dhasade, Divyansh Jhunjhunwala, Milos Vujasinovic, Gauri Joshi, Anne-Marie Kermarrec (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23209.txt.

Introduction

Akash Dhasade and co-authors explore, in a 2025 paper, the trade between accuracy and size when merging AI models. Merging fine-tuned models into one is cheap and gives a jack-of-all-trades, but it usually trails the specialists. Keeping every specialist, though, costs storage. Their framework, FlexMerge, can emit merged models at any size in between, and it works with several merging recipes under one roof.

Charting the whole range brought surprises. Even a merged model twice the usual size gained accuracy steeply. And the ranking of recipes flipped as size grew: methods that lose at the single-model end can win further along. Tests on vision and language benchmarks, with up to thirty tasks, back the picture. Size, the authors say, is a dial merging research has ignored.

Catalog record

CREATORAkash Dhasade, Divyansh Jhunjhunwala, Milos Vujasinovic, Gauri Joshi, Anne-Marie Kermarrec
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Navigating the Accuracy-Size Trade-Off with Flexible Model Merging Akash Dhasade, Divyansh Jhunjhunwala, Milos Vujasinovic, Gauri Joshi, Anne-Marie Kermarrec Abstract Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into a single model often suffers from an accuracy gap with respect to individual fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high costs. We propose FlexMerge , a novel data-free model merging framework to flexibly generate merged models of varying sizes, spanning the spectrum from a single merged model to retaining all individual fine-tuned models. FlexMerge treats fine-tuned models as collections of sequential blocks and progressively merges them using any existing data-free merging method, halting at a desired size. We systematically explore the accuracy-size trade-off exhibited by different merging algorithms in combination with FlexMerge . Extensive experiments on vision and NLP benchmarks, with up to 30 30 tasks, reveal that even modestly larger merged models can provide substantial accuracy improvements over a single model. By offering fine-grained control over fused model size, FlexMerge provides a flexible, data-free, and high-performance solution for diverse deployment scenarios. 1 Introduction In recent years, the pre-training followed by fine-tuning paradigm has become the leading approach in both NLP ( NLP ) and computer vision, showcasing remarkable success on a wide range of tasks [ 16 , 17 , 18 , 5 ] . Pre-trained models (PTMs), which learn generalized features from large-scale datasets, serve as powerful starting points, enabling fine-tuning to achieve superior performance on downstream tasks with less labeled data. This has led to an exponential growth in the number of fine-tuned models driven further by the availability of open-source repositories [ 42 , 67 ] . However, deploying individual fine-tuned models for specific tasks incurs high storage and deployment costs. The alternative is MTL ( MTL ), which aims to jointly train a single model across multiple tasks [ 64 , 56 ] . But MTL comes with its own drawbacks, such as significant computational overhead and the need to simultaneously access the data from all tasks, which might be infeasible due to privacy constraints [ 31 ] . Figure 1 : (a) The fine-tuned model for each task is a sequence of blocks. (b) FlexMerge iteratively merges block pairs until reaching the desired size, e . g ., size 1.75\times as shown. (c) FlexMerge enables several data-free merging algorithms to achieve a “favorable” accuracy-size trade-off, with substantial accuracy gains early on and large size reductions without losing accuracy in the end (shaded areas). To mitigate these limitations, model merging has emerged as a promising solution, allowing the combination of multiple fine-tuned models into a single model without access to training data. To this end, several model merging methods have been proposed [ 26 , 19 , 66 , 72 , 27 ] . However, a single model is often unable to perfectly resolve parameter conflicts between tasks, leaving an accuracy gap with respect to the individual fine-tuned models [ 79 , 26 ] . This gap becomes more significant as a higher number of models are merged [ 72 , 27 ] . To mitigate this issue, some methods leverage additional data to facilitate merging [ 41 , 73 , 63 , 74 ] . Yet, the data-dependency might be difficult to meet in practice due to privacy constraints or proprietary restrictions, leading to a growing focus on data-free model merging techniques [ 26 , 19 , 76 , 72 , 27 ] . Nevertheless, in the absence of data, the accuracy gap remains significant, highlighting the need for novel solutions. We argue that an effective solution to this challenge is to go beyond the conventional one model approach, and merge into model(s) of bigger sizes. Merging multiple fine-tuned models naturally presents a trade-off between maintaining accuracy and achieving model compactness, dictated by the size of the merged model. This trade-off spans a spectrum: at one extreme, retaining all individual fine-tuned models for each task achieves maximal accuracy but at the cost of larger overall size; at the other, fully merging all tasks into a single model minimizes size but sacrifices accuracy. Despite this clear trade-off, a systematic investigation of the accuracy-size relationship in model merging has been lacking. In this light, we pose two key research questions: (RQ1) How can we derive merged models across the full range of model sizes in a data-free manner? and (RQ2) What is the nature of the accuracy-size trade-off exhibited by different data-free merging algorithms? Figure 2 : FlexMerge enables large accuracy gains when just doubling the deployed model size and attains full accuracy well before the maximum size. In response to (RQ1), we propose FlexMerge , a flexible framework that enables data-free fusion into model(s) of any desired size. At its core, FlexMerge treats each fine-tuned model as composed of sequential blocks, as illustrated in Figure 1 (a), whose granularity can be controlled ( e . g ., a transformer block, a few layers, or even a single layer). It then takes a bottom-up approach starting with all fine-tuned models with their respective blocks and greedily merging a pair of blocks with the highest cosine similarity in each merging iteration. This merging can leverage any existing data-free merging method such as Task Arithmetic (TA) [ 27 ] , TIES-Merging [ 72 ] , EMR-Merging [ 26 ] , Consensus [ 66 ] , etc ., applied at the block-level. With each merging iteration, the size of the deployed model is reduced, and the process can be halted once the desired size is met. For instance, in Figure 1 (b), the merging is halted when the merged model is 1.75\times the size of a single fine-tuned model. The entire merging process in FlexMerge needs no additional data or tuning, making FlexMerge fully data-free . In response to (RQ2), we demonstrate with FlexMerge that a range of data-free merging algorithms exhibit highly favorable accuracy-size trade-offs. Remarkably, the accuracy-size trade-off is characterized by steep gains in accuracy for even modestly bigger merged models beyond one model, followed by steady improvements, reaching near fine-tuning accuracy well before the maximum size. We term this trade-off a “favorable trade-off”, as opposed to a linear or an unfavorable trade-off indicated in Figure 1 (c). To illustrate this in practice, Figure 2 charts the merged model accuracy versus deployed size for 8 tasks (top) and 30 tasks (bottom) using the ViT-B/32 model, with TA [ 27 ] and Consensus [ 66 ] as the respective merging methods. \medcirc and \square annotate the accuracy at both ends of the spectrum i . e ., lowest fused size and retaining all fine-tuned models respectively. FlexMerge + TA gains 13.5\% in average accuracy when going from 1\times to 2\times while FlexMerge + Consensus gains 8.5\% when doubling the size from approximately 3\times to 6\times . We note that Consensus requires storing masks and the pre-trained parameters alongside the unified parameters [ 66 ] , resulting in the lowest possible size of \approx 3\times for 30 tasks. We observe that the steep rise is followed by relatively slower accuracy growth in the middle. Yet, a

Read the complete text