Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

Zihao Wang, Enneng Yang, Lu Yin, Shiwei Liu, Li Shen · 2025

Read Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing by Zihao Wang, Enneng Yang, Lu Yin, Shiwei Liu, Li Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01548.txt.

Introduction

Model merging combines several fine-tuned expert models into one multi-task model at low cost, but public models can be merged without their makers' consent. Zihao Wang, with four co-authors, warns that unauthorized merging may infringe developers' rights and leak private information held in a model. Past defenses mostly detected merging after the fact instead of preventing it.

Their mechanism, MergeLock, actively disrupts a model's parameters so it can no longer be merged, while leaving its own output unchanged. The trick uses a symmetry inside transformer attention: pairs of invertible matrices are applied to two attention branches, pushing the model away from the shared parameter space of other fine-tuned models. Experiments across tasks support the protection. Wang and the team offer MergeLock for secure model sharing.

Catalog record

CREATORZihao Wang, Enneng Yang, Lu Yin, Shiwei Liu, Li Shen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01548

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing Zihao Wang, Enneng Yang, Lu Yin, Shiwei Liu, Li Shen Abstract Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetuned models become publicly available, concerns about the safety of model merging have emerged. Unauthorized merging may infringe on developers’ rights and risk leaking sensitive personal information. Most existing methods focus on detecting whether a merged model originates from a specific source model, but fail to effectively prevent illegal merging. In this paper, we propose MergeLock , an active protection mechanism that disrupts model parameters to render them unmergeable, thereby directly preventing unauthorized model merging. Specifically, leveraging the inherent symmetry of the attention mechanism in Transformer-based models, we randomly sample two pairs of invertible matrices and apply them to the Query-Key (QK) and Value-Output (VO) branches. This transformation keeps the model’s output unchanged while pushing it away from the shared parameter space of other finetuned models. Extensive experiments across both vision and language tasks demonstrate that MergeLock can degrade the performance of merged models by over 95% when a protected model is involved in most cases, demonstrating its effectiveness. Moreover, we further demonstrate that merged models protected by MergeLock cannot be effectively recovered using low-cost restoration methods, further enhancing robustness against unauthorized merging. The code is available at https://github.com/hetailang/Merge-Lock . 1 Introduction Multi-task learning (MTL) enables a single model to handle multiple tasks simultaneously, in contrast to training separate models for each task. This shared architecture significantly reduces storage requirements and improves inference efficiency ( Caruana 1997 ; Vandenhende et al. 2022 ; Zheng et al. 2023 ) . Owing to these advantages, MTL has gained popularity in various domains, including computer vision ( Chen et al. 2018 ; Yang et al. 2023 ; Liu et al. 2019 ) , natural language processing ( Collobert & Weston 2008 ; Dong et al. 2015 ) , recommendation systems ( Ma et al. 2018 ; Tang et al. 2020 ) , and speech recognition ( Ravanelli et al. 2020 ; Zhao et al. 2019 ) . However, traditional MTL methods require training on all relevant datasets, which complicates data collection and poses challenges for stable training, particularly when handling a large number of tasks simultaneously. Model merging ( Wortsman et al. 2022 ; Matena & Raffel 2022 ; Tang et al. 2024 ; Yang et al. 2024a ) has emerged as a promising alternative to traditional MTL methods, aiming to address their limitations. Instead of training a unified model from scratch on all tasks, model merging constructs a multi-task model by directly manipulating the parameters of independently trained, task-specific models. This approach eliminates the need for extensive data collection and retraining, thereby greatly reducing computational and storage overhead. The most straightforward merging method is weight averaging ( Utans 1996 ) , which simply averages the parameters of multiple models to obtain a merged model. Another representative technique is task arithmetic ( Ilharco et al. 2023 ) , which constructs a task vector by computing the difference between a fine-tuned model and its corresponding pretrained model, and then applies these vectors to create new models for downstream tasks. Recent state-of-the-art methods, such as Ties-Merging ( Yadav et al. 2023a ) , AdaMerging ( Yang et al. 2024c ) , Dare ( Yu et al. 2024 ) , WEMoE ( Shen et al. 2024 ) and TSV-M ( Gargiulo et al. 2025 ) , are all built upon the task arithmetic framework, offering improved compatibility and performance in complex multi-task settings. As more models become publicly available, model merging is expected to play an increasingly critical role across a wide range of domains and applications. Figure 1: (a) Left : To prevent model merging, transformation matrices A,A^{-1},B,B^{-1} are inserted into the query (Q), key (K), value (V) and output (O) branches of the self-attention layers. This transformation disrupts the parameter space to block effective merging, while preserving the original model’s output equivalence. (b) Right : Visualization of our unmergeable strategy in the loss landscape. Normally, two finetuned models (FT1, FT2) derived from the same pretrained model (P) lie within the same loss basin 1, and their merged model ( {\color[rgb]{0,1,0}\checkmark} ) achieves low loss. However, after applying our unmergeable transformation to FT1, the model (FT1 ′ ) is pushed into a different basin 2. As a result, merging FT1 ′ with FT2 produces a model ( {\color[rgb]{1,0,0}\times} ) that falls into a high-loss region. As model merging continues to gain traction, several critical challenges have emerged. A primary concern is that open-source models are becoming increasingly vulnerable: since model merging is both easy to perform and difficult to detect or trace, it poses a significant threat to the intellectual property (IP) of model developers ( Cong et al. 2024 ) . Moreover, model merging can compromise safety alignment; maliciously trained models may exploit the merging process to extract personally identifiable information (PII) or infer membership information (MI), thereby introducing serious privacy risks ( Guo et al. 2025 ) . Although existing techniques such as watermarking ( Li et al. 2023a ) and fingerprinting ( Cao et al. 2021 ) have been proposed, they primarily serve as detection mechanisms rather than preventive solutions ( Cong et al. 2024 ) , leaving unauthorized model merging largely unprevented. Therefore, there is an urgent need for preventive solutions against unauthorized model merging. In this paper, we propose MergeLock , a method designed to make models unmergeable, i.e., resistant to model merging, without compromising their original performance. Under this protection scheme, any unauthorized attempt to merge the protected model with others will result in a severely degraded model that performs poorly across all evaluation datasets. Specifically, inspired by the existence of multiple equivalent parameter spaces (basins) in deep neural networks ( Entezari et al. 2022 ; Ainsworth et al. 2023 ) , we apply an equivalent transformation to the parameters of the self-attention layers (see Fig. 1 (a)). These layers are a core component of Transformer-based architectures. This transformation moves the protected model away from the shared loss basin typically occupied by other fine-tuned models originating from the same pretrained model (see (FT1 ′ & FT2) vs. (FT1 & FT2) in Fig. 1 (b)). When an unmergeable model is merged with a regular model, the merged model exhibits a sharp increase in loss due to the significant discrepancy between their parameter spaces. Extensive experiments across both vision and language tasks demonstrate that MergeLock effectively degrades the performance of merged models. Even when parameter alignment techniques are applied to the protected model’s parameters, it remains difficult to recover acceptable performance. More specifically, in our experiments, the model protected by MergeLock degrades the overall performance of the merged model by approximately 95% , and the application of an alignment method recovers only about 5% of the original performance. One additional advantage of our method is that, developers retain full control over the distribution of protected models: by sharing a secret key (i.e., transformation matrices), authorized users can restore the original model. To summarize, the main contributions of this paper are in three aspects: •

Read the complete text