1bit -Merging: Dynamic Quantized Merging for Large Language Models

Shuqi Liu, Yuxuan Yao, Bowei He, Zehua Liu, Xiongwei Han, Mingxuan Yuan, Han Wu, Linqi Song · 2025

Read 1bit -Merging: Dynamic Quantized Merging for Large Language Models by Shuqi Liu, Yuxuan Yao, Bowei He, Zehua Liu, Xiongwei Han, Mingxuan Yuan, Han Wu, Linqi Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10743.txt.

Introduction

Shuqi Liu and colleagues present 1bit-Merging, a method to merge large language models with low storage cost. Their 2025 arXiv paper addresses a need created by specialist models. Fine-tuned models excel in narrow areas like chat, math, or code. Classic merging blends them into one static model. That blend often dulls the skill of each specialist.

Routing methods keep accuracy but store much more data. The new framework pairs routing with one-bit quantized task vectors. Different skills live in different layers of a model. Chat skill sits mainly in attention layers, while math and code sit in MLP layers. Targeted compression exploits that split. Tests on two model families matched or beat prior methods. Storage needs fell sharply at the same time.

Catalog record

CREATORShuqi Liu, Yuxuan Yao, Bowei He, Zehua Liu, Xiongwei Han, Mingxuan Yuan, Han Wu, Linqi Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10743

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

1bit -Merging: Dynamic Quantized Merging for Large Language Models Shuqi Liu, Yuxuan Yao, Bowei He, Zehua Liu, Xiongwei Han, Mingxuan Yuan, Han Wu, Linqi Song Abstract Recent advances in Large Language Models (LLMs) have led to specialized models excelling in specific domains, creating a need for efficient model merging techniques. While traditional merging approaches combine parameters into a single static model, they often compromise task-specific performance. However, task-specific routing methods maintain accuracy but introduce substantial storage overhead. We present 1bit -Merging, a novel framework that integrates task-specific routing with 1-bit quantized task vectors to balance performance and storage efficiency. Our approach further leverages the observation that different task-specific models store knowledge in distinct layers, such as MLP and attention layers, enabling targeted compression strategies. Through extensive experiments with LLaMA2 and Mistral model families across general knowledge, mathematical reasoning, and code generation tasks, we demonstrate that 1bit -Merging not only outperforms traditional model merging methods but also achieves better storage efficiency than task-specific routing approaches, maintaining 94.53% of their performance while reducing storage requirements to 55.02% on merging Chat, Math and Code fine-tuned models on Mistral 7B models. Figure 1: While individually fine-tuned models excel only in their specialized domains, our 1bit -Merging achieves superior performance across all domains. † † footnotetext: † Corresponding author. 1 Introduction Large language models (LLMs) have achieved remarkable progress, demonstrating favourable performance on a wide range of tasks Touvron et al. 2023 ; Zhao et al. 2023 . As a variety of fine-tuned (FT) models for specific domains have emerged, there is a growing need to combine their specialized capabilities into a single model ( Yang et al. 2024 ; Goddard et al. 2024 ) . While multi-task learning offers one solution Sanh et al. 2022 ; Fifty et al. 2021 , it requires extensive computational resources and simultaneous access to all task-specific datasets. Recent advances in parameter-space model merging Wortsman et al. 2022 ; Ilharco et al. 2023 ; Yadav et al. 2023 ; Yu et al. 2024 provide an efficient alternative, which combines pre-trained (PT) and FT models through parameter manipulation without additional training overhead, resulting in a unified model with comprehensive capabilities. Within the framework of model merging, task vectors Ilharco et al. 2023 ; Yadav et al. 2023 ; Yu et al. 2024 are pivotal for encoding task-specific knowledge. Defined as parametric deltas between PT and FT model weights, these vectors theoretically facilitate cross-model capability integration via linear arithmetic operations. Nonetheless, traditional merging often sacrifices task-specific performance, resulting in a noticeable gap compared to individual expert models. For instance, merging the math-specific , and code-specific models may result in a model that performs better in math than the code-specific model, but worse in code tasks. To alleviate such challenges, merging with task-specific routing ( Muqeeth et al. 2024 ; Lu et al. 2024 ) dynamically prioritizes relevant task vectors based on input data, effectively maintaining accuracy by isolating task-specific parameters. However, this routing-based merging strategy introduces substantial storage overhead, as it necessitates the preservation of all task vectors. SVD-based and quantization-based approaches are proposed to resolve the problem Wang et al. 2025 ; Liu et al. 2024 , but require complex high-dimensional computations or utilize the same quantization method for all components, contradicting previous observations Xu et al. 2025 ; Liu et al. 2025 that different layers possess varying importance. Building upon earlier studies that highlight the varying sensitivity across layers ( Liu et al. 2025 ) , we further investigate and identify differing levels of importance among functional modules, such as MLP and Attention, which can be leveraged for model merging. Notably, our empirical findings reveal that task-specific models encode knowledge in distinct modules. For example, chat-oriented models predominantly store knowledge within attention layers, enabling the compression of MLP layers. In contrast, math- and code-related models primarily encode knowledge in MLP layers, allowing for the compression of attention layers. In this work, we introduce 1bit -Merging, a novel dynamic merging framework that integrates task-specific routing with adaptive 1-bit quantized task vectors. Recognizing the substantial redundancy inherent within task vectors, we implement 1-bit compression, which significantly reduces storage requirements without notably compromising model capacities. Building upon the compressed task vectors, our method employs task-specific routing to establish a task-specific base model. This base model serves as the foundation for integrating the remaining compressed task vectors, ensuring that each task leverages the most relevant and efficient parameters. 1bit -Merging thus offers a balanced solution that maintains the performance advantages of task-specific routing while addressing the storage inefficiencies in existing methods. Empirically, we conducted extensive experiments by comparing 1bit -Merging with existing model merging approaches. We merged three widely used fine-tuned models focused on general knowledge (Chat), mathematical reasoning (Math), and code generation (Code) from the LLaMA2 and Mistral families. Our results show that 1bit -Merging outperforms traditional model merging methods while offering better storage efficiency than task-specific routing approaches. The task vector compression method preserves and often enhances the capabilities of FT models. By combining these compressed task vectors with our dynamic routing strategy, 1bit -Merging achieves superior performance. As shown in Figure 1 , it surpasses individually fine-tuned models across all domains. To sum up, our contributions include: (1) We discover that different task-specific models store knowledge in distinct layers, enabling targeted compression strategies based on specific compression position. (2) We propose a novel dynamic merging framework that integrates task-specific routing with 1-bit quantized task vectors. (3) Through comprehensive evaluations, we demonstrate that 1bit -Merging not only outperforms conventional merging methods but also achieves superior storage efficiency compared to task-specific routing. It retains 94.53% of the performance of full-precision models while reducing storage requirements to 55.02% , when merging on Mistral 7B models. 2 Related Work Model merging, a paradigm to enhance model generalization and robustness by synthesizing heterogeneous neural networks, is broadly categorized into static and dynamic approaches. Static Model Merging Static model merging primarily explores general strategies for combining models, such as Average Merging ( Wortsman et al. 2022 ) , Task Arithmetic ( Ilharco et al. 2023 ) , and TIES-Merging ( Yadav et al. 2023 ) . Notably, Wortsman et al. 2022 first demonstrated that even a straightforward weight averaging of base models can improve both the performance and robustness of downstream tasks. Building on this, Task Arithmetic ( Ilharco et al. 2023 ) refines the merging process by introducing task vectors, proposing that simple arithmetic operations on these vectors can effectively modify models and yield a better merged model. Expanding upon the concept of task vectors, methods like DARE ( Yu et al. 2024 ) and TIES-Merging ( Yadav et al. 2023 ) adopt pruning-then-scaling techniques to merge task vectors, based on the premise that not all parameters equally contribute to the final performance. However, static merging of models from different domains often sacrifices task-specific performance to strike a balance between generalization capacity and task-specific effectiveness. To this end, we explore the dynamic model merging in this work. Dynamic Model Merging Additionally, another line of studies focuses on routing-based model merging. For instance, SMEAR ( Muqeeth et al. 2024 ) propose a routing-based merging paradigm where parameter fusion is implemented through weighted averaging guided by router

Read the complete text