Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging
Shi Jie Yu, Sehyun Choi · 2025
Read Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging by Shi Jie Yu, Sehyun Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18580.txt.
Introduction
Shi Jie Yu and Sehyun Choi propose a smarter way to merge saved snapshots of a language model into one better model when only small adapter modules were trained. Merging checkpoints can save training time, but simply averaging them treats good and bad snapshots alike. Their method weights each snapshot by its performance, using training loss or step count, with one penalty knob to shape the weighting.
Across math reasoning, preference alignment, and instruction tuning tasks, the weighted merges beat plain averaging. Weighting by loss worked best, lifting accuracy by up to five percent over the uniform merge. It even beat the final checkpoint alone. The method adds almost no computing cost.
Catalog record
| CREATOR | Shi Jie Yu, Sehyun Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18580
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging Shi Jie Yu, Sehyun Choi Abstract Checkpoint merging is a technique for combining multiple model snapshots into a single superior model, potentially reducing training time for large language models. This paper explores checkpoint merging in the context of parameter-efficient fine-tuning (PEFT), where only small adapter modules (e.g. LoRA) are trained. We propose Metrics-Weighted Averaging (MWA), a simple yet effective method to merge model checkpoints by weighting their parameters according to performance metrics. In particular, we investigate weighting by training loss and by training steps, under the intuition that lower-loss or later-step checkpoints are more valuable. We introduce a formula with a penalty factor to adjust weight distribution, requiring only one hyperparameter regardless of the number of checkpoints. Experiments on three fine-tuning tasks (mathematical reasoning, preference alignment, and general instruction tuning) show that MWA consistently produces merged models that outperform the naive uniform average of checkpoints. Notably, loss-weighted merging often yields the best results, delivering up to 5% higher task accuracy than the baseline uniform merge and even surpassing the final individual checkpoint’s performance. These findings validate checkpoint merging for PEFT and demonstrate that a metric-driven weighting heuristic can efficiently boost model performance with minimal computational overhead. A Preprint 1 Introduction Model merging is a technique that merges the parameters of two or more deep learning models into a single model ( Yang et al., 2024 ) . This method has seen great success in research community recently ( Goddard et al., 2025 ) , by merging different models finetuned on multiple datasets and obtaining a single model with all of their capabilities ( Wortsman et al., 2022 ) . In this paper, we explore checkpoint merging, which is akin to model merging and merges the parameters of two or more checkpoints within the same training run into the single model checkpoint. The primary objectives of checkpoint merging is to create 1) a merged checkpoint post-training that is superior to all available checkpoints, or 2) a merged checkpoint superior to the model checkpoint at current time step during training . The latter objective, which may help to reduce training time, is especially beneficial for state-of-the-art large language models (LLMs) pre-training which requires substantial compute resources to train ( Hoffmann et al., 2022 ) . The problem with existing model merging methods is that they are either overly naive (e.g. Uniform Soup ( Wortsman et al., 2022 ) ) or overly complex (e.g. Fisher-Weighted Averaging ( Matena and Raffel, 2022 ) ). Simpler methods often underfits and struggle to derive an optimal model soup, while complex methods require high compute or implementation costs in exchange for relatively marginal gains in performance, which is also often not generalizable. Our proposed method, Metrics Weighted Averaging (MWA), strikes a balance by adopting a heuristical approach that is (1) simple enough to be efficiently implemented and, at the same time, (2) yield significant performance gains. Metrics-Weighted Averaging (MWA) ties the importance of a model to its performance in a given metric relative to other models in the model soup. To that end, we propose various formulas to heuristically calculate the weight of the checkpoints using different metrics. We then performed several experiments on well-studied finetuning datasets and benchmarks to empirically show that our MWA method can bring superior performance while introducing minimal overhead in weight computation. The experiment results demonstrate the ability of Metrics Weighted Averaging to identify an improved distribution of weights in a model soup within a short number of iterations. Recently, Parameter-Efficient Finetuning (PEFT) has become a popular alternative to full-finetuning LLMs due to their cost-effectiveness and powerful performance. While traditional model soups approach is applied on fully-tuned model checkpoints, we challenge this scheme and apply our MWA on the PEFT checkpoints. This could unlock even more cost-effective finetuning for higher performance gains on the target task. In this study, we specifically examine checkpoints consisting of low-rank adaptation (LoRA) modules. LoRA is a PEFT approach that has gained significant traction in recent years ( Hu et al., 2021 ) . We show notable performance gains on applying LoRA MWA on these checkpoints, especially for Math datasets: our best performing model merged from checkpoints via LoRA MWA outperformed (1) the last checkpoint in the model soup by 2.62%, (2) its Uniform Soup version by 2.84%, and (3) the last merging checkpoint final checkpoint in the model soup by 5.84% in the challenging GSM-Plus math benchmark. The contributions of this paper can be summarized as follows: 1. We validate the effectiveness of checkpoint merging specifically in the context of parameter-efficient fine-tuning. 2. We propose a metric-driven approach to find the optimal merging weights and its corresponding formula. 3. Lastly, we empirically demonstrate the effectiveness of our proposed method on various finetuning tasks. In particular, we discover that training loss serves as a highly reliable metric for metrics-weighted averaging. 2 Related Work Model Merging or model fusion, is a deep learning technique that combines the parameters or predictions of two or more pre-training models into a single model ( Wortsman et al., 2022 ) . It improves overall performance by ensembling the abilities of multiple models to compensate for the biases and errors of a single model. Model merging serves as an efficient technique to create multi-task models, or improve the performance on a single task, without the need for retraining ( Ilharco et al., 2023 ) . The most common type of model merging is parameter-wise merging, where corresponding parameters from multiple models are merged via a linear combination to obtain a new set of parameters. State-of-the-art model merging methods predominantly use parameter-wise merging, and so does our proposed metrics-weighted averaging method. Checkpoint Merging is a specific subset of model merging that merges multiple checkpoint models in a single training trajectory. Existing literature on checkpoint merging is severely limited. To the best of our knowledge, there is only one other paper that extensively studies checkpoint merging in the context of transformer-based language models ( Liu et al., 2024a ) . Liu et al. explores parameter-wise checkpoint merging via checkpoint merging, specifically in the full pre-training stage. This paper adds on to existing research in threefold. (1) It introduces a new technique for checkpoint merging. (2) It explores checkpoint merging on post-training stages, including supervised fine-tuning and human preference alignment. (3) It studies the effectiveness of checkpoint merging on parameter-efficient modules. Parameter-Efficient Fine-Tuning (PEFT) reduces the number of parameters that need to be fine-tuned by updating a small subset of the model’s parameters rather than the entire network ( Xu et al., 2023 ) . This approach significantly reduces computational cost and memory usage while maintaining a similar performance to full fine-tuning. It has seen increasing popularity with the advent of large language models, with LoRa ( Hu et al., 2021 ) emerging as the most prominent PEFT method. While many LoRa derivatives such as DoRa ( Liu et al., 2024b ) and QLoRa ( Dettmers et al., 2023 ) has emerged since LoRa’s inception, this paper mainly focuses on the vanilla LoRa implementation. 3 Method In this seection, we will describe our method, Metrics-Weighted Averaging (MWA) and its two sub-categories, Loss-Weighted Averaging and Steps-Weighted Averaging. 3.1 Metrics-Weighted Averaging Metrics-Weighted