An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

Yuan Cao, Dezhi Ran ∗ †, Yuzhe Guo, Mengzhou Wu, Simin Chen, Linyi Li, Wei Yang, Tao Xie ∗ · 2026

Read An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse by Yuan Cao, Dezhi Ran ∗ †, Yuzhe Guo, Mengzhou Wu, Simin Chen, Linyi Li, Wei Yang, Tao Xie ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09463.txt.

Introduction

Yuan Cao and colleagues identify task-level merging collapse, where certain mixes of task-specialist models lose performance sharply after merging, under every merging method. Model merging promises reuse of fine-tuned models without retraining, yet some task pairs clash badly. Their experiments and statistics tie collapse to representational incompatibility between tasks, while parameter-space conflict measures barely correlate. That finding challenges the standard wisdom in the merging literature, and they supply a theoretical account of the phenomenon.

Catalog record

CREATORYuan Cao, Dezhi Ran ∗ †, Yuzhe Guo, Mengzhou Wu, Simin Chen, Linyi Li, Wei Yang, Tao Xie ∗
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40495
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse Yuan Cao, Dezhi Ran ∗ †, Yuzhe Guo, Mengzhou Wu, Simin Chen, Linyi Li, Wei Yang, Tao Xie ∗ Abstract Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology. † † affiliationnotice: * Co-corresponding authors. \dagger Project Leader. 1 Introduction Large language models (LLMs) ( 6 ; 5 ; 2 ; 15 ; 51 ; 16 ) have demonstrated remarkable success in numerous downstream tasks from natural language processing ( 45 ; 19 ) to code generation ( 33 ; 8 ; 26 ; 30 ) . While LLMs achieve strong general performance ( 2 ; 15 ; 51 ; 16 ) , adapting them to specific domains through fine-tuning remains computationally intensive ( 49 ) , particularly when multiple specialized variants are required for different tasks or applications ( 46 ; 49 ) . This challenge has spurred interests in parameter-efficient methods ( 23 ; 49 ) to combine and adapt pretrained models without exhaustive retraining. Among emerging solutions, model merging ( 58 ; 47 ; 25 ; 49 ; 55 ; 34 ) has shown particular promise as a computationally efficient approach to combine multiple fine-tuned LLMs derived from the same base model. By merging model parameters directly, model merging aims to consolidate specialized capabilities into a single unified model while avoiding the costs of additional training. By directly combining model parameters, merging leverages the widely observed linear mode connectivity ( 12 ; 60 ) between fine-tuned variants to create multi-capability models. Task arithmetic methods ( 58 ; 47 ; 25 ) and their recent extensions ( 49 ; 55 ; 34 ) have shown particular effectiveness for specific task combinations. However, a critical question remains underexplored: What fundamental limitations govern which tasks can be successfully merged without model-merging collapse (defined in Section 2 )? In this paper, we identify and characterize the phenomenon of task-level merging collapse. Through comprehensive empirical analysis spanning five state-of-the-art merging methods ( 58 ; 47 ; 25 ; 49 ; 55 ) , diverse LLM architectures ( 15 ; 51 ) , and fine-tuning approaches ( 22 ; 23 ) , we observe a consistent and alarming pattern: model merging can catastrophically fail when combining certain task combinations, despite each individual model performing well in isolation. Contrary to the prevailing focus on parameter conflicts ( 49 ) , our statistical analysis reveals that parameter-space metrics show minimal correlation. To explain this phenomenon and find better correlated metric, we introduce the first theoretical framework for analyzing model merging through the lens of rate-distortion theory ( 4 ) . Under the Locally Modified Components (LMC) assumption, we prove a fundamental bound relating merging distortion to the geometry of hidden representations. Our dimension-dependent theorem establishes that for representations in R^{d} , no convex merging method can achieve distortion below \Delta^{2}\cdot\frac{d}{2(d+1)} , where \Delta is the diameter of task-specific representation clusters. Our theoretical analysis aligns with the empirical findings, where hidden-state representational incompatibility at the task level is strongly correlated with merging collapse, providing a better understanding of what drives merging degradation and actionable insights for guiding task selection in improving model merging. In summary, our main findings and contributions are as follows: • Task-level representational incompatibility drives merging collapse. We demonstrate that merging failure is fundamentally determined by task compatibility rather than methodology, with certain task combinations consistently causing catastrophic performance degradation across all merging approaches. Through controlled experiments, we establish that task-level representational conflicts strongly predict merging failure, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. • Dimension-aware theoretical framework. We formalize task-level merging collapse using information theory, proving that for representations in \mathbb{R}^{d} , the minimum achievable hidden-state distortion is bounded by \Delta^{2}\cdot\frac{d}{2(d+1)} (Theorem 1 ). This dimension-dependent bound establishes precise quantitative limits on task mergeability, connecting empirical collapse to fundamental information-theoretic constraints. • Empirical validation of theoretical explanation. We validate our theoretical predictions through extensive experiments across diverse model architectures, tasks, and merging methods. Our results confirm that representation incompatibility statistically correlates with merging collapse, with observed merging collapse following the theoretical explanations. These findings provide a principled explanation for previously puzzling cases of merging collapse. 2 PRELIMINARY The development lifecycle of LLMs ( 5 ) typically follows a two-stage paradigm: pre-training and fine-tuning. While the pre-trained model develops a wide-ranging understanding, the representations learned during pre-training are not optimized for any specific application, necessitating further fine-tuning on task-specific data with targeted objectives. Let M_{\theta} denote a Foundation Model (FM) with trainable parameters \theta , which can represent either the entire model for full fine-tuning or a subset of parameters for parameter-efficient fine-tuning techniques ( 23 ; 32 ; 21 ) . In collaborative FM development environments, multiple teams often independently fine-tune the same base model for different capabilities or on different data distributions ( 43 ) , resulting in a collection of specialized models. Consider a set of tasks \mathcal{T}=\{t_{1},t_{2},...,t_{n}\} . The based FM M_{\theta_{0}} is fine-tuned on each task t_{i} , resulting in fine-tuned model M_{\theta_{i}} with updated parameters \theta_{i} . The distributed development creates the need for effective approaches to consolidate these independently evolved models. Model merging. Building upon the independent fine-tuning efforts described above, there arises a crucial need to integrate the strengths of multiple specialized models. Model merging ( 47 ; 58 ; 25 ; 10 ; 38 ; 49 ; 55 ; 52 ) refers to the process of combining multiple independently fine-tuned models with their parameter updates into a unified model that preserves the capabilities of its constituents. Given the updated parameters \{\theta_{1},...,\theta_{n}\} obtained from fine-tuning the common base model M_{\theta_{0}} , the goal of model merging is to produce a consolidated model M_{\theta_{merged}} that preserves the task-specific knowledge from each fine-tuned model. In order to achieve merging task-specific knowledge, many existing works ( 18 ; 20 ; 25 ; 10 ; 38 ) leverage Task vectors , which provides a concise representation of the model update resulting from fine-tuning on a specific task. Formally, let \theta_{0}\in\mathbb{R}^{d} denote the pre-trained model weights, and \theta_{t}\in\mathbb{R}^{d} the weights after fine-tuning on task t . The task vector is defined as \tau_{t}=\theta_{t}-\theta_{0} , capturing both the direction and magnitude of parameter changes induced by fine-tuning. In other words, \tau_{t} encodes the

Read the complete text