Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
Zeping Yu, Sophia Ananiadou · 2025
Read Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs by Zeping Yu, Sophia Ananiadou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16703.txt.
Introduction
Zeping Yu and Sophia Ananiadou tackle a known flaw of multimodal large language models in their 2025 arXiv paper. When such models are tuned on image and text tasks, they often lose much of their base skill with plain language, a loss the authors call catastrophic forgetting.
Their fix, Locate-then-Merge, needs no new training. It first finds the key settings in the model, then merges them with care. A neuron-level step keeps neurons that shifted a lot, since those likely hold new visual skill, and softens neurons that shifted less. Tests on thirteen benchmarks show the method beats older merging tools and cuts false claims about image content.
Catalog record
| CREATOR | Zeping Yu, Sophia Ananiadou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16703
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs Zeping Yu, Sophia Ananiadou Abstract Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM’s language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts—neurons likely responsible for newly acquired visual capabilities—while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation. 1 Introduction Multimodal large language models (MLLMs) ( Liu et al. 2023 ; Anil et al. 2023 ; Chen et al. 2024 ; Wu et al. 2024 ; Hurst et al. 2024 ; Bai et al. 2025 ) have advanced rapidly by adapting pretrained large language models (LLMs) Brown et al. 2020 ; Ouyang et al. 2022 ; Yang et al. 2024a ; Grattafiori et al. 2024 through multimodal instruction tuning. Among various modalities, vision has received the most attention and become the primary focus for enhancing multimodal LLMs Liang et al. 2024 ; Li et al. 2025 . By introducing vision-language connectors and training with image-text pairs, MLLMs have demonstrated impressive performance on vision-language tasks such as visual question answering ( Antol et al. 2015 ) and visual reasoning Hudson and Manning 2019 . However, recent studies ( Ratzlaff et al. 2024 ; Zhang et al. 2024 ) find that although visual instruction tuning can obtain visual capabilities, it often severely degrades the original general language abilities of the base LLMs, a phenomenon known as catastrophic forgetting ( Goodfellow et al. 2013 ; Kirkpatrick et al. 2017 ; Kemker et al. 2018 ) . Particularly on complex language understanding and reasoning benchmarks such as ARC-Challenge ( Clark et al. 2018 ) and GSM8K ( Cobbe et al. 2021 ) , finetuned MLLMs perform significantly worse than their original LLMs. Alarmingly, even strong open-source models like Llama3 suffer from this degradation, limiting the generalization and practical deployment of MLLMs across diverse and complicated language tasks. Figure 1: Neuron-Fusion in MLLMs. After visual tuning, some neurons exhibit larger changes than others. Neuron-Fusion selectively preserves neurons with significant parameter changes while suppressing those with smaller changes. This targeted fusion enables the model to retain newly acquired visual capabilities while minimally affecting its general language abilities. Although catastrophic forgetting in MLLMs poses a significant challenge, systematic studies addressing this issue remain limited. Recent years have witnessed the emergence of several novel model merging techniques Wortsman et al. 2022 ; Ilharco et al. 2022 ; Yadav et al. 2023 ; Yu et al. 2024 , demonstrating the potential of parameter fusion to alleviate catastrophic forgetting. However, as an emerging research area, model merging still lacks a systematic framework to guide the design and evaluation of effective methods. Moreover, existing methods were primarily developed for single-modal LLMs, and their application to MLLMs remains largely unexplored. It is unclear whether these methods can effectively mitigate catastrophic forgetting in MLLMs. This highlights the urgent need for targeted solutions that can recover language abilities while preserving visual adaptation. In this paper, we propose Locate-then-Merge, a general framework for training-free parameter fusion that decouples the process into two stages: locating important parameters and selectively merging them. Within this framework, we further develop Neuron-Fusion, a neuron-level fusion method designed to mitigate catastrophic forgetting, as illustrated in Figure 1. Our approach is motivated by an intuitive hypothesis: neurons with large changes during visual instruction tuning are likely to store newly acquired visual capabilities, whereas widespread but small changes across many neurons can cumulatively cause catastrophic forgetting in language ability. Therefore, Neuron-Fusion preserves the contributions of large-change neurons while suppressing the influence of small-change neurons. This design aims to retain the general language abilities of the base LLM while maintaining its acquired visual skills. Empirical results demonstrate that Neuron-Fusion surpasses five state-of-the-art model merging methods across 13 language and vision benchmarks, evaluated on two leading open-source MLLMs. Furthermore, generation analysis shows that Neuron-Fusion effectively reduces context hallucination, enhancing output quality and controllability. Our contributions are summarized as follows: a) We systematically validate that visual instruction tuning induces catastrophic forgetting even in powerful MLLMs, as evidenced by significant performance degradation on multiple language understanding and reasoning benchmarks, indicating that catastrophic forgetting remains a widespread and persistent challenge in MLLMs. b) We propose Locate-then-Merge, a general parameter fusion framework that unifies existing merging strategies: locating important parameters and merging them. Within this framework, we introduce Neuron-Fusion, a neuron-level selection and fusion method that identifies a small subset of neurons with significant parameter changes as key paths for visual adaptation, which is useful for mitigating catastrophic forgetting in MLLMs. c) We conduct extensive experiments across 13 benchmarks and two powerful open-source MLLMs to validate our approach. Our method is compared against several state-of-the-art model merging techniques, consistently demonstrating superior performance in mitigating catastrophic forgetting while preserving visual adaptation. 2 Related Work 2.1 Multimodal Large Language Models A typical MLLM consists of three components: a modality encoder, a modality connector, and a pre-trained LLM. In this work, we focus on the vision modality. First, a pre-trained vision encoder such as CLIP Radford et al. 2021 is employed to transform images into visual embeddings. Second, a vision-language connector, such as a lightweight two-layer MLP Liu et al. 2023 or cross-attention layers Alayrac et al. 2022 , maps these embeddings into the feature space of the LLM. Finally, the pre-trained LLM, such as Llama3 Grattafiori et al. 2024 , generates output tokens conditioned on both the textual and visual inputs. During the multimodal instruction tuning phase, the parameters of both the connector and the LLM are jointly finetuned to adapt to vision-language tasks. 2.2 Catastrophic Forgetting and Model Merging Catastrophic forgetting Goodfellow et al. 2013 refers to the phenomenon in which a machine learning model loses previously acquired knowledge when learning new capabilities. Luo et al. 2023 observe that catastrophic forgetting frequently occurs in LLMs during finetuning, and that larger models tend to forget even more information. Zhu et al. 2024 further investigate this phenomenon in the context of visual tasks. In addition, Zhang et al. 2024 and Ratzlaff et al. 2024 report that visual instruction tuning not only improves multimodal capabilities but also significantly impairs the general language abilities of the underlying LLMs. Ven et al. 2024 hypothesize that catastrophic forgetting arises because the parameters learned on new tasks deviate substantially from the optimal parameters for previous tasks. Model merging has emerged as an effective technique for combining the parameters of different models to construct a universal model, without requiring access to the training data Yang et al. 2024b . Alexandrov et al. 2024 demonstrate that model merging can be beneficial for mitigating catastrophic forgetting. Recent works Wortsman et al. 2022 ; Ilharco et al. 2022 ; Yadav