Curvature-Guided Mixing for MLLM Adaptation
Jinglong Yang †, Jiaxuan He †, Wenjian Huang, Zhan Zhuang, Jianguo Zhang * · 2026
Read Curvature-Guided Mixing for MLLM Adaptation by Jinglong Yang †, Jiaxuan He †, Wenjian Huang, Zhan Zhuang, Jianguo Zhang * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24963.txt.
Introduction
Jinglong Yang and four colleagues tackle catastrophic forgetting when multimodal large language models are fine tuned on specialised tasks. Specialisation erodes the general abilities the base model had. Model merging can fight this, but existing methods are heuristic. Their framework, Curvature-Guided Mixing, merges pretrained and fine tuned models on firmer theory. A second order Hessian approximation of the loss landscapes yields an optimal closed form soft mixing ratio.
The ratio blends parameters according to their relative task specific curvatures, so sensitive weights move less. A robust hard mixing variant performs sparse parameter selection guided by a new curvature aware score. Experiments span LLaVA-1.5 and Qwen-2.5VL across several tasks. Both variants beat existing merging methods on the trade off between task gains and keeping general knowledge. Yang and colleagues release their code publicly. Curvature, not guesswork, decides how much of each model survives the merge.
Catalog record
| CREATOR | Jinglong Yang †, Jiaxuan He †, Wenjian Huang, Zhan Zhuang, Jianguo Zhang * |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24963
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Curvature-Guided Mixing for MLLM Adaptation Jinglong Yang †, Jiaxuan He †, Wenjian Huang, Zhan Zhuang, Jianguo Zhang * Abstract Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose Curvature-Guided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form “soft mixing” ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM † , a robust “hard mixing” variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen-2.5VL across multiple downstream tasks show that CGM and CGM † consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026 . † † footnotetext: These authors contributed equally. * * footnotetext: Corresponding author. † † footnotetext: Funding: This work is supported by National Natural Science Foundation of China (Grant No. 62276121) and Innovation Team and Talents Cultivation Program of National Administration of Traditional Chinese Medicine, No. ZYYCXTD-D-202403. 1 Introduction Multimodal Large Language Models (MLLMs) have emerged as powerful foundation models capable of joint understanding, generation, and planning across vision and language modalities [ 21 , 18 , 47 , 1 , 3 ] . Their impressive performance stems from pre-training on web-scale datasets, which endows them with a broad, general-purpose knowledge base. While this enables strong zero-shot performance on many tasks, adapting these models to specialized or novel domains requires targeted fine-tuning. This adaptation process, however, often leads to a critical issue: the model’s newly acquired skills come at the expense of losing its foundational, pre-trained abilities—a phenomenon known as catastrophic forgetting [ 44 , 25 , 15 ] . The central challenge, therefore, is to develop a methodology that can effectively instill new, task-specific knowledge into an MLLM while safeguarding the vast and robust general intelligence it already possesses. Figure 1 : Performance comparison of our methods (CGM and CGM † ) against baselines for LLaVA fine-tuned on OKVQA. We evaluate general knowledge retention (Pre-Avg: average performance of pre-training tasks), specialization on the new task (Target), and the harmonic mean of both (Hscore) to measure the overall balance. Several recent works have explored this challenge through model merging [ 13 , 46 ] and parameter-efficient fine-tuning [ 20 , 37 , 35 , 24 ] . However, these approaches remain limited in effectively balancing task adaptation and knowledge retention. For example, Spider [ 13 ] merges parameters using heuristic scoring functions without a solid theoretical foundation, while Model Tailor [ 46 ] employs second-order information [ 7 ] but optimizes only for downstream performance. As a result, these methods tend to overfit to the new task and neglect the preservation of pre-trained general knowledge, underscoring the need for a principled framework that jointly optimizes both objectives. To overcome these limitations, we introduce C urvature- G uided M ixing ( CGM ), a theoretically grounded framework for merging pre-trained and fine-tuned models. Our approach begins by formulating a clear objective: to find a new set of weights that simultaneously minimizes the loss with respect to both the fine-tuning and general pre-training tasks. We model the loss landscapes around the fine-tuned and pre-trained optima using a second-order Taylor approximation, which captures the local geometry, or “curvature”, of each loss surface via its respective Hessian matrix. To illustrate this, consider Figure 2 , which visualizes two hypothetical loss landscapes corresponding to the fine-tuning and pre-training tasks. A naive interpolation between their optima would likely fall into a region of high loss for both. Guided by local curvature, our CGM method navigates this anisotropic landscape by identifying that the first parameter direction, critical for the fine-tuning task (due to its high curvature), should be adopted from the fine-tuned model, while the second direction, critical for the pre-training task, should be retained from the pre-trained model. This enables CGM to effectively integrate the “skills” learned during fine-tuning with the foundational knowledge from pre-training, achieving low loss on both tasks and mitigating catastrophic forgetting. By mathematically formulating and solving this joint optimization problem (as detailed in Section 3 ), we derive a closed-form per-parameter mixing ratio. This “soft mixing” rule provides a non-heuristic and theoretically grounded solution. It elegantly demonstrates that the optimal mixing ratio for each parameter is directly determined by its relative curvature in the two loss landscapes. Figure 2 : A conceptual illustration of the motivation behind CGM. The fine-tuning loss (orange) and pre-training loss (blue) landscapes exhibit conflicting anisotropic curvatures: the fine-tuning loss is sharp along one axis, whereas the pre-training loss is sharp along the other. CGM leverages this second-order geometry to locate a balanced joint minimum, as detailed in Section 3 . While soft mixing provides a closed-form solution for the approximated loss, it is fundamentally a dense interpolation that updates every parameter. This approach carries a significant risk: a dense update can be disruptive, potentially destroying existing knowledge. We hypothesize that a sparse update strategy is more robust. The intuition is to preserve the vast majority of the model’s parameters and only modify the most critical subset. This approach, which is analogous to imposing an \mathbf{L_{0}} constraint on the update, ensures a minimal, targeted modification. To implement this, we propose a “hard mixing” strategy, termed CGM † . This method reframes the problem as a sparse selection task. Our method starts from the fine-tuned model—which has acquired the new skill but suffered from forgetting—and treats the pre-trained parameters as a “knowledge reservoir” to be sparsely re-integrated. For each parameter, a discrete decision is made on whether to revert to the pre-trained value. We derive a simple, curvature-aware score to rank parameters in the learnable layers. By selecting only the top- K\% of parameters (where K denotes the sparsity ratio) most critical for general knowledge and least critical for the new task, CGM † performs a sparse, targeted reversion that effectively preserves foundational knowledge while maintaining the newly acquired skill and avoiding the pitfalls of dense interpolation. Figure 3 : Radar plots illustrating the performance trade-off between downstream adaptation and general knowledge retention. The “Target Task” axis shows performance on the fine-tuned task, while all other axes measure general pre-trained capabilities. Our methods, CGM and CGM † demonstrate superior balance by achieving high target-task performance while simultaneously preserving pre-trained knowledge. In summary, our main contributions are as follows: • We propose Curvature-Guided Mixing (CGM) , a novel and theoretically grounded method that optimally merges fine-tuned and pre-trained models by minimizing a joint, curvature-aware loss objective. • We introduce CGM † , a variant that reformulates the merging task as a sparse parameter selection problem, providing an efficient and effective mechanism for re-injecting pre-trained knowledge to mitigate forgetting. • We conduct extensive experiments across diverse datasets and MLLM backbones, demonstrating that our proposed methods substantially outperform prior approaches in balancing downstream task performance and the preservation of general capabilities. 2 Related Work 2.1 Catastrophic Forgetting in MLLMs Multimodal Large Language Models (MLLMs)