The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs

Lucas Bandarkar Nanyun Peng · 2025

Read The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs by Lucas Bandarkar Nanyun Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18356.txt.

Introduction

Lucas Bandarkar and Nanyun Peng show, in a 2025 paper, that merging separately trained model parts transfers skills into low-resource languages surprisingly well. Large language models still stumble outside major languages, and training data for tasks like math barely exists in most tongues.

First, the authors confirm that the parameters driving math skill barely overlap those driving language skill. That separation invites modular training: tune a math expert and a language expert apart, then combine them. Their best recipe, layer swapping, beat baselines across three languages, four models, and two tuning styles. Undoing useless updates afterwards explained part of the magic.

Catalog record

CREATORLucas Bandarkar Nanyun Peng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs Lucas Bandarkar Nanyun Peng Abstract Large language models (LLMs) still struggle across tasks outside of high-resource languages. In this work, we investigate cross-lingual transfer to lower-resource languages where task-specific post-training data is scarce. Building on prior work, we first validate that the subsets of model parameters that matter most for mathematical reasoning and multilingual capabilities are distinctly non-overlapping. To exploit this implicit separability between task and target language parameterization, we develop and analyze numerous modular frameworks to improve the composition of the two during fine-tuning. These methods generally employ freezing parameters or post hoc model merging to assign math and language improvement to different key parts of the LLM. In the absence of in-language math data, we demonstrate that the modular approaches successfully improve upon baselines across three languages, four models, and two fine-tuning paradigms (full and LoRA). Furthermore, we identify the most consistently successful modular method to be fine-tuning separate language and math experts and model merging via Layer-Swapping ( Bandarkar et al. 2025 ) , somewhat surprisingly. We offer possible explanations for this result via recent works on the linearity of task vectors. We further explain this by empirically showing that reverting less useful fine-tuning updates after training often outperforms freezing them from the start. 1 Introduction Post-training large language models (LLMs) on labeled text data is a critical step in developing models for real-world applications. However, when these LLMs are fine-tuned for lower-resource languages, significant challenges arise due to the pretrained model’s limited capabilities. Although in recent years the broader scaling of pretraining and increased investment in additional languages Dang et al. 2024b ; Llama et al. 2024 have led to major improvements, pretrained LLMs still struggle to understand and generate text in all but a few languages ( Romanou et al. 2025 ; Qin et al. 2025 ) . This pretraining disparity is further exacerbated by the lack of available high-quality multilingual fine-tuning data ( Singh et al. 2024 ) and the significant cost to procure such annotated data (even through machine translation). For many of the capabilities developers target during post-training (e.g., instruction-following, reasoning, or safety) there are only sufficient open-source data available in English, Chinese, and a handful of other languages. This motivates the need for better cross-lingual transfer: the generalization of learned capacities from high-resource languages to lower-resource ones ( Hu et al. 2020 ; Philippy et al. 2023 ) . Despite recent releases of massive mixture-of-expert LLMs ( Team 2024b ; DeepSeek-AI et al. 2025 ; Team 2025 ) , a large majority of modern LLMs are dense , meaning that all parameters are active during training and inference. However, even within dense LLMs, recent works have found separability in where and how varying capabilities are represented ( Yin et al. 2024 ; Yao et al. 2024 ) . For example, multilingual capabilities are typically concentrated in the top and bottom transformer layers and multi-head attention parameters of an LLM ( Chang et al. 2022 ; Choenni et al. 2024 ) . This notably contrasts mathematical reasoning capabilities being encoded mainly in the middle transformer layers ( Hanna et al. 2023 ; Stolfo et al. 2023 ) . In the context of cross-lingual transfer, this functional separation motivates modular approaches to fine-tuning, which distinct model components can be trained, swapped, or merged ( Bengio et al. 2020 ; Pfeiffer et al. 2023 ) for efficient and flexible multi-objective optimization. Figure 1: Illustration of the three methods that induce modularity by imposing target language capabilities (brown) and mathematical reasoning (blue) on separate LLM parameters. [1] is from Bandarkar et al. 2025 In this work, we explore several modular approaches for composing target task and target language capabilities in off-the-shelf dense LLMs. Our goal is to induce modularity by exploiting the differences in parameters that are most relevant to mathematical reasoning versus multilingual capabilities. We focus on the prevalent scenario where task-specific data is scarce in the target language but readily available in English. We address this by working with two datasets; one English math dataset for supervised fine-tuning (SFT) and one general, multi-task SFT dataset in the target language. Using the target languages of Bengali, Swahili, and Telugu, we evaluate the methods on the multilingual math benchmark, MGSM ( Shi et al. 2023 ) . With these datasets, we evaluate numerous training paradigms that incentivize the model, to varying degrees, to learn multilingual or math capabilities in specific parameters. We organize the settings along two axes: (1) whether the models are optimized separately or together over the two SFT datasets and (2) whether the same or separate model parameters are trained on the datasets. When the models are trained separately, we combine the learned capabilities using model merging methods such as variants of Layer-Swapping ( Bandarkar et al. 2025 ) . To train separate model parameters, we start by dividing all parameters into two partitions according to prior work: (1) one set allocated to target language training and (2) one set to English math. Only allocated parameters are fine-tuned, while the opposite partition is frozen. We additionally develop a method to train separate parameters in a single, joint training by frequently freezing and unfreezing parameters to simulate simultaneous training. Despite the strong starting capabilities of the four LLMs and the data-constrained setting, our experimental results show that all of the modular solutions outperform our baselines, despite being subject to varying training constraints. This implies that intentional separation of parameters and/or training improves the compositionality of task and language capabilities. Amongst our modular solutions, we surprisingly find that post hoc model merging via Layer-Swapping outperforms more coordinated multi-task fine-tuning approaches. To contextualize this counterintuitive result, we explore recent academic literature that help explain the phenomenon. We provide empirical evidence for training all model parameters, even if large portions will be discarded during Layer-Swapping. While these subsets of task vectors are unproductive, freezing them during fine-tuning leads to less optimal updates to the target parameters. Notably, we rationalize that the fine-tuning task vectors ( \Delta s) are quite linear within individual parameter blocks ( Dai et al. 2025 ) , meaning they can be added, scaled, or interpolated as linear components ( Adilova et al. 2024 ) . Overall, we enumerate the following principal contributions of this work: • We develop and synthesize a number of modular solutions that each increase compositionality for cross-lingual transfer compared to non-modular baselines, demonstrated through extensive experiments. • Of the modular methods, we find that fine-tuning all parameters and then merging via Layer-Swapping performs best on average. • We provide a mix of theoretical and empirical explanations to explain the surprising success of Layer-Swapping relative to alternatives. Training Description Base Model Partial LoRA Partial SFT LoRA Full SFT Math-only 19.0% 18.0% 19.5% 18.9% 19.6% Language-only 19.0% 19.2% 19.8% 19.7% 20.3% Data mixing 19.0% - - 19.7% 20.4% Simultaneous SFT 19.0% 20.4% 21.0% - - Layer-Swapping 19.0% 20.0% 20.4% 20.8% 21.5% Table 1: Summary Table of Results. Each value represents the average across four models, three languages, and multiple training runs on MGSM in 2-shot evaluations. The last row represents “Separate Training” while the “Partial” trainings correspond to “Separate

Read the complete text