What Causes Knowledge Loss in Multilingual Language Models?
Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti, Genta Indra Winata · 2025
Read What Causes Knowledge Loss in Multilingual Language Models? by Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti, Genta Indra Winata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20356.txt.
Introduction
Khelli, Cahyawijaya, Purwarianti, and Winata ask what causes knowledge loss in multilingual language models, in a paper from twenty twenty-five. Fine-tuning on new languages can wipe out earlier learning, a flaw called catastrophic forgetting. Their study homes in on how the gaps between languages shape the model's inner form. Fifty-two languages are tested, with adapters of ranks that vary.
Adapters share their parameters in full, in part, or not at all across languages. Scripts that are not Latin prove more at risk of loss. Latin-script languages back stronger transfer across tongues. Sharing parameters through adapters can cut the loss while keeping knowledge safe. The choice of script turns out to be a real design point in practice.
Catalog record
| CREATOR | Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti, Genta Indra Winata |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Causes Knowledge Loss in Multilingual Language Models? Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti, Genta Indra Winata Abstract Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer. 1 Introduction Cross-lingual transfer in natural language processing (NLP) models has demonstrated enhanced performance in multilingual contexts compared to monolingual settings, largely due to the advantages of leveraging cross-lingual knowledge Hu et al. 2020 ; FitzGerald et al. 2023 ; Winata et al. 2023b ; Winata et al. 2024 . Typically, training occurs only once simultaneously, where all available data is processed in a single training run. However, in real-world applications, data is often received sequentially over time, highlighting the importance of continuous model updates to maintain performance Rolnick et al. 2019 . Unlike humans, who can retain prior knowledge while acquiring new skills, neural network models often struggle to preserve previously learned information when fine-tuned on new tasks, which known as catastrophic forgetting, a decline in performance on earlier tasks after the model is exposed to new data Winata et al. 2023a . To mitigate this issue, several studies have investigated continual learning strategies and the implementation of adapters Badola et al. 2023 as viable solutions. This limitation poses a significant challenge for multilingual NLP, as models must adapt to new languages while retaining previously acquired linguistic knowledge. Without an effective learning strategy, models risk performance degradation, rendering them less suitable for long-term deployment. Figure 1: Pipeline for various approaches in lifelong learning. In our lifelong learning framework, we employ a LoRA-based approach where the parameters of the base model, denoted as \theta , remain fixed, and for VANILLA, the model parameters are updated at all times. We explore the phenomenon of multilingual knowledge loss by comparing the effects of training with both shared and non-shared parameters. Lifelong learning is essential for integrating new annotated data across languages without requiring full retraining of systems. As language evolves and new data becomes available, models must adapt incrementally to minimize computational costs. This approach helps maintain efficiency and scalability, while addressing the challenge of catastrophic forgetting, which has been explored in various studies Liu et al. 2021 ; Winata et al. 2023a ; Badola et al. 2023 ; M’hamdi et al. 2023 . However, there is a lack of systematic analysis on this issue in multilingual contexts. This study aims to fill that gap by investigating factors contributing to catastrophic forgetting beyond model parameters, including how linguistic differences can affect representational learning and lead to knowledge erosion when learning multiple languages sequentially. In this study, we investigate the effects of non-shared, partially shared, and fully shared parameters in a multilingual context, examining 52 languages through the use of LoRA adapters with varying ranks and different sharing model parameter settings as shown in Figure 1 . Our primary focus is to assess the impact of parameter sharing on model performance, while also conducting a comprehensive analysis of the role that different languages play in catastrophic forgetting. Additionally, we explore sequential learning to identify when performance drops occur and whether these declines are influenced by the introduction of newly learned languages or the cumulative number of previously learned languages. Our contributions can be summarized as follows: • We examine the factors contributing to knowledge loss in multilingual language models, focusing on aspects such as language diversity, parameter sharing strategies, and base model selection within a lifelong learning framework for massively multilingual learning. • We assess cross-lingual transferability and introduce multi-hop metrics to better understand the impact of language skills on model performance. • We analyze model parameter adaptation to investigate trends in the model’s ability to learn languages in a lifelong learning context. 2 Methodology 2.1 Task Setup A sequence of T tasks is structured as an ordered set of datasets \mathcal{D}=\{D_{1},D_{2},\dots,D_{t},\dots,D_{T}\} , where each dataset D_{t} corresponds to a specific task t , representing a distinct language. The model, parameterized by \theta_{t} , undergoes iterative updates, with parameters at step t+1 being derived from those at step t through the function f(\theta_{t},D_{t}) . These updates are performed using gradient-based optimization to maximize the log-likelihood over dataset D_{t} . 2.2 Training Methods We use XLM-R {}_{\text{BASE}} Conneau et al. 2020 as our base model and compare key methods with E5 instruct Wang et al. 2024 for evaluating the consistency of the findings. A classification layer is added on top of the encoder model, tailored sequence label of the slot filling. For adapter-based approaches, only the parameters within the adapter modules are updated during training. MULTI . A single model is trained on all languages simultaneously, optimizing over the entire dataset \mathcal{D} : \displaystyle\theta_{\text{MULTI}}=\arg\max_{\theta}\sum_{t=1}^{T}\log p(D_{t}\mid\theta). (1) MONO . Each language/task has its own independently trained model \theta_{t} : \displaystyle\theta_{t}=\arg\max_{\theta}\log p(D_{t}\mid\theta), (2) \displaystyle\forall t\in\{1,\dots,T\}. (3) VANILLA . A single model is trained incrementally, updating parameters sequentially: \displaystyle\theta_{t+1}\leftarrow f(\theta_{t},D_{t}),\forall t\in\{1,\dots,T-1\}. (4) SHARED LoRA . A single LoRA adapter \phi is trained while keeping the base model \theta_{0} frozen: \displaystyle\phi_{s}\leftarrow f(\phi^{\prime}_{t},D_{t}),\theta=\theta_{0},\forall t\in\{1,\dots,T-1\}. (5) NON-SHARED LoRA . Each language has its own separate LoRA adapter \phi_{t} , while keeping the base model \theta_{0} frozen: \displaystyle\phi_{t}=\arg\max_{\phi}\log p(D_{t}\mid\theta_{0},\phi),\forall t\in\{1,\dots,T\}. (6) The specific ordering of languages used in the VANILLA is specified in Appendix Table 3 . 2.3 Model Parameters Adaptation We utilize low-rank adapters LoRA Hu et al. 2021 for training parameters to analyze the effectiveness to have sharing parameters. It is a parameter-efficient fine-tuning method for large pre-trained models leveraging the intrinsic low-dimensionality of parameter updates, reducing the need for full model adaptation. Instead of modifying dense layers directly, It freezes the pre-trained weights and introduces trainable low-rank matrices, significantly minimizing the number of learnable parameters and enhancing fine-tuning efficiency. Formally, given a pre-trained weight matrix W_{0}\in\mathbb{R}^{d\times k} , LoRA constrains the update \Delta W to a low-rank decomposition: \Delta W=BA, (7) where B\in\mathbb{R}^{d\times r} and A\in\mathbb{R}^{r\times k} , with rank r\ll\min(d,k) . This decomposition ensures that only A and B are updated while W_{0} remains fixed. Consequently, the forward pass is expressed as: h=W_{0}x+\Delta Wx=W_{0}x+BAx, (8) where x is the input vector, and h is the output. The low-rank update \Delta Wx is scaled by a constant factor \frac{\alpha}{r} , analogous to a learning rate, to regulate the magnitude of the update. LoRA offers key advantages: it enhances memory and computational efficiency by limiting trainable parameters, reducing resource requirements, and enabling modular fine-tuning. Its linear structure ensures no additional inference latency , while its linear property allows seamless integration. By leveraging low-rank adaptation, LoRA enables scalable and efficient model adaptation without compromising