Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities

Xiaoyu Luo, Yiyi Chen, Johannes Bjerva, Qiongxiu Li · 2025

Read Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities by Xiaoyu Luo, Yiyi Chen, Johannes Bjerva, Qiongxiu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15722.txt.

Introduction

Xiaoyu Luo, Yiyi Chen, Johannes Bjerva, and Qiongxiu Li study memorization in language models that work in many languages in a 2025 paper. Such models can learn some training text by heart, which raises risk. Past work looked mostly at single language models and tied memorization to how much training text a language had.

The team checked 95 languages across models of unlike size and design. That simple tie did not fully explain the pattern. Using a graph measure that weighs how alike two languages are, they found that among close languages, the one with less training text often showed more memorization. The result argues for judging these models with language links in view.

Catalog record

CREATORXiaoyu Luo, Yiyi Chen, Johannes Bjerva, Qiongxiu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities Xiaoyu Luo, Yiyi Chen, Johannes Bjerva, Qiongxiu Li Abstract We present the first comprehensive study of Memorization in Multilingual Large Language Models (MLLMs), analyzing 95 languages using models across diverse model scales, architectures, and memorization definitions. As MLLMs are increasingly deployed, understanding their memorization behavior has become critical. Yet prior work has focused primarily on monolingual models, leaving multilingual memorization underexplored, despite the inherently long-tailed nature of training corpora. We find that the prevailing assumption, that memorization is highly correlated with training data availability, fails to fully explain memorization patterns in MLLMs. We hypothesize that the conventional focus on monolingual settings, effectively treating languages in isolation, may obscure the true patterns of memorization. To address this, we propose a novel graph-based correlation metric that incorporates language similarity to analyze cross-lingual memorization. Our analysis reveals that among similar languages, those with fewer training tokens tend to exhibit higher memorization, a trend that only emerges when cross-lingual relationships are explicitly modeled. These findings underscore the importance of a language-aware perspective in evaluating and mitigating memorization vulnerabilities in MLLMs. This also constitutes empirical evidence that language similarity both explains Memorization in MLLMs and underpins Cross-lingual Transferability , with broad implications for multilingual NLP 1 1 1 We release our code at: https://github.com/xiaoyuluoit97/MLLM_memorization . . 1 Introduction Large Language Models (LLMs) demonstrate increasingly strong capabilities in processing and understanding multiple languages Conneau et al. (2020) , resulting in advancements across a wide range of natural language processing (NLP) tasks Choi et al. (2021) ; Pikuliak et al. (2021) . MLLMs, in particular, empower global users to interact in their native languages, offering wide-reaching benefits in accessibility and productivity. Figure 1: Overview of our Framework for Analyzing Memorization in MLLMs using Language Similarity Graph-based Correlation Analysis. However, LLMs are also known to memorize portions of their training data Carlini et al. (2021) , raising serious concerns such as the leakage of copyrighted content Chang et al. (2023) and personal information Staab et al. () . While memorization in monolingual LLMs has been widely studied, how it manifests in multilingual models remains underexplored . Prior work predominantly attributes memorization to data volume, positing that frequent tokens or duplicated content are disproportionately memorized Carlini et al. (2022) . This echoes findings from computer vision, where long-tail examples are disproportionately memorized ( Feldman and Zhang, 2020 ; Jiang et al., 2020 ; Garg et al., 2023 ) , resulting in increased privacy and fairness risks Li et al. (2024b) ; Gao et al. (2023) ; Tramèr et al. (2022) . However, MLLMs introduce a unique complexity: languages are not processed independently but in a joint space, often sharing lexical, morphological, and syntactic features. While prior memorization research has largely focused on monolingual models and settings Carlini et al. (2021) ; Carlini et al. (2022) , without explicitly examining the role of cross-lingual similarity, our work explores how such relationships may shape memorization dynamics. For instance, typologically similar languages like Turkish and Azerbaijani may interact during training in ways that affect their memorization patterns. Moreover, low-resource languages naturally occupy the long tail of the data distribution, introducing complex dynamics that are poorly understood. Together, these challenges raise important questions that motivate our investigation. For example, to what extent does memorization in MLLMs correspond to training data volume, as suggested by long-tail distribution assumptions? How might cross-lingual relationships influence memorization behavior across languages? And can memorization in one language lead to unintended leakage in another, particularly among similar languages? To answer these questions, we conduct the first large-scale study of memorization in MLLMs, uncovering critical limitations of existing research and offering a novel language-aware perspective (see Fig. 1 for an overview of our framework). Our key contributions are: • Revisiting the Long-Tail Assumption : We show that memorization in multilingual settings cannot be fully explained by training data volume or token frequency. In many cases, low-resource languages exhibit lower memorization rates than high-resource counterparts. • Language Similarity-Aware Correlation Metric: We introduce a novel graph-based correlation metric that incorporates typological and statistical similarities between languages, enabling structured analysis of cross-lingual memorization dynamics. • Cross-Lingual Memorization Insights: Using our metric, we find that languages with high similarity exhibit interconnected memorization behaviors, affording fundamental grounding for cross-lingual transferability. • Comprehensive and Robust Evaluation: We assess memorization using both generation-based and likelihood-based metrics, and validate our findings across over 95 languages, multiple LLM architectures (encoder-only and decoder-based) of varying scales, demonstrating consistent and generalizable trends. 2 Related work 2.1 Memorization in LLM Memorization in deep neural networks has long been recognized as a critical issue, with implications for privacy, fairness, and generalization Feldman and Zhang (2020) ; Garg et al. (2023) ; Chang and Shokri (2021) ; Li et al. (2025) . These concerns have been empirically confirmed in LLMs. Carlini et al. (2019) first show that generative models can inadvertently memorize and reproduce rare, sensitive training data. Carlini et al. (2021) further demonstrate that large models like GPT-2 can regurgitate unique sequences even if they appear only once in the training corpus. Carlini et al. (2022) systematically quantify memorization patterns across model scales and architectures, while Kim et al. (2023) focus on personally identifiable information (PII) memorized by LLMs, proposing ProPILE to assess leakage from the perspective of data subjects. Recent work has formalized memorization risk, particularly distinguishing between discoverable and extractable memorization Carlini et al. (2021) ; Nasr et al. (2023) . The latter refers to information that an adversary can extract without direct access to the training set, posing realistic threats to deployed models. Studies have shown that LLMs, including GPT, T5, and others, can leak hundreds to millions of training sequences depending on model sizes, data duplication, and prompt strategies Nasr et al. (2023) ; Carlini et al. (2022) . Beyond quantifying leakage, several studies have advanced the understanding of memorization mechanisms and measurement approaches. Chen et al. (2024) analyze how model and context size affect transitions between unmemorized and memorized outputs. Liu et al. (2024) propose the forgetting curve, a corpus-agnostic method to reliably measure memorization capability across architectures. Li et al. (2024a) introduce ROME, revealing how token length and prediction confidence relate to memorization without relying on training data access. Haviv et al. (2022) demonstrate that recall of memorized sequences follows a two-stage process of early promotion and later confidence amplification in transformer models. Stoehr et al. (2024) localize memorization to specific low-layer attention heads and high-gradient parameters, showing such content is harder to unlearn. While such risks have been studied in monolingual settings, memorization behavior in multilingual LLMs remains underexplored, with the exception of Cavalin et al. (2024) , especially for low-resource languages occupying the long tail of the training distribution. 2.2 Cross-lingual Transferability & Language Similarity Cross-lingual transfer entails the representation of texts in multiple natural languages in a shared multilingual space. The paradigm of representations for cross-lingual transfer has shifted from word embeddings ( Mikolov et al., 2013 ; Ammar et al., 2016 ; Vulić et al., 2019 ) to contextual embeddings ( Conneau et al., 2019 ; Devlin et al., 2019 ; Raffel et al., 2020 ) . Previous work investigating cross-lingual transferability mainly

Read the complete text