Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics
Carter Blum* Katja Filippova* Ann Yuan* Asma Ghandeharioun Julian Zimmert, Fred Zhang Jessica Hoffmann Tal Linzen Martin Wattenberg Lucas Dixon Mor Geva · 2025
Read Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics by Carter Blum* Katja Filippova* Ann Yuan* Asma Ghandeharioun Julian Zimmert, Fred Zhang Jessica Hoffmann Tal Linzen Martin Wattenberg Lucas Dixon Mor Geva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11017.txt.
Introduction
Carter Blum and co-authors study a weak spot in large language models in a 2025 paper. A model may learn a fact in one language yet fail, or make up an answer, when asked in another. To find why, the team trained small models from scratch on made-up data in many languages.
Two things shaped the outcome. One was how much the language of a fact gave away the fact itself. The other was how easy the language was to spot. When facts and language were loosely tied, models built shared internal forms and facts crossed languages well. When the tie was tight, forms stayed split. The team uses this view to explain past results and to suggest better training.
Catalog record
| CREATOR | Carter Blum* Katja Filippova* Ann Yuan* Asma Ghandeharioun Julian Zimmert, Fred Zhang Jessica Hoffmann Tal Linzen Martin Wattenberg Lucas Dixon Mor Geva |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11017
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics Carter Blum* Katja Filippova* Ann Yuan* Asma Ghandeharioun Julian Zimmert, Fred Zhang Jessica Hoffmann Tal Linzen Martin Wattenberg Lucas Dixon Mor Geva Abstract Large language models (LLMs) struggle with cross-lingual knowledge transfer: they hallucinate when asked in one language about facts expressed in a different language during training. This work introduces a controlled setting to study the causes and dynamics of this phenomenon by training small Transformer models from scratch on synthetic multilingual datasets. We identify a learning phase wherein a model develops either separate or unified representations of the same facts across languages, and show that unification is essential for cross-lingual transfer. We also show that the degree of unification depends on mutual information between facts and training data language, and on how easy it is to extract that language. Based on these insights, we develop methods to modulate the level of cross-lingual transfer by manipulating data distribution and tokenization, and we introduce metrics and visualizations to formally characterize their effects on unification. Our work shows how controlled settings can shed light on pre-training dynamics and suggests new directions for improving cross-lingual transfer in LLMs. 1 Introduction Language models hallucinate facts. This behavior has been attributed to training and sampling noise, gaps in pretraining data ( Xu et al. 2024 ) , and misaligned incentives in post-training ( Schulman 2023 ) . However, these fail to explain cross-lingual factual errors: cases where models accurately answer questions when posed in the same language as the training data, yet hallucinate when prompted in a different (often lower-resource) language ( Goldman et al. 2025 ) . Failures of cross-lingual transfer exacerbate disadvantages faced by speakers of underrepresented languages, and increasing model scale does not solve the problem ( Aggarwal et al. 2025 ; Qi et al. 2023 ) . LLMs have been found to develop both a lingua franca for factual knowledge (typically based on English) and distinct language silos ( Aggarwal et al. 2025 ; Lim et al. 2025b ; Schut et al. 2025 ; Lim et al. 2025a , inter alia) , and their hidden representations can be language-agnostic or language-specific depending on the layer ( Wang et al. 2025 ) . However the root cause of these phenomena is not understood, as most research on cross-lingual transfer analyzes models as static artifacts. Such analysis, while valuable, cannot explain how knowledge arises during training, and therefore cannot lead to effective pre-training interventions. While some have investigated the training dynamics of knowledge acquisition in multilingual LLMs ( Zeng et al. 2025 ; Liu et al. 2025 ) , their approach is non-interventional and does not establish a causal link between data properties and cross-lingual transfer. In this work, we study what causes, and how to mitigate cross-lingual hallucinations. We use a “Petri dish” methodology, training small transformer models from scratch on synthetic datasets and systematically varying their distributional properties. This setup allows us to analyze models’ learning dynamics during pre-training. In particular, we identify a crucial early phase where a model develops either unified or separate representations for identical facts across languages, and find that the degree of representational unification, computed over training examples, is predictive of cross-lingual generalization in the fully trained model. Our study reveals two primary causes of unification. First, expressing the same information in different languages facilitates the development of shared cross-lingual representations. This builds on findings from monolingual research ( Allen-Zhu 2024 ) where including multiple paraphrases of the same fact in training was found to improve recall. Second, and more surprisingly, we find that the distributional properties of the monolingual (non-parallel) portion of the dataset can induce representational separation. Namely, separation occurs when the language of an example is both easy to extract, and itself a useful prior for predicting the response distribution. Beyond our synthetic setting, our framework explains the role of shared script in cross-lingual transfer that has been consistently observed ( Patil et al. 2022 ; Qi et al. 2023 ) . In addition, we show that our measures of representational unification correlate with cross-lingual factual accuracy in the Gemma model family ( Riviere et al. 2024 ) . In summary, our core contributions are: 1. We introduce a Petri dish setup in which same-language generalization is reliably observed, while cross-lingual transfer can be independently modulated (Sec. 2 ). 2. We analyze pre-training dynamics and identify a crucial early phase where a model develops either unified or separate representations (Sec. 3 ). 3. We show that cross-lingual transfer can be improved without increasing the amount of multi-lingual data, but rather by changing properties of the monolingual data such that the model takes longer to learn the language feature (Sec. 4 ). 4. We introduce a metric to characterize unification of representations across languages that is strongly predictive of cross-lingual knowledge transfer (Sec. 5 ). 5. We extend our analyses to LLMs and show that our unification metric correlates with cross-lingual factual accuracy in Gemma 2B (Sec. 6 ). Finally, we note that our Petri dish model of cross-lingual knowledge transfer could be interpreted more generally as a model of transfer across semantic paraphrases. Thus, we believe our study can have applications beyond factual recall - we discuss these in Sec. 7 . 2 Cross-lingual Factual Recall in a Petri Dish (a) KG is populated with events of two types (b) Templates to express birth-city facts in EN and ES Figure 1: Birth and death events are created for every entity by sampling from a set of birth and death years (disjoint) and cities (same pool). A dataset is comprised of monolingual (events expressed in either EN or ES ) and parallel (events expressed in both languages) data. Arrows point from a particular event to the training templates (solid line) or in-language test set templates (dashed). All verbalizations in the other language are part of the cross-lingual test set. For simplicity, only birth templates with arg0, arg2 are shown. In this section we describe our Petri dish methodology, from creating pre-training data to training and evaluating models. First, we create a synthetic knowledge graph by generating language-agnostic events (e.g., birth, death ). From a single event we derive multiple facts , each of which has subject and attribute arguments. For example, it may be a fact that Alice Brown (subject) was born in Berlin (birth-place attribute), or that Alice Brown was born in the year 1902 (birth-year attribute) (see Fig. 1(a) ). In our experiments the attributes are sampled uniformly, which naturally results in a variability in attribute frequencies (e.g., there may be more people born in Berlin than in Paris ). We create events by deciding on the population size (e.g., 1,000 individuals) and the attribute sets (e.g., 100 cities where one can be born or die) and adding a single event for every person and event type. Once built, the KG is frozen and its events serve as the basis for all subsequent experiments. That is, all training datasets describe the same set of facts. However while their informational content is identical, datasets vary with respect to how facts are expressed. Synthetic Languages We develop synthetic languages to express the KG. Each