A Post-trainer’s Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics

Luísa Shimabucoro†, Ahmet Üstün, Marzieh Fadaee, Sebastian Ruder · 2025

Read A Post-trainer’s Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics by Luísa Shimabucoro†, Ahmet Üstün, Marzieh Fadaee, Sebastian Ruder (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16677.txt.

Introduction

Luisa Shimabucoro and co-authors map how skill in one tongue leaks to kin tongues when chat models take late train on mixed word sets, in a paper from 2025. Late train on tales in many tongues is how models go world wide, yet leak rules stay dim. Two model clans to vast size took turns on set mixes for three task kinds.

Sum work, rule heed, and math thought all ride in the tests, solo and mixed. Leak strength shifts with each mix of task, tongue share, and clan, with no lone term in charge. The team names the mix terms that make leak pay in the field. Past lore of one term at a time fails here.

Catalog record

CREATORLuísa Shimabucoro†, Ahmet Üstün, Marzieh Fadaee, Sebastian Ruder
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Post-trainer’s Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics Luísa Shimabucoro†, Ahmet Üstün, Marzieh Fadaee, Sebastian Ruder Abstract In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a clear understanding of the dynamics that enable cross-lingual transfer remains elusive. This study examines cross-lingual transfer (CLT) dynamics in realistic post-training settings. We study two model families of up to 35B parameters in size trained on carefully controlled mixtures of multilingual data on three generative tasks with varying levels of complexity (summarization, instruction following, and mathematical reasoning) in both single-task and multi-task instruction tuning settings. Overall, we find that the dynamics of cross-lingual transfer and multilingual performance cannot be explained by isolated variables, varying depending on the combination of post-training settings. Finally, we identify the conditions that lead to effective cross-lingual transfer in practice. † † footnotetext: Corresponding author: [email protected] 1 Introduction Figure 1: Overview of our experimental framework. We investigate cross-lingual transfer performance improvements during the instruction tuning stage by varying: 1) task type; 2) fine-tuning setting (single-task or multi-task); 3) quantity of multilingual data; and 4) model size. Runs use a fixed amount of English data and increasing amounts of multilingual data. Instruction tuning is crucial for large language models (LLMs) to develop the ability to follow instructions and to complete more complex tasks. ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) Despite its broad adoption and the continued development of strong multilingual LLMs ( Dubey et al. 2024 ; Aryabumi et al. 2024 ) , performance transfer dynamics during multilingual post-training remain unclear. Creating high-quality instruction data is expensive ( Dubey et al. 2024 ) , which makes it infeasible to create large amounts of data in every language for each task. In order to efficiently post-train LLMs in such resource-constrained multilingual settings, they must rely on cross-lingual transfer from the large amount of available English data, combined with limited multilingual data. Understanding the dynamics of cross-lingual transfer during post-training is thus key to building robust models that can excel in increasingly complex tasks across a diverse set of languages. Prior work has examined cross-lingual transfer (CLT) behavior at an aggregate level ( Muennighoff et al. 2023 ; Shaham et al. 2024 ; Chen et al. 2024a ) , but less attention has been given to how dynamics vary across individual tasks, languages, and training scenarios. However, such exploration is critical for developing more robust and generalizable multilingual models, as it allows for targeted improvements that account for the unique challenges posed by specific linguistic and task-related contexts. In this work, we perform a systematic investigation of multilingual post-training, expanding our current understanding of cross-lingual transfer dynamics. We examine models’ scaling behavior through the lens of multilingual data mixtures during post-training with regard to three important dimensions: task type, model size, and training setting (i.e., single-task or multi-task) across 12 languages. Prior work ( Kew et al. 2024 ; Shaham et al. 2024 ; Chen et al. 2024a ) has been limited by focusing on small-scale models, a single general instruction-following task, and small data budgets. We go beyond these by assessing models with both 7B and 35B parameters on 3 tasks with varying complexity levels (summarization, general instruction-following, and mathematical reasoning). Furthermore, we investigate cross-lingual transfer with incrementally increasing multilingual data, using up to 75k unique samples per run, considerably more than prior works. Lastly, we go beyond training on a single task and study the dynamics of training jointly on multiple tasks and languages. Task Train data Test data Metric Seen langs. Unseen langs. Summarization XLSum XLSum RougeL Instruction Following ShareGPT Aya-Dolly200 LLM-as-a-judge en,es,fr,ja,zh ar,ko,pt Mathematical Reasoning mCoT-Math MGSM Accuracy Table 1: Dataset references: XLSum ( Hasan et al. 2021 ) , ShareGPT, Aya-Dolly200 ( Singh et al. 2024 ) , mCoT-Math ( Lai and Nissim 2024 ) , MGSM ( Shi et al. 2022 ) . Overall, our findings are the following: 1. Multilingual performance improvements—but not cross-lingual transfer—are dependent on task type . Different tasks scale differently when presented with more multilingual data (e.g. mathematical reasoning benefits much more from adding more data) while cross-lingual transfer behaves in a similar way across all tasks. Unseen performance tracks seen performance and despite CLT enabling improvements for the unseen languages it does not enable bridging the performance gap between seen and unseen languages. 2. Cross-lingual transfer manifests differently on models of varying scale . We find that with increasing scale CLT becomes more efficient and the performance gap between seen and unseen languages narrows. At scale, most performance gains are realized using English data only and/or with little multilingual data. 3. Single and multi-task training settings scale differently . We demonstrate the impact of the training setting where task interference can lead to fluctuations in performance and a wider seen–unseen languages performance gap in multi-task learning. These effects, however, diminish at scale. 4. Different tasks benefit from different data mixtures . While linguistically-oriented tasks (e.g., instruction following, summarization) benefit from data in non-Latin script languages (which tend to have a lower zero-shot performance), for reasoning tasks (e.g., mathematical reasoning) the model learns more efficiently by training on Latin script data only. 2 Realistic Mulilingual Post-training Our goal is to conduct a comprehensive exploration of multilingual post-training dynamics that reflects the settings of state-of-the-art multilingual LLMs ( Dubey et al. 2024 ; Aryabumi et al. 2024 ) . We make several assumptions to operationalize these settings in our experiments: English data availability We assume that sufficient English data is available for each task. We sample a fixed set of English samples for each task and vary the amount of multilingual data. Single-task experts vs multi-task generalists While it is common to adapt models to specific tasks—often via parameter-efficient fine-tuning ( Pfeiffer et al. 2023 ) — state-of-the-art models are generally trained on many tasks at once ( Dubey et al. 2024 ; Aryabumi et al. 2024 ) . Consequently, we investigate the cross-lingual transfer dynamics in both single and multi-task instruction tuning. Model scale Most prior works consider only models up to 7B models. However, as some behavior only emerges at scale ( Wei et al. 2022 ) , we hypothesize that cross-lingual transfer dynamics also vary with model size. We therefore perform all experiments on both 7B and 35B base models. Language selection and uniform sampling Prior work frequently trained jointly on only two languages ( de Souza et al. 2024 ; Zhang et al. 2024 ) while state-of-the-art models are trained on data across multiple languages. In order to ensure comparable results across tasks, we train on English data in addition to a fixed set of 4 languages (Spanish, French, Japanese, and simplified Chinese) from which we sample uniformly. † † † Uniform sampling has been shown to be the most efficient sampling strategy in multilingual settings ( Khanuja et al. 2023 ) . We evaluate models on these languages as well as a set of unseen languages for each task. 3 Experimental Setting Figure 2: Average performance across seen and unseen languages relative to English across different tasks for 7B (left) and

Read the complete text