From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
Yingli Shen, Wen Lai, Shuo Wang, Ge Gao, Kangyang Luo, Alexander Fraser, Maosong Sun · 2025
Read From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora by Yingli Shen, Wen Lai, Shuo Wang, Ge Gao, Kangyang Luo, Alexander Fraser, Maosong Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14045.txt.
Introduction
Yingli Shen and six co-authors build a multilingual dataset for language models in a 2025 paper. Training on mixed-language text helps models learn rare languages, but unaligned text teaches cross-language meaning poorly. Texts matched line by line across many languages give a stronger signal. Their corpus draws on translated talks.
The dataset spans over a hundred languages, with up to fifty versions aligned for one talk. They test the best ways to use such matched data, from further pretraining to tuning on instructions. Across six multilingual benchmarks, models trained on matched data beat those trained on unaligned text. The work offers a path to better models for low-resource languages.
Catalog record
| CREATOR | Yingli Shen, Wen Lai, Shuo Wang, Ge Gao, Kangyang Luo, Alexander Fraser, Maosong Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 147 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14045