From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora

Yingli Shen, Wen Lai, Shuo Wang, Ge Gao, Kangyang Luo, Alexander Fraser, Maosong Sun · 2025

Read From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora by Yingli Shen, Wen Lai, Shuo Wang, Ge Gao, Kangyang Luo, Alexander Fraser, Maosong Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14045.txt.

Introduction

Yingli Shen and six co-authors build a multilingual dataset for language models in a 2025 paper. Training on mixed-language text helps models learn rare languages, but unaligned text teaches cross-language meaning poorly. Texts matched line by line across many languages give a stronger signal. Their corpus draws on translated talks.

The dataset spans over a hundred languages, with up to fifty versions aligned for one talk. They test the best ways to use such matched data, from further pretraining to tuning on instructions. Across six multilingual benchmarks, models trained on matched data beat those trained on unaligned text. The work offers a path to better models for low-resource languages.

Catalog record

CREATORYingli Shen, Wen Lai, Shuo Wang, Ge Gao, Kangyang Luo, Alexander Fraser, Maosong Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE147 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14045

Browse the full library · About the Tower of Babel