Revisiting Multilingual Data Mixtures in Language Model Pretraining

Negar Foroutan, Paul Teiletche, Ayush Kumar Tarun, Antoine Bosselut · 2025

Read Revisiting Multilingual Data Mixtures in Language Model Pretraining by Negar Foroutan, Paul Teiletche, Ayush Kumar Tarun, Antoine Bosselut (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25947.txt.

Description

Negar Foroutan, Paul Teiletche, Ayush Kumar Tarun, and Antoine Bosselut revisit how mixing many languages in pre training affects a language model. A common worry is the curse of multilinguality, the idea that adding languages must hurt each one. They train models with one point one and three billion parameters on corpora that range from twenty five to four hundred languages.

Foroutan and the team find that mixing English with other languages does not have to hurt either group, if each language gets enough tokens. Using English as a pivot, a high resource bridge language, helps across families, but a pivot from the same family is not reliably better. They also find no strong curse as language count grows.

Catalog record

CREATORNegar Foroutan, Paul Teiletche, Ayush Kumar Tarun, Antoine Bosselut
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25947

Browse the full library · About the Tower of Babel