Data Mixture Optimization:

2025

Read Data Mixture Optimization: (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21023.txt.

Description

The authors present a framework for choosing data mixtures in language model training in a 2025 paper. The mix of data sources shapes how well a model performs, yet choices often rest on intuition or costly trial and error. Scaling laws can help, but stretching small test results to large runs rests on brittle assumptions.

Their framework treats curation as a step by step decision problem. A Bayesian method weighs data mix, model size, and training length together, using cheap small runs to inform costly large ones while tracking uncertainty. A simulator is offered to speed work on such methods.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21023

Browse the full library · About the Tower of Babel