Data Mixture Optimization:
2025
Read Data Mixture Optimization: (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21023.txt.
Description
The authors present a framework for choosing data mixtures in language model training in a 2025 paper. The mix of data sources shapes how well a model performs, yet choices often rest on intuition or costly trial and error. Scaling laws can help, but stretching small test results to large runs rests on brittle assumptions.
Their framework treats curation as a step by step decision problem. A Bayesian method weighs data mix, model size, and training length together, using cheap small runs to inform costly large ones while tracking uncertainty. A simulator is offered to speed work on such methods.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21023