Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Jingwei Li, Xinran Gu, Jingzhao Zhang · 2026
Read Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization by Jingwei Li, Xinran Gu, Jingzhao Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08022.txt.
Description
Li and colleagues study how to mix training data for large language models. Searching mixes on the target model costs much compute, while older mixture scaling laws extrapolate poorly to large models. They propose CAMEL, a capacity-aware mixture law that models validation loss through the joint effect of model size and mixture. A second law predicts benchmark accuracy from validation loss, and a budget plan spreads runs across scales. On Mixture-of-Experts models, their method cuts mixture costs by half and lifts downstream scores by up to 3%.
Catalog record
| CREATOR | Jingwei Li, Xinran Gu, Jingzhao Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49643 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08022