MixMin: Finding Data Mixtures via Convex Minimization

Anvith Thudi, Evianne Rovers, Yangjun Ruan, Tristan Thrush, Chris J. Maddison · 2025

Read MixMin: Finding Data Mixtures via Convex Minimization by Anvith Thudi, Evianne Rovers, Yangjun Ruan, Tristan Thrush, Chris J. Maddison (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10510.txt.

Introduction

Blending training data from many sources is now standard, yet the best recipe proportions are usually guessed. Anvith Thudi and four co-authors formalize that guesswork in a 2025 paper. Their framing is two-level: the best mixture is the one whose trained model scores best on the target task, a goal that is normally out of reach.

The key insight is that this hard goal turns convex, and therefore solvable, as the model class grows large. Their gradient method, MixMin, optimizes the mixture directly. It was the only method that improved every mixture it touched in their tests. On a mid-size language model, less than 0.2 percent extra compute bought gains of 1 to 5 percent on several test sets. Mixtures tuned on small models also lifted larger ones, a hint that good recipes travel across scale.

Catalog record

CREATORAnvith Thudi, Evianne Rovers, Yangjun Ruan, Tristan Thrush, Chris J. Maddison
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MixMin: Finding Data Mixtures via Convex Minimization Anvith Thudi, Evianne Rovers, Yangjun Ruan, Tristan Thrush, Chris J. Maddison Abstract Modern machine learning pipelines are increasingly combining and mixing data from diverse and disparate sources, e.g., pre-training large language models. Yet, finding the optimal data mixture is a challenging and open problem. We formalize this data mixing problem as a bi-level objective: the best mixture is the one that would lead to the best model for a downstream objective. Unfortunately, this objective is generally intractable. In this paper, we make the observation that the bi-level data mixing objective becomes convex as our model class becomes larger. We develop and study a gradient-based approach for optimizing this convex objective, which we call MixMin, and test it on language modeling and chemistry tasks. MixMin was the only method that uniformly improved the data mixture in all our experiments. With MixMin, we improved the data mixture using less than 0.2 % 0.2\% additional compute for a pythia- 410 ​ M 410M model trained on 8.2 ​ B 8.2B tokens, resulting between 1-5% relative improvement to negative log likelihood on PIQA, ARC Easy, SciQ, and OpenWebMath. Crucially, we found that MixMin mixtures for smaller models improved training of larger models, suggesting that MixMin mixtures may be scale-invariant. When mixing bioassay data to train an XGBoost model, we saw improvements to average precision scores of 0.03 − 0.15 0.03-0.15 . 1 Introduction Figure 1 : Optimizing M ixMin to find the mixture weights requires training a few cheap models for each source and a target dataset. Given the mixture weights we train a more expensive model using the mixture. Recent progress in ML has come from training on vast web-scale data ( Dubey et al., 2024 ; Touvron et al., 2023 ; Raffel et al., 2020 ; Achiam et al., 2023 ) . These models are known to generalize to data-poor downstream tasks, a core motivation behind scaling training data. However, as we increase the amount and diversity of data sources we can train on, a core challenge becomes choosing how to weigh these sources when training. The effectiveness of a model pre-trained on surrogate data is known to be directly related to how similar the surrogate (or “pretraining”) data is to the downstream task (target distribution) ( Ben-David et al., 2010 ; Isik et al., 2024 ; Jain et al., 2024 ; Pouget et al., 2024 ) . Formally, finding the best mixture of data sources to train on for a downstream loss poses a bi-level optimization which is hard to optimize. We must find the mixture whose risk minimizer optimizes the downstream loss. This generally only admits expensive zero-order approaches, such as grid search ( Dubey et al., 2024 ; Blakeney et al., 2024 ) . Past work has considered learning to simulate the loss oracle for unseen mixtures ( Liu et al., 2024 ) , or leveraging a large pool of existing models ( Thrush et al., 2024 ) . Other approaches consider solving data mixing objectives agnostic to a downstream loss, but are known to not consistently improve the baseline of training on all the data with their natural proportions (when evaluated on diverse downstream losses) ( Xie et al., 2024 ; Jiang et al., 2024 ; Liu et al., 2024 ; Fan et al., 2023 ; Held et al., 2025 ) . In this paper we show that the bi-level optimization for data mixing reduces to a convex minimization as the model classes become larger, which we can solve cheaply with gradient based methods. A key insight is that by restricting the loss functions to cross-entropy (CE) or mean-squared error (MSE), we gain additional structure which simplifies the data mixing objective. Practically, we show we can find the best mixture by first training a (cheap) proxy model for each source, learning the best mixture of their outputs for the target dataset (convex minimization), and then using this weighting to remix the data, see Figure 1 . First, we note that when our models are relatively close to Bayes optimal (the best possible functions), data mixing is equivalent to learning a linear model over the predictions of a fixed set of models (a convex objective). Learning this linear model requires comparatively little data to training modern large models, and is relatively cheap to optimize. Our reduction is specific to CE or MSE, where we leverage the fact the Bayes optimal model for a mixture is the mixture of the Bayes optimal models for each source. However, this reduction does not yet make data mixing amenable to first-order optimization. To evaluate our convex objective’s gradient we need the Bayes optimal model for each source. A key challenge is that the proxy models needed to evaluate this convex objective can be expensive to compute. We find that in practice, we can use cheaply computed models and still have good data mixing performance. We call this approach to optimizing the convex objective M ixMin . Empirically, we found the performance of M ixMin did not significantly degrade between using proxy models computed with 100\% the cost of a full training run to 1\% . In other cases, the M ixMin mixture significantly improved over baselines, but the ensemble of proxy models performed much worse than retraining, suggesting they were far from Bayes optimal but did not hinder M ixMin performance. We empirically compared data mixing with M ixMin to current baselines on both language modeling and chemistry tasks. For language modeling we considered optimizing the mixture of source domains in SlimPajama ( cer, 2023 ) for PIQA ( Bisk et al., 2020 ) , ARC Easy ( Clark et al., 2018 ) , SciQ ( Welbl et al., 2017 ) , or OpenWebMath ( Paster et al., 2023 ) . In all cases we found M ixMin improved the baselines for log likelihood, improving or maintaining the gap as we increased the scale of the models from pythia- 160M to pythia- 410M ( Biderman et al., 2023 ) . Moreover, for pythia- 410M the mixtures we used took ~0.15\% the compute of the training run to find and improved the negative log likelihood by 1-5\% . For our chemistry experiments we considered mixing assay datasets in PubChem ( Beaini et al., 2023 ; Kim et al., 2016 ) to improve the performance of an XGBoost ( Chen & Guestrin, 2016 ) model on several held-out assay datasets. In all cases, we observed MixMin improved over the natural distribution (the standard data mixture after filtering for this domain ( Salem et al., 2020 ; Li et al., 2022 ; Ye et al., 2018 ) ) as we increased the number of surrogates to mix over, improving average precision scores between 0.03-0.15 . We note an additional benefit of optimizing data mixtures for chemistry tasks is that the found M ixMin weights could provide interpretability; we highlight patterns M ixMin found in PCBA for predicting assays (e.g., distinct but predictive cytotoxicity assays). To summarize, our contributions are: 1. Observing data mixing reduces to a single convex minimization as model classes become more expressive 2. Proposing M ixMin which approximately solves the convex objective by using cheap proxy

Read the complete text