Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma · 2026
Read Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time by Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15220.txt.
Introduction
Michael Y. Hu and four others argue that data mixing is an online decision recurring across all of language-model training, not a one-phase problem. Old methods serve one phase, need proxy models, or assume fixed domains. Their OP-Mix algorithm mocks candidate mixtures by interpolating low-rank adapters trained on the current model, grounding search in actual learning shifts without other proxies. Across pretraining, ongoing mid-training, and ongoing instruction tuning, OP-Mix finds near-optimal mixtures at a fraction of rival compute.
Catalog record
| CREATOR | Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53814 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma Abstract Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key determinant of model quality; in continual learning and adaptation, it governs what is retained and acquired. Yet existing data mixing methods address only one phase of this lifecycle at a time: some require smaller proxy models tied to a single training phase, others assume a fixed domain set, and continual learning lacks principled guidance altogether. We argue that data mixing is fundamentally an online decision making problem—one that recurs throughout training and demands a single, unified solution. We introduce OP-Mix (On-Policy Mix), a data mixing algorithm that operates across the entire language model training lifecycle. Our main insight is that candidate data mixtures can be cheaply simulated by interpolating between low-rank adapters trained directly on the current model, eliminating separate proxy models and ensuring the search is always grounded in the model’s actual learning dynamics. Across pretraining, continual midtraining, and continual instruction tuning, OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of the baselines. In pretraining, OP-Mix improves upon training without mixing by 6.3% in average perplexity. For continual learning, OP-Mix matches the performance of both retraining and on-policy distillation while using 66% and 95% less overall compute, respectively. OP-Mix suggests a different view of language model training: not a sequence of distinct phases, but a single continuous process of learning from data. Figure 1 : Overview of OP-Mix . OP-Mix aims to cheaply estimate optimal data mixing ratios in a continual setting. (1) Train a lightweight LoRA adapter on new domains to estimate future performance. (2) Interpolate adapters to simulate different data mixtures without retraining and then estimate the optimal mixture ratio. (3) Train the base model with the computed mixture. Figure 2 : OP-Mix (purple) Pareto-dominates the performance-efficiency frontier when tested alongside baselines across pretraining, continual midtraining, and continual instruction tuning. 1 Introduction Language models are trained on carefully curated data mixtures, yet the science of constructing the right mixture remains nascent. The dominant approach—training small proxy models on candidate mixtures and extrapolating to full training—is combinatorially expensive and scales poorly as the number of domains grows ( 53 ; 24 ; 5 ) . Furthermore, most data mixing approaches are specialized towards pretraining and assume a fixed domain set ( 4 ; 13 ; 21 ; 50 ; 6 ) : in practice, available training domains evolve continuously as new tasks are defined, new corpora are collected, and new capabilities are prioritized. This induces a natural continual learning problem, where the goal is to incorporate new data without catastrophically forgetting what the model has already learned. We ask: What is the right data mix, and how do we find it efficiently as the data itself keeps changing? We propose OP-Mix ( O n- P olicy Mix), an algorithm that estimates optimal data mixtures by combining two insights. First, rather than train separate proxy models for each candidate data mixture, OP-Mix trains a single low-rank adapter (LoRA, 20 ) per data domain directly from the current model, keeping the proxy model on-policy with the model being trained— i.e. , reflective of its current state. Second, it uses linear interpolation between LoRAs as a proxy for the loss surface of full data mixing, following recent works ( 44 ; 42 ) . This bypasses the need to retrain proxies for every different data mix ratio, escaping the combinatorial explosion of training runs. These two insights allow OP-Mix to search over data mixtures with minimal additional compute, no separate proxy models, and natural accommodation to new domains: when a new dataset arrives, we simply train another LoRA and re-fit the mixture. We evaluate OP-Mix across three stages of the language model lifecycle—pretraining ( 31 ; 9 ) , continual midtraining ( 29 ; 23 ) , and continual instruction tuning ( 47 ) —and find that our single algorithm suffices for all three. In pretraining, OP-Mix improves over no data mixing by 6.3% in average perplexity and matches the best data mixing baseline’s performance while using 14% less compute. In continual midtraining, OP-Mix achieves the performance of full retraining at a fraction of the cost. Finally, in continual instruction tuning, OP-Mix composes with on-policy self-distillation ( 38 ; 25 ; 55 ) , yielding further gains without modifications to either algorithm. Our contributions are as follows: 1. The first universal data mixing algorithm: OP-Mix is the first data mixing algorithm that both expands to new data domains and simulates candidate mixtures without separate proxy models. This enables OP-Mix to continually mix data even as the data evolves, overcoming the need for a different algorithm at each phase of the training pipeline (§ 3 ). 2. State-of-the-art across the entire training lifecycle: A single instantiation of OP-Mix achieves state-of-the-art performance in pretraining, continual midtraining, and continual instruction tuning, demonstrating that phase-specific algorithms are unnecessary. (§ 4 ). 3. OP-Mix enables continual learning, matching on-policy distillation with 95% less compute: Applied atop standard SFT during continual instruction tuning, OP-Mix recovers the gains of self-distillation finetuning (SDFT, 38 ) at a fraction of the cost. Combining OP-Mix with SDFT also yields further gains, suggesting that data mixing can be an independent axis of improvement from training objective (§ 4.1 ). 2 Background: Data Mixing and Its Limitations Let \mathcal{D}=\{D_{1},D_{2},\dots,D_{m}\} be a set of m data domains, where domain D_{i} has N_{i} tokens. A data mixture is a probability vector p\in\triangle^{m-1} , where training on R total tokens uses p_{i}\cdot R tokens from domain D_{i} . We denote a language model of S parameters trained for R tokens on mixture p as \text{LM}(S,R,p) and measure its performance on downstream task j\in[J] as f_{j}(\text{LM}(S,R,p)) . We assume the training objective is to minimize a weighted sum F=\sum_{j}w_{j}\cdot f_{j}(\text{LM}(S,R,p)) , where weights w_{j} are user-specified. Here, metrics intended to be maximized ( e.g. , accuracy) are negated. Batch continual learning. During training, we may periodically receive k new datasets D_{m+1},\dots,D_{m+k} , in which case the updated domain set becomes \mathcal{D}\cup\{D_{m+1},\dots,D_{m+k}\} . For example, these k new datasets may be instruction fine-tuning datasets, introduced after pretraining. We may then aim to minimize the loss across both pretraining and instruction tuning datasets. Data mixing. Data mixing algorithms automate the process of finding the mixture p that minimizes F . The core idea in most data mixing algorithms is fitting a simple model \hat{f_{i}}(p) that predicts the future performance f_{i} as a function of the performance on the current data mixture p (see 4 for review). One can then minimize \hat{f_{i}}(p) to estimate an optimal mixture. Algorithm Adapts to new data? No proxy models? ADO ( 21 ) ✗ ✓ Aioli ( 4 ) ✗ ✗ DoGE ( 13 ) ✗ ✓ GRAPE ( 12 ) ✗ ✓ OLMix ( 5 ) ✓ ✗ RegMix ( 24 ) ✗ ✗ MergeMix ( 44 ) ✗ ✓ OP-Mix (ours) ✓ ✓ Table 1 : OP-Mix is the only method that expands the data mixture