R&B : Domain R egrouping and Data Mixture B alancing for Efficient Foundation Model Training
Albert Ge, Tzu-Heng Huang, John Cooper, Avi Trost, Ziyi Chu, Satya Sai Srinath Namburi GNVV, Ziyang Cai, Kendall Park, Nicholas Roberts, Frederic Sala · 2025
Read R&B : Domain R egrouping and Data Mixture B alancing for Efficient Foundation Model Training by Albert Ge, Tzu-Heng Huang, John Cooper, Avi Trost, Ziyi Chu, Satya Sai Srinath Namburi GNVV, Ziyang Cai, Kendall Park, Nicholas Roberts, Frederic Sala (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00358.txt.
Introduction
Albert Ge and co-authors present in 2025 a method called R and B for mixing training data for large models at lower cost. Older mixing plans group data by source or task, which can miss fine shades of meaning. They also grow costly as the number of groups rises.
R and B first regroups data by how close items are in meaning, making finer groups. It then balances the mix with a matrix built from group slopes seen during training, with no extra checks needed. Tests on five data sets show it matches or beats top mixing plans with very little added compute time.
Catalog record
| CREATOR | Albert Ge, Tzu-Heng Huang, John Cooper, Avi Trost, Ziyi Chu, Satya Sai Srinath Namburi GNVV, Ziyang Cai, Kendall Park, Nicholas Roberts, Frederic Sala |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
R&B : Domain R egrouping and Data Mixture B alancing for Efficient Foundation Model Training Albert Ge, Tzu-Heng Huang, John Cooper, Avi Trost, Ziyi Chu, Satya Sai Srinath Namburi GNVV, Ziyang Cai, Kendall Park, Nicholas Roberts, Frederic Sala Abstract Data mixing strategies have successfully reduced the costs involved in training language models. While promising, such methods suffer from two flaws. First, they rely on predetermined data domains (e.g., data sources, task types), which may fail to capture critical semantic nuances, leaving performance on the table. Second, these methods scale with the number of domains in a computationally prohibitive way. We address these challenges via R&B , a framework that re-partitions training data based on semantic similarity ( R egroup ) to create finer-grained domains, and efficiently optimizes the data composition ( B alance ) by leveraging a Gram matrix induced by domain gradients obtained throughout training. Unlike prior works, it removes the need for additional compute to obtain evaluation information such as losses or gradients. We analyze this technique under standard regularity conditions and provide theoretical insights that justify R&B ’s effectiveness compared to non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness of R&B on five diverse datasets ranging from natural language to reasoning and multimodal tasks. With as little as 0.01% additional compute overhead, R&B matches or exceeds the performance of state-of-the-art data mixing strategies. 1 Introduction Large language models depend on vast, diverse datasets, but the shift to general-purpose foundation models has created a fundamental imbalance: potential training data vastly exceeds available computational resources. This has driven the development of data-efficient strategies that maximize performance while minimizing compute costs. Among these approaches, data mixing is particularly promising. By optimizing the composition of training data used—rather than simply increasing its volume—we can achieve comparable or superior performance with significantly fewer computational resources. A wide variety of data mixture optimization techniques have been proposed [ 1 ; 2 ; 3 ; 4 ; 5 ] . These adjust the relative proportions (“mixture”) of training data from different predefined domains, also known as skills . Skills are often assigned to data based on human judgments or on source metadata [ 6 ] . For example, in an instruction-tuning dataset, skills might include domain categories like open-question answering or summarization. For other datasets, skills may be defined based on where the data was scraped from (Wikipedia, StackOverflow, etc.). We find, however, that these coarse human-defined categorizations fail to capture the optimal groupings for data mixing. That is, human categorizations of skills are suboptimal when used for the development of LLM capabilities . Consider the Dolly-15k instruction-tuning dataset, which categorizes its data into general domains such as open-question answering, information extraction, and summarization [ 7 ] . Rather than directly optimizing these coarse categories, our approach first re-partitions the data into finer-grained, semantically-grouped skills (Fig. 1 , left). Optimizing the proportions across these semantically-clustered skills can significantly improve training performance over that of the general predefined domains. These improvements are even more pronounced when the number of skill partitions is optimized (Fig. 1 , right). However, the more granular semantic-based clustering approach has a critical drawback . As the number of skills increases, prior data mixing methods become computationally prohibitive. Existing approaches typically require additional evaluations—either through forward passes over evaluation datasets for each skill or by computing per-skill gradient information derived from target tasks. To overcome this limitation, we propose an efficient gradient-based approach that leverages information already computed during training, bypassing the need for these expensive evaluations. These insights motivate our approach, R egroup & B alance ( R&B ), a two-stage framework for efficient data mixture optimization. First, we repartition ( R egroup) training data into semantically coherent clusters based on embedding similarity. Then, we dynamically optimize domain weights ( B alance) to capture individual domain contributions and cross-domain relationships, leveraging domain gradients computed throughout normal training. This produces the best of all worlds: it unlocks the performance gains in fine-grained skill clusters while dramatically reducing computational complexity. Figure 1 : Instead of using pre-determined domains (e.g., by task type), we find that it is often better to first repartition the data into finer-grained, semantically related domains. Optimizing the proportions of these new semantic domains can significantly improve training performance. Theoretically, we derive insights to characterize the importance of semantic-based clustering in data mixing strategies and the optimality of R&B . Empirically, we validate R&B across five diverse data settings, encompassing natural-language, instruction-following, reasoning, and multimodal tasks. R&B only requires an additional 0.01% compute overhead, which cuts computational FLOPs by more than 99% relative to existing approaches, all while maintaining or improving model performance . We summarize our contributions as follows: 1. We establish that semantic-based categorizations of skills are superior to human-defined categories for foundation model data mixing algorithms (Section 3.1 ). 2. We introduce R&B , an efficient and theoretically justified two-stage framework that first repartitions data into semantically coherent clusters of skills, then dynamically reweights skill mixtures using their gradients (Sections 3 , 3.1 , and 3.3 ). 3. We theoretically and empirically demonstrate that R&B scales effectively with increased skill counts (Section 4 ). 2 Related Work Our work builds upon prior works that study how to effectively curate and compose training data. We summarize three directions: data mixing, data selection, and scaling laws for data mixing. Group-level Data Mixing. Prior work falls into two categories: static methods, which learn domain proportions before training, and dynamic methods, which adjust them as training progresses. Among the former, Fan et al. 2024 uses a smaller model as a proxy to find mixture weights based on each domain’s learning contribution, then applies them for the full model. Xie et al. 2023 optimizes worst-case loss with a reference and proxy model, treating the resulting weights as domain proportions. These methods, though efficient, often neglect interactions between domains. Among the latter methods, Chen et al. 2023 model cross-domain interactions with a pre-built skills graph to optimize data composition. Building on it, Chen et al. 2024 introduces an online method that estimates domain interactions using training history information. However, as the domain count rises, these methods become computationally expensive. Sample-level Data Selection. These techniques represent a more granular approach to improve dataset quality by evaluating individual samples. Several methods have been proposed, including using gradient similarity [ 8 ; 9 ; 10 ; 11 ] , reward functions [ 12 ] , object detection models [ 13 ] , and embedding similarity metrics [ 14 ] . A related line of work is through deduplication, which removes redundant or nearly identical examples using clustering [ 15 ; 16 ; 17 ] . Our work integrates these perspectives, repartitioning data into finer-grained groups while simultaneously optimizing domain mixtures. Scaling Laws. These works model and predict performance when training on diverse domains. Ge et al. 2024 develop a bivariate scaling law to jointly model domain proportion and data volume, while Ye et al. 2024 propose a composite exponential law that accounts for cross-domain interactions. Liu et al. 2024 approach the problem empirically by training small models with varying mixtures to fit a predictive regression model, Kang et al. 2024 build on this to derive