Chameleon : A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning

Wanyun Xie, Francesco Tonin, Volkan Cevher · 2025

Read Chameleon : A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning by Wanyun Xie, Francesco Tonin, Volkan Cevher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24844.txt.

Introduction

Wanyun Xie, Francesco Tonin, and Volkan Cevher present Chameleon, a flexible framework for mixing training data for language models, published in 2025. The blend of data domains strongly shapes how well a model generalizes. Older reweighting methods cost much compute and demand retraining whenever new data arrives.

Chameleon scores each domain's importance with leverage scores inside a learned embedding space. A domain affinity matrix captures which domains share representations, and the scores set a mixture that favors those shared domains. New data slots in by computing its embedding, with no proxy retraining. Experiments showed gains in three settings: cheaper pretraining mixtures, quick adaptation to new data with better few-shot reasoning, and finetuning mixtures that beat uniform mixing on every domain tested.

Catalog record

CREATORWanyun Xie, Francesco Tonin, Volkan Cevher
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chameleon : A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning Wanyun Xie, Francesco Tonin, Volkan Cevher Abstract Training data mixtures greatly impact the generalization performance of large language models. Existing domain reweighting methods often rely on costly weight computations and require retraining when new data is introduced. To this end, we introduce a flexible and efficient data mixing framework, Chameleon , that employs leverage scores to quantify domain importance within a learned embedding space. We first construct a domain affinity matrix over domain embeddings. The induced leverage scores determine a mixture that upweights domains sharing common representations in embedding space. This formulation allows direct transfer to new data by computing the new domain embeddings. In experiments, we demonstrate improvements over three key scenarios: (i) our computed weights improve performance on pretraining domains with a fraction of the compute of existing methods; (ii) Chameleon can adapt to data changes without proxy retraining, boosting few-shot reasoning accuracies when transferred to new data; (iii) our method enables efficient domain reweighting in finetuning, consistently improving test perplexity on all finetuning domains over uniform mixture. Our code is available at https://github.com/LIONS-EPFL/Chameleon . 1 Introduction Pretraining large language models (LLMs) relies heavily on vast and diverse datasets, encompassing sources such as academic papers, books, and code repositories ( Brown et al., 2020 ; Dubey et al., 2024 ) . The composition of these datasets significantly influences the generalization capabilities and downstream performance of LLMs. Domain reweighting, which involves adjusting the relative contributions of different domains in the training dataset, has emerged as a critical aspect of LLM training ( Gao et al., 2020 ; Du et al., 2022 ) . However, obtaining optimal domain weights is a challenging problem due to factors such as data quality, diversity, inter-domain overlap, and task-specific complexities ( Shen et al., 2023 ; Longpre et al., 2024 ) . Early domain reweighting methods relied on manual selection, often favoring high-quality sources like Wikipedia and academic texts ( Brown et al., 2020 ; Gao et al., 2020 ) . While intuitive, these approaches are neither optimal nor scalable. Recent work explores computational strategies for optimizing domain mixtures, such as DoReMi ( Xie et al., 2023 ) and DoGE ( Fan et al., 2024b ) , which use a small proxy model to derive domain weights for training a larger base model. Though effective, these methods are computationally expensive and have limited practical applicability. We contend that an ideal data-mixing method should ( i ) improve universal generalization, the fundamental goal of domain reweighting; ( ii ) adapt to domain modifications – data naturally evolves between preparation and LLM training, making frequent recalibration impractical; ( iii ) handle different training stages such as pertaining and fine-tuning. Most existing methods are limited to pretraining scenarios and do not consider either domain changes or the fine-tuning stage where domain specificity often plays a larger role. In this work, we introduce Chameleon , a novel and efficient framework for data mixing that addresses these challenges. Our method computes domain weights directly from learned embeddings using kernel ridge leverage scores (KRLS), which quantify the importance of each domain based on its contribution to the overall embedding space. Unlike existing approaches, Chameleon reduces the data-mixing compute and eliminates the need for frequent retraining of proxy models when domains change. Instead, it constructs a domain affinity matrix to capture relationships between domains and computes leverage scores that guide domain reweighting. Our data-centric approach enables seamless adaptation to new data and flexibility across both pretraining and fine-tuning stages. Specifically, our contributions are summarized as follows: • We propose Chameleon , an efficient data-mixing framework that leverages KRLS to quantify domain representativeness from embedded data. Inverse KRLS-based domain weights emphasize general knowledge for pertaining. We empirically demonstrate that Chameleon matches DoReMi and DoGE in pretraining performance at a fraction of their cost. • As a data-centric method, Chameleon can flexibly adapt to changes in domain composition without retraining proxy models, enhancing its practicality in real scenarios. It outperforms baselines at 1% of the retraining cost, even as domains double. • We extend domain reweighting to fine-tuning, where KRLS-based weights emphasize domain-specific uniqueness. Empirical results show consistent perplexity improvements across all domains on both natural language and code datasets. From a practical standpoint, Chameleon significantly lowers the computational burden associated with domain reweighting, making advanced LLM training pipelines more accessible to researchers with limited resources. Indeed, our KRLS-based scores are computationally inexpensive, hyperparameter-robust, and converge quickly. This efficiency is particularly advantageous when incorporating new data, where our method can be applied directly without re-running the entire proxy pipeline. By bridging the gap between pretraining and fine-tuning, our method provides a unified and agile framework for efficient as well as effective data mixing across all stages of LM training. 2 Related Work Domain Reweighting. Domain reweighting improves LLM pretraining by balancing data contributions from different sources. In this setting, online adaptation strategies require frequent recalibration and monitoring ( Albalak et al., 2023 ; Jiang et al., 2024 ; Fan et al., 2024a ) . Two closely related approaches are DoReMi ( Xie et al., 2023 ) and DoGE ( Fan et al., 2024b ) . DoReMi trains both a reference and a proxy model using Group DRO ( Sagawa et al., 2020 ) to mitigate excess domain loss, while DoGE tracks domain-specific gradients during proxy training. Both methods are inefficient: DoReMi depends on the reference model’s quality and requires training two models, while DoGE incurs high gradient tracking costs. Their domain weights fluctuate significantly during training ( Figure 3 ). In this setting, our work develops an offline method that achieves uniformly strong performance across domains without relying on downstream task knowledge. Other offline methods, such as Data Mixing Laws ( Ye et al., 2024 ) and RegMix that–in contrast to ours–requires access to the downstream tasks ( Liu et al., 2024 ) , use multiple proxy models to search for optimal data mixtures, revealing that domain weights transfer across model sizes. Additionally, studies on data and model scaling laws provide further insights into domain weighting strategies ( Kang et al., 2024 ; Ye et al., 2024 ) . However, they critically rely on the scaling strategy and do not have an easy way of adapting to domain expansion. Table 1 : Data-mixing methods capabilities comparison. DoReMi DoGE Chameleon Generalization ✓ ✓ ✓ Downstream ✓ ✓ ✓ New data ✗ ✗ ✓ Finetuning ✗ ✗ ✓ Kernel Ridge Leverage Scores (KRLS). The notion of statistical leverage score ( Gareth et al., 2013 ) is used in best-rank approximation to define an importance score for the rows in a matrix by their influence on the optimal low-rank approximation ( Mahoney & Drineas, 2009 ) , with approximation error guarantees for sampling ( Li et al., 2013 ) . Ridge leverage scores are also proposed with additional regularization term ( Cohen et al., 2015 ; Cohen et al., 2017 ; Rudi et al., 2018 ) . Leverage scores are extended to the kernel setting in ( Bach, 2013 ) , namely kernel ridge leverage scores. KRLS sampling is extensively used to accelerate kernel methods ( Alaoui & Mahoney, 2015 ; Musco & Musco,

Read the complete text