The Best Instruction-Tuning Data are Those That Fit
Dylan Zhang, Qirun Dai, Hao Peng · 2025
Read The Best Instruction-Tuning Data are Those That Fit by Dylan Zhang, Qirun Dai, Hao Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04194.txt.
Introduction
Dylan Zhang, Qirun Dai, and Hao Peng argue that the best data for instruction tuning a language model are the data that fit that model. Their 2025 paper notes that fine-tuning data are often sampled from other models, whose style sits outside the target model's own distribution. At scale that mismatch can blunt gains and even hurt performance.
Their method, GRAPE, gathers candidate answers from several models for each instruction and picks the one the target model rates most likely. Training then proceeds as usual. In tests the method beat strong baselines while using far less data, and it held up on real post-training sets for known model families. Less data and fewer training rounds still gave top results.
Catalog record
| CREATOR | Dylan Zhang, Qirun Dai, Hao Peng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04194
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang, Qirun Dai, Hao Peng Abstract High-quality supervised finetuning (SFT) data are essential for unlocking pretrained LLMs’ capabilities. Typically, instructions are paired with responses from various sources by humans annotators or other LMs, which are often out of the distribution of the target model to be finetuned. This, at scale, can lead to diminishing returns and even hurt the models’ performance and robustness. We hypothesize that SFT is most effective with data aligned to the model’s pretrained distribution and propose GRAPE— a novel SFT framework that tailors supervision to the target model. For each instruction, it g athers r esponses from various sources, and selects the one that a ligns most closely to the target model’s p r e trained distribution, as measured by the normalized probability. We then proceed with standard SFT with these selected responses. We first evaluate GRAPE with a controlled experiment, where we sample various solutions for each question in UltraInteract from multiple models and finetune on GRAPE-selected data using LMs from different families including LLaMA.1-8B, Mistral-7B and Qwen2.5-7B. GRAPE significantly outperforms strong baselines, including distilling from the strongest model with absolute gain up to 13.8% averaging across benchmarks, and a baseline trained on 3 × \times more data with maximum 17.3% performance improvements. GRAPE’s strong performance generalizes to off-the-shelf SFT data. We use GRAPE to subsample responses from the post-training data used for Tulu3 and Olmo-2. GRAPE can outperform strong baselines with 4.5 times the data by 6.1% and state-of-the-art data selection approaches by 3.9% on average performance. Remarkably, using 1/3 data and half number of epochs, GRAPE allows LLaMA.1-8B to surpass the performance of Tulu3-SFT by 3.5%. Our findings highlight that aligning supervision with the pretrained distribution offers a simple yet powerful way to improve SFT efficiency and performance. 1 Introduction High-quality, large-scale supervised data is crucial for supervised fine-tuing(SFT; Databricks, 2023 ; Köpf et al., 2023 ; Zhao et al., 2024a ; Zheng et al., 2024 ). A common practice of collecting SFT data involves sampling responses from strong language models, predominantly focusing on expanding the size of the dataset and improving the overall quality of the responses ( Sun et al., 2023 ; Taori et al., 2023 ; Wang et al., 2023 ; Xu et al., 2024b ; Chen et al., 2024 ) . However, recent research suggests that there is more complex dynamics involved ( Xu et al., 2024c ) . A plateau effect in synthetic data scaling, where performance either stagnates or even declines as the size of the synthetic data increases beyond a certain point, has been widely observed. This phenomenon arises due to issues such as diminishing diversity ( Padmakumar & He, 2024 ; Guo et al., 2023 ) and distortion in the data distribution ( LeBrun et al., 2021 ) , which ultimately undermine the base model’s performance and robustness ( Alemohammad et al., 2024 ; Gerstgrasser et al., 2024 ; Shumailov et al., 2023 ; Dohmatob et al., 2024 ; Hataya et al., 2023 ; Martínez et al., 2023a ; Martínez et al., 2023b ; Bohacek & Farid, 2023 ; Briesch et al., 2023 ) . Thus, effective SFT requires more than scaling up the data; it often needs “tailoring” the data to the unique characteristics of the target model. Existing works focus on enhancing the model’s existing knowledge and capabilities ( Du et al., 2023 ) and optimizing the curriculum progression for instruction tuning ( Zhao et al., 2024b ; Lee et al., 2024 ; Feng et al., 2023 ; Setlur et al., 2024 ) . They typically find questions the model should best learn, instead of what answers the model should imitate. Meanwhile, tailoring the responses to the target model has been a crucial ingredient for the success of later phases of LLM development, particularly through on-policy preference learning ( Tajwar et al., 2024 ; Zhang et al., 2024c ; Zhang et al., 2024a ; Miao et al., 2024 ; Gulcehre et al., 2023 ; Azar et al., 2023 ; Tang et al., 2024a ; Zhuang et al., 2023 ) , and on-policy/online reinforcement learning ( Guo et al., 2024b ; Liu et al., 2024d ; Zhou et al., 2024c ) . Inspired by these insights, we hypothesize that SFT can similarly benefit from aligning data with the model, the core idea behind GRAPE. For each instruction, GRAPE gathers and selects response(s) from various sources that are closest to the target model’s pretrained distribution. This is achieved by calculating the probability of each response using the target model and selects the one with the highest length-normalized probability(§ 3 ).After obtaining these more “in-distribution” responses, GRAPE proceeds with standard SFT without any modification to the training. Unlike existing datasets that usually contains one-size-fits-all responses for each instruction without customization ( Yu et al., 2024 ; Yuan et al., 2024b ; Lian et al., 2023b ; Teknium, 2023 ) , GRAPE curates model-dependent SFT datasets that better matches the base model’s distribution, better mitigating the risks associated with distribution shift like spurious correlations ( Zhou et al., 2024d ) and catastrophic forgetting ( Luo et al., 2025 ; Kotha et al., 2024 ) , while posing minimum overhead of single forward pass over the candidate set. In return, GRAPE allows better downstream performance with reduced training compute. To GRAPE’s advantage, many existing datasets share overlapping instructions but contain different high-quality responses, e.g., the instructions in Flan ( Longpre et al., 2023 ) , GSM-8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021b ) , and the post-training recipes that re-use SFT instructions for preference learning ( Lambert et al., 2024 ; OLMo et al., 2025 ) . Therefore, GRAPE can directly select, for each model, the best fit(s) among the off-the-shelf responses without having to produce new responses, which proves effective in our experiments (§ 5 ). We first validate our hypothesis through extensive controlled experiments on a reasoning dataset with chain-of-thoughts, UltraInteract-SFT ( Yuan et al., 2024b ) , and demonstrate the importance of supervising base models with in-distribution responses(§ 4 ). We experimented on 4 popular pretrained LMs from Mistral ( MistralAI, 2024c ) , Llama3.1 ( Dubey et al., 2024 ) and Qwen2.5 ( Hui et al., 2024 ) families. Notably, models fine-tuned with GRAPE-selected responses outperform those trained on a 3 \times larger datasets up to 17.3\% absolute gain on average performances even with less compute, even surpass models trained on responses from the strongest teacher under consideration— Llama3.1-405B-Instruct —by significant margins. We then experiment with a more realistic setting for general-domain instruction-tuning, collecting responses from post-training data for Tulu3 and Olmo-v2. Again, GRAPE demonstrates its effectiveness by outperforming state-of-the-art data selection approaches by avg. 4.6% and a strong baseline trained on all these available data that is 4.5 times larger by up to 6.1%. Remarkably, GRAPE allows finetuning a Llama3.1-8B base model to exceed the performance of Tulu-8B-SFT using 1/3 data and half of the epochs. Our results reveal that distributional alignment is a crucial and previously underappreciated dimension of effective instruction tuning. GRAPE introduces this perspective with a simple, scalable algorithm that consistently outperforms strong baselines