ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning

Yang Wu, Huayi Zhang , Yizheng Jiao, Lin Ma, Xiaozhong Liu, Jinhong Yu, Dongyu Zhang, Dezhi Yu, Wei Xu · 2024

Read ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning by Yang Wu, Huayi Zhang , Yizheng Jiao, Lin Ma, Xiaozhong Liu, Jinhong Yu, Dongyu Zhang, Dezhi Yu, Wei Xu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00631.txt.

Introduction

Yang Wu and co-authors present a way to pick training data for teaching language models a specific job, in this 2024 paper. Instruction tuning works only as well as the examples chosen for it. Most pickers grab examples that look similar to the target task's data. The catch is that the usual training loss often fails to show which examples truly help on the job that counts.

Their framework, ROSE, chooses data by reward instead: how much each candidate example should raise the model's score on the target task. That flips the search from resemblance to usefulness. Tested across tasks, reward-driven picks aim to beat the similarity habit that dominates the field. Better data choice can lift a small tuning budget further than more data dumped in blindly.

Catalog record

CREATORYang Wu, Huayi Zhang , Yizheng Jiao, Lin Ma, Xiaozhong Liu, Jinhong Yu, Dongyu Zhang, Dezhi Yu, Wei Xu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00631

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning Yang Wu, Huayi Zhang , Yizheng Jiao, Lin Ma, Xiaozhong Liu, Jinhong Yu, Dongyu Zhang, Dezhi Yu, Wei Xu Abstract Instruction tuning has underscored the significant potential of large language models (LLMs) in producing more human-controllable and effective outputs in various domains. In this work, we focus on the data selection problem for task-specific instruction tuning of LLMs. Prevailing methods primarily rely on the crafted similarity metrics to select training data that aligns with the test data distribution. The goal is to minimize instruction tuning loss on the test data, ultimately improving performance on the target task. However, it has been widely observed that instruction tuning loss (i.e., cross-entropy loss for next token prediction) in LLMs often fails to exhibit a monotonic relationship with actual task performance. This misalignment undermines the effectiveness of current data selection methods for task-specific instruction tuning. To address this issue, we introduce ROSE, a novel R eward- O riented in S truction data s E lection method which leverages pairwise preference loss as a reward signal to optimize data selection for task-specific instruction tuning. Specifically, ROSE adapts an influence formulation to approximate the influence of training data points relative to a few-shot preference validation set to select the most task-related training data points. Experimental results show that by selecting just 5% of the training data using ROSE, our approach can achieve competitive results compared to fine-tuning with the full training dataset, and it surpasses other state-of-the-art data selection methods for task-specific instruction tuning. Our qualitative analysis further confirms the robust generalizability of our method across multiple benchmark datasets and diverse model architectures. 1 Introduction While large language models (LLMs) are widely recognized for their strong generalization capabilities, many fields require enhanced domain-specific performance, e.g., health monitoring ( Kim et al. 2024b ) , legal question answering ( Wu et al. 2024 ) , and mathematics tutoring ( Li et al. 2023 ) . Instruction tuning has emerged as a popular method for adapting foundation models to specialized tasks, which typically involves curating a high-quality training dataset. Although recent advancements in open-source datasets and synthetic data generation have facilitated the generation of large training datasets, it is widely acknowledged that the quality of training data is more crucial than its quantity in instruction tuning ( Chen & Mueller 2024 ; Xia et al. 2024 ) . Consequently, practitioners must carefully select high-quality data to enhance the model’s capabilities for specific tasks. This challenge is further compounded by the complexity of domain-specific requirements and the black-box properties of LLMs, rendering it nearly infeasible for humans to manually select the most suitable training set. Therefore, developing more effective data selection methods is becoming increasingly crucial for reducing training costs and efficiently optimizing instruction tuning for specific tasks. Despite various methods proposed for instruction tuning data selection ( Du et al. 2023 ; Mekala et al. 2024 ) , identifying high-quality instruction tuning data for target tasks remains a significant challenge. Existing methods typically design specific similarity metrics to select a set of candidates samples whose distribution aligns with that of the target task data. For instance, DSIR ( Xie et al. 2023 ) uses n-gram feature similarity to enhance the selection process of relevant data samples. Moreover, RDS ( Zhang et al. 2018 ) and LESS ( Xia et al. 2024 ) calculate the similarity on the model embedding and gradient space to capture task-specific semantics and model characteristics, respectively. Despite their successes, these similarity based methods are fundamentally limited. Figure 1: Illustration of our Reward-Oriented Instruction Data Selection (ROSE) approach compared with the traditional method. Unlike the traditional focus on minimizing validation loss, ROSE maximizes task-specific reward to more precisely align with real-world task performance. Our analysis reveals that these strategies hinge on Empirical Risk Minimization (ERM), selecting training data that mirrors the target task data distribution by minimizing training loss, particularly next-token prediction loss. However, it is widely acknowledged that next-token prediction loss often fails to accurately reflect a model’s real-world performance (e.g., alignment degree with human preference, reasoning ability on complex math problem) on target task ( Zhou et al. 2024 ; Tay et al. 2021 ) . This significant gap between the implicit theoretical foundation of similarity-based methods and practical task performance limits the effectiveness of these methods in instruction tuning for task-specific fine-tuning. In response to these insights, we introduce a novel reward-oriented instruction data selection (ROSE) method in Figure 1 , which shifts the intrinsic selection objective from minimizing validation cross entropy loss to maximize the reward for the target task. Inspired by Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) , our approach utilizes a few-shot set of pairwise samples as the task-specific preference validation set, which we assume reflects the desired LLM’s performance on the target task, and we use the DPO loss function to approximate the expected reward value of the trained LLM on the preference validation data. By leveraging the gradient-based influence estimation techniques, ROSE is able to select the instruction tuning samples that leads a downstream LLM with optimized performance on the target task. We conduct comprehensive experiments across various datasets and model architectures. Experimental results show that our method outperforms existing similarity-based techniques, including token-wise, embedding-based, and gradient-based methods. These results suggest that focusing on reward maximization, rather than loss minimization, is a promising new direction for improving task-specific fine-tuning outcomes. The main contributions are as follows: • Identifying Limitations of Similarity-Based Approaches: Our analysis reveals the limitations of the implicit theoretical foundations of the common similarity-based data selection methods. These methods focus on minimizing training loss (e.g., next-token prediction loss), which often fails to capture real-world task performance. • Introducing Reward-Oriented Data Selection: We shift the data selection objective from loss minimization to reward maximization on the target task dataset, which contains a small set of user-provided positive and negative samples that reflect the task-specific performance. • Propose to Approximate Reward with DPO Loss: Our method incorporates gradient-based influence estimation techniques to select high-quality training data that optimizes task reward, improving the fine-tuning process. • Comprehensive Experimental Validation: Through experiments on various datasets and models, we demonstrate that ROSE consistently outperforms existing similarity-based methods, such as token-wise, embedding-based, and gradient-based approaches, in task-specific fine-tuning. 2 Preliminaries and Background Problem Definition. We tackle the challenge of data selection for task-specific instruction tuning. Our goal is to curate a subset \mathcal{D}_{\text{train}} from a broad and comprehensive instruction tuning corpus \mathcal{D} , such that training a model on \mathcal{D}_{\text{train}} to maximize a reward r that reflects true performance on a task-specific validation set \mathcal{D}_{\text{val}} , therefore performs well on test dataset \mathcal{D}_{\text{test}} . \mathcal{D}_{\text{val}} can be a few-shot dataset involving multiple target tasks, and the \mathcal{D}_{\text{test}} is a fixed sample set with the same tasks in \mathcal{D}_{\text{val}} . We use \Omega parametrized by \theta to denote the model used for data selection and \Gamma parametrized by \theta^{\prime} to represent the final trained model. Analysis of Similarity Based Methods. Let \mathbb{Z}=\mathbb{R}^{d} be the d-dimensional intrinsic representation space, where the similarity-based methods calculate the similarity between the training and validation samples. Let p_{t}(z) and p_{v}(z) be the

Read the complete text