RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning

Zehua Liu, Shuqi Liu, Tao Zhong, Mingxuan YuanHuawei Noah’s Ark Lab [email protected], [email protected] · 2026

Read RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning by Zehua Liu, Shuqi Liu, Tao Zhong, Mingxuan YuanHuawei Noah’s Ark Lab [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09253.txt.

Introduction

Zehua Liu and three co-authors use bad samples, not just good ones, to tune chat models. Plain tune uses dear expert text. A next path drafts many replies, keeps the best, and tunes on them. That path throws the rest away. Yet a bad reply holds sense: it shows what to shun, and how close a near miss can be.

Their method is called RIFT. A reward score grades all drafts, good and bad. Tune then uses each draft with a weight from its grade. No draft is waste. A reader who tunes models gets a data lean path. It may cut the need for expert text while it lifts the last score.

Catalog record

CREATORZehua Liu, Shuqi Liu, Tao Zhong, Mingxuan YuanHuawei Noah’s Ark Lab [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-09253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning Zehua Liu, Shuqi Liu, Tao Zhong, Mingxuan YuanHuawei Noah’s Ark Lab [email protected], [email protected] Abstract While Supervised Fine-Tuning (SFT) and Rejection Sampling Fine-Tuning (RFT) are standard for LLM alignment, they either rely on costly expert data or discard valuable negative samples, leading to data inefficiency. To address this, we propose Reward Informed Fine-Tuning (RIFT), a simple yet effective framework that utilizes all self-generated samples. Unlike the hard thresholding of RFT, RIFT repurposes negative trajectories, reweighting the loss with scalar rewards to learn from both the positive and negative trajectories from the model outputs. To overcome the training collapse caused by naive reward integration, where direct multiplication yields an unbounded loss, we introduce a stabilized loss formulation that ensures numerical robustness and optimization efficiency. Extensive experiments on mathematical benchmarks across various base models show that RIFT consistently outperforms RFT. Our results demonstrate that RIFT is a robust and data-efficient alternative for alignment using mixed-quality, self-generated data. † † footnotetext: † Corresponding author. 1 Introduction The rapid scaling of Large Language Models (LLMs) has made effective post-training adaptation essential 42 ; 5 ; 2 . Supervised Fine-Tuning (SFT) 26 ; 30 , which minimizes the negative log-likelihood of expert demonstrations, constitutes the standard approach for aligning models with desired behaviors. However, the efficacy of SFT is heavily dependent upon the availability of high-quality demonstration data, which are generally difficult and costly to curate. More critically, a distributional mismatch between the pre-training data or initial model capabilities and the SFT data can lead to degraded performance, a phenomenon often described as catastrophic forgetting or alignment tax 16 ; 21 ; 14 ; 9 . Figure 1: Efficiency and performance of post-training methods for Qwen2.5-Math-1.5B fine-tuned on MATH. Left : average accuracy against peak memory utilization (training efficiency); Right : per-dataset accuracy (generalization). RIFT surpasses strong baselines in accuracy while requiring less computational memory. To mitigate these data-related limitations in SFT, Rejection Sampling Fine-Tuning (RFT) 40 ; 1 has emerged as a lightweight yet effective alternative. The underlying principle of RFT is straightforward: by sampling multiple responses from a base model to a given prompt and subsequently selecting only those that surpass a predefined quality threshold, one can construct a refined, higher-quality dataset for a subsequent round of SFT. Unlike conventional SFT that relies on pre-constructed static datasets, RFT generates its training data through the model’s own sampling process. This self-generation strategy inherently promotes alignment between the data distribution used for fine-tuning and the model’s own output distribution. Furthermore, the quality and correctness of the selected data is ensured through an external verification mechanism or a well-defined scoring function. Despite its simplicity and advantages, the standard RFT paradigm exhibits a critical shortcoming: it discards all sub-threshold (negative) samples outright. This discard policy neglects the potential informational value these samples carry regarding model failure modes. Consequently, it not only wastes computational resources expended during generation, but may also impair the model to learn distinctions between correct and incorrect outputs, thereby limiting its capacity to refine its understanding of subtle errors. To better utilize all generated responses, including those rejected by quality thresholds in RFT, we propose Reward Informed Fine-Tuning (RIFT) . RIFT constitutes a simple and efficient extension of standard SFT. In contrast to RFT, which discards low-scoring candidates via hard thresholding, RIFT retains every sampled trajectory and assigns it a scalar reward derived from a quality evaluation metric. The RIFT objective modifies the standard negative log-likelihood loss by reweighting each sample’s contribution proportionally to its assigned reward. This design ensures that positive-reward samples encourage correct behaviors, whereas negative-reward samples provide reduced or negative gradients. Nevertheless, a naive integration of the reward signal with the logarithmic probability term presents a significant practical challenge. As we will demonstrate in Section 3 , directly multiplying these components produces a loss function that is unbounded from below, inevitably leading to severe training collapse. To address this fundamental issue, we introduce a principled framework for loss function formulation, which is designed to ensure guaranteed training stability and maintain optimization efficiency. Figure 1 compares RIFT with strong baselines (e.g., DPO 29 , DFT 34 ) on Qwen2.5-Math-1.5B: RIFT achieves comparable or superior accuracy at substantially lower peak memory utilization. Empirically, RIFT delivers consistent and substantial improvements across model scales and alignment settings on mathematical reasoning benchmarks. RIFT outperforms SFT, DFT 34 ), RFT 40 and DPO 29 in both in-distribution accuracy and out-of-distribution generalization on Qwen2.5-Math (1.5B/7B) 38 , Qwen3-1.7B 36 , and DeepSeek-R1-Distill-Qwen-1.5B 7 . Unlike RFT 40 , which critically depends on strong base models to generate high-quality rollouts, RIFT remains stable and effective even with moderately capable models. In off-policy settings, RIFT consistently surpasses DPO 29 across models. Notably, RIFT eliminates the need for a reference model, offering a simpler and more resource-efficient alternative for alignment. 2 Related Works LLM Post-training Supervised Fine-Tuning (SFT) has become the standard post-training paradigm for adapting pretrained models to specific tasks using high-quality, labeled datasets ( 41 ; 4 ) . Although the availability of high-quality instruction-following datasets ( 6 ; 25 ; 45 ; 32 ) has significantly enhanced the efficacy of SFT, studies ( 8 ; 13 ; 27 ) indicate that SFT often suffers from overfitting and suboptimal generalization. To mitigate the challenges of SFT data curation, Reinforcement Learning (RL) has emerged as a powerful alternative for post-training. Reinforcement Learning from Human Feedback (RLHF) ( 27 ) and Reinforcement Learning from Verifiable Reward (RLVR) ( 10 ; 31 ) are widely used to align models with human preferences and enhance reasoning, supported by algorithms like DPO ( 29 ) , SimPO ( 24 ) , GRPO ( 31 ) , and DAPO 39 . However, in contrast to SFT, RL-based training is often more complex, necessitating intricate engineering frameworks ( 44 ) . Improving SFT Motivated by the success of RL methods, a growing body of research aims to enhance SFT by integrating RL principles. RFT ( 40 ) utilizes self-generated trajectories filtered for correctness as training data. Other approaches re-frame RL objectives within an SFT framework, such as integrating importance sampling ( 28 ) or adopting PPO-style clipped surrogates ( 46 ) . However, these methods typically require a reference model, imposing a procedural complexity that aligns them more closely with RL paradigms than the simplicity of conventional SFT. In a parallel line of inquiry, other research focuses on directly modifying the SFT loss without a reference model. For example, DFT ( 34 ) rescales the SFT objective at each token by its probability. Building on this, 18 proposed a unified framework for designing loss objectives, demonstrating that DFT can be considered a special case within their formulation. Following this direction, our proposed method, RIFT, refines the loss function to enhance performance while preserving the simplicity of standard SFT. Figure 2: A comparative overview of RFT and RIFT. Unlike RFT rejects negative samples and only trains on positive ones, RIFT repurposes negative samples through a unified reward-informed loss. To ensure stable optimization, a linear surrogate is applied to negative samples to prevent loss collapse. 3 Methodology In this section, we present the theoretical framework and methodology of RIFT (Reward-Informed Fine-Tuning), a

Read the complete text