Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
Yibo Wang, Hai-Long Sun, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang · 2026
Read Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs by Yibo Wang, Hai-Long Sun, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08198.txt.
Introduction
Yibo Wang and six co-authors improve self-play fine-tuning for large language models. Their 2026 arXiv paper notes that self-play methods adapt models where expert labels are scarce. They do so by training on responses the model itself generates across iterations. A flaw is that reward advantages of labeled over synthetic responses can vanish during training. That vanishing leads to unstable optimization in the SPIN method. The authors argue triplets beat pairs for stable, effective training. Their change targets steadier gains when human annotation is limited.
Catalog record
| CREATOR | Yibo Wang, Hai-Long Sun, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08198
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs Yibo Wang, Hai-Long Sun, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang Abstract Recently, self-play fine-tuning ( 𝚂𝙿𝙸𝙽 \mathtt{SPIN} ) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, 𝚂𝙿𝙸𝙽 \mathtt{SPIN} is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to unstable optimization . Moreover, the utilization of reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel T riplet-based S elf- P lay f I ne-tu N ing ( 𝚃 - 𝚂𝙿𝙸𝙽 \mathtt{T}\mbox{-}\mathtt{SPIN} ) method that integrates two key designs. First, beyond current advantages, 𝚃 - 𝚂𝙿𝙸𝙽 \mathtt{T}\mbox{-}\mathtt{SPIN} additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, 𝚃 - 𝚂𝙿𝙸𝙽 \mathtt{T}\mbox{-}\mathtt{SPIN} introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of 𝚃 - 𝚂𝙿𝙸𝙽 \mathtt{T}\mbox{-}\mathtt{SPIN} over 𝚂𝙿𝙸𝙽 \mathtt{SPIN} , but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, 𝚃 - 𝚂𝙿𝙸𝙽 \mathtt{T}\mbox{-}\mathtt{SPIN} achieves comparable or even better performance with only 25 % 25\% samples, highlighting its effectiveness when faced with scarce annotated data. 1 Introduction Large language models (LLMs) have demonstrated powerful generalization capabilities and profound impact across various fields ( Ouyang et al. 2022 ; Huang and Chang 2023 ; Qin et al. 2023 ; Bai et al. 2023 ; Achiam et al. 2023 ; Liu et al. 2023 ; Jiang et al. 2024a ; Wu et al. 2024a ; Zhao et al. 2024 ; Chen et al. 2024a ; Liu et al. 2024 ; Hui et al. 2024 ; Guo et al. 2024a ; Jiang et al. 2024b ; Anthropic 2024a ; Google 2024 ; OpenAI 2024 ; Sun et al. 2025a ; Sun et al. 2025b ; Lu et al. 2025 ; Jiang et al. 2025 ; Qian et al. 2026 ) . The remarkable success of LLMs is inherently linked to the utilization of vast expert annotations. Nevertheless, considerable curation costs for high-quality data pose major obstacles to downstream-task adaptation ( Wang et al. 2024a ; Tan et al. 2024 ) . Consequently, numerous efforts have been made to handling this challenge, e.g., data augmentation ( Dai et al. 2023 ; Zhu et al. 2023 ; Guo et al. 2024b ) and model distillation ( Gunasekar et al. 2023 ; Luo et al. 2024 ; Xin et al. 2024 ) . Recently, a promising research direction known as self-play fine-tuning has been introduced ( Chen et al. 2024b ) , and has since been applied to various LLM fields ( Yuan et al. 2024a ; Wu et al. 2024b ; Gao et al. 2024a ) . The core idea of self-play fine-tuning is to progressively refine the LLM by competing against itself. Formally, it can be framed as a two-player game between a main player and an opponent player, each of which corresponds to an instance policy of the same LLM. In this game, the main player aims to distinguish expert-annotated responses from generated ones, while the opponent player strives to produce high-quality synthetic responses that can deceive the main player. Based on this framework, Chen et al. 2024b introduce the seminal method named \mathtt{SPIN} , which is designed to optimize the relative reward advantages of annotated responses over generated ones, and employs the policy from previous iteration as a reference for reward computations. However, with the evolution of LLM, the relative advantage between two types of data gradually diminishes over time. When the relative advantage is eliminated, the optimization objective of \mathtt{SPIN} degenerates into a constant independent of the policy, allowing any policy to become an optimal solution and thus leading to unstable optimization . Moreover, the incorporation of reference policy in \mathtt{SPIN} causes a misalignment between the reward formulation used for training and the log likelihood that guides response generation. Empirical evidences in Figure 3 reveal that although annotated responses yield higher rewards than generated responses, their log likelihoods become lower than synthetic samples during iterations. In other words, higher rewards do not necessarily lead to higher generation priority. Figure 1 : Comparisons of three strategies: (a) supervised fine-tuning requires large amounts of annotated data to train \pi_{\theta} ; (b) self-play fine-tuning operates with limited annotated data and iteratively generated samples, and employs the previous policy \pi_{\theta_{t}} as a reference for updates; (c) triplet-based self-play fine-tuning employs triplet inputs, i.e., annotated data, synthetic samples, and proto-synthetic ones from the initial policy \pi_{\theta_{0}} , and updates \pi_{\theta} without auxiliaries from any reference policies. To address these limitations, we propose a new self-play fine-tuning method, termed T riplet-based S elf- P lay f I ne-tu N ing ( \mathtt{T}\mbox{-}\mathtt{SPIN} ). The basic intuition is to incorporate historical advantages for stable evolution, and eliminate the reference policy for training-generation alignments. Specifically, we introduce two key modifications to the self-play framework. First, the main player is required not only to distinguish between real and synthetic data, but also discriminate proto-synthetic samples produced by the initial policy. By incorporating triplet inputs of real, synthetic and proto-synthetic samples, \mathtt{T}\mbox{-}\mathtt{SPIN} optimizes current and historical advantages simultaneously. Even if the current advantage diminishes (i.e., synthetic samples become indistinguishable from annotated ones), the historical advantages between synthetic responses and proto-synthetic responses still exist, ensuring a stable optimization over the policy model. Second, the opponent player maximizes the confidence of synthetic responses with an entropy constraint , where the confidence measures how likely the main player believes a response comes from the annotated data. Through theoretical analysis, we demonstrate that the incorporation of the entropy constraint removes the need for reference policy, aligning the principles of training and generation. Empirical results show that \mathtt{T}\mbox{-}\mathtt{SPIN} consistently maintains higher log-likelihoods for annotated data compared to synthetic samples. We conduct experiments on Zephyr-7B ( Tunstall et al. 2023 ) and Mistral-7B ( Jiang et al. 2023 ) , and select 50k annotated samples from Ultrachat200k ( Ding et al. 2023 ) as the training set. The performance is evaluated across diverse tasks to assess multi-dimensional capabilities of LLMs, including math and logic, multi-domain knowledge, commonsense reasoning, and instruction-following. Experimental results demonstrate that \mathtt{T}\mbox{-}\mathtt{SPIN} significantly improves performance, particularly in math and instruction-following, and achieves superior and stable performances during iterations compared to \mathtt{SPIN} . Furthermore, we also show that \mathtt{T}\mbox{-}\mathtt{SPIN} , using only 25\% annotated samples, can achieve comparable performance (or even better on certain tasks) to supervised fine-tuning with the full training set, highlighting the effectiveness of \mathtt{T}\mbox{-}\mathtt{SPIN} in addressing the scarcity of annotated data. 2 Preliminary In this section, we provide a brief overview of three fine-tuning strategies (i.e., supervised fine-tuning, self-play fine-tuning, and our \mathtt{T}\mbox{-}\mathtt{SPIN} ) for LLMs. The comparisons are also illustrated in Figure 1 . 2.1