UFT: Unifying Supervised and Reinforcement Fine-Tuning

Mingyang Liu, Gabriele Farina & Asuman Ozdaglar · 2025

Read UFT: Unifying Supervised and Reinforcement Fine-Tuning by Mingyang Liu, Gabriele Farina & Asuman Ozdaglar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16984.txt.

Introduction

Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar merge two ways of training language models into one. Their paper, from twenty twenty-five, is called UFT. Supervised tuning teaches from worked examples and is cheap, but it can overfit. Reward-based tuning generalizes better, yet leans on a strong base model.

Their unified method lets the model search for answers while still learning from good examples. That blend bridges rote learning and real reasoning. Tests across model sizes show it beats each parent method alone. Theory in the paper proves a sharp speedup on long reasoning tasks. The old reward method's sample cost grew in an explosive way that the new one breaks.

Catalog record

CREATORMingyang Liu, Gabriele Farina & Asuman Ozdaglar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16984

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UFT: Unifying Supervised and Reinforcement Fine-Tuning Mingyang Liu, Gabriele Farina & Asuman Ozdaglar Abstract Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT’s inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks. “学而不思则罔,思而不学则殆。” "Learning without thinking leads to confusion; thinking without learning is perilous." —— 孔子 (Confucius) 1 1 footnotetext: The source code is available at https://github.com/liumy2010/UFT . 1 Introduction Figure 1 : (top left, top right, middle, bottom). The illustration of SFT, RFT, SFT-RFT, and UFT, respectively. SFT-RFT refers to applying RFT after an initial SFT stage ( DeepSeek-AI et al., 2025 ; Zeng et al., 2025 ) . (Top, center). shows the annotation usage of different algorithms over training. Curves are slightly shifted for better visibility. Figure 2 : Presentation for different algorithms’ accuracy when trained on Countdown ( Wikipedia contributors, 2025 ) , MATH(3,4,5) (level 3-5 only) ( Hendrycks et al., 2021 ; Zeng et al., 2025 ) , and the Knights and Knaves logic puzzle (Logic) ( Xie et al., 2025 ) . Accuracy is averaged over Qwen2.5 models of sizes 0.5B, 1.5B, and 3B ( Qwen et al., 2025 ) . Base refers to the model without fine-tuning, and R^{3} is the curriculum reinforcement learning baseline ( Xi et al., 2024 ) . The figure shows that UFT outperforms both SFT and RFT, while the relative performance of SFT and RFT varies depending on task complexity. When humans learn a new subject, we typically practice with problem sets (thinking) and try to understand the solutions when we encounter difficulties (memorizing). There are also counterparts in fine-tuning LLMs, which is • Supervised Fine-Tuning (SFT). Memorizing the collected reasoning trace (solution) by maximizing the log-likelihood of it. • Reinforcement Fine-Tuning (RFT). Exploring the reasoning space of LLM and improving the performance according to the signal from a verifier of the final answer (thinking). However, unlike humans, learning and thinking are disentangled during the training of language models. Specifically, prior work ( DeepSeek-AI et al., 2025 ; Zhou et al., 2023 ; Muennighoff et al., 2025 ; Liu et al., 2025 ; Zeng et al., 2025 ) typically applies either SFT or RFT throughout the fine-tuning phase, or applies RFT only after SFT completes (cf. Figure 1 ). The choice of the proper fine-tuning algorithm depends on the LLM’s capacity and the task’s complexity. Specifically, when the LLM is weak, SFT typically works better since the LLM cannot explore the correct answer during reinforcement learning ( Pan et al., 2025 ) , due to the sparse reward caused by the verifier-based reward model. On the other hand, when the LLM is strong, RFT generalizes better ( Xie et al., 2025 ; Chu et al., 2025 ) . To get the best of both worlds, we propose Unified Fine-Tuning (UFT), which unifies SFT and RFT and enriches the reinforcement learning signal with supervised feedback, enabling the model to acquire new knowledge during fine-tuning more efficiently. In Figure 1 , SFT-RFT refers to the common practice of initiating reinforcement learning from a supervised fine-tuned model, as widely adopted in the literature ( DeepSeek-AI et al., 2025 ; Zeng et al., 2025 ) . As shown in Figure 1 (top left), SFT uses full annotations (solutions) throughout training, whereas RFT does not use any annotations at all ( Figure 1 , top right). Similarly, SFT-RFT begins with SFT using full annotations, but once the RFT phase starts, it discards all annotations and relies entirely on exploration. In contrast, our method, UFT, offers a smooth transition from SFT to RFT, preserving the annotation signal early on and gradually reducing it as the model becomes capable of self-guided reasoning. The most relevant work to UFT is Learning Reasoning through Reverse Curriculum Reinforcement Learning ( \text{R}^{3} ) ( Xi et al., 2024 ) , which proposes a curriculum learning method that concatenates the problem with a slice of the solution (hint, cf. Figure 4 left). While \text{R}^{3} treats hints primarily as exploration aids, UFT further integrates them as part of the supervision signal. This unification enables reinforcement learning not just to search, but to learn from existing solutions, effectively raising the performance ceiling imposed by the model’s pretraining capacity (cf. Figure 2 ). A detailed comparison with related work is postponed to Appendix A . Figure 2 shows the accuracy of different algorithms over time, while the training set is Countdown ( Wikipedia contributors, 2025 ; Pan et al., 2025 ) , MATH(3,4,5) (levels 3–5 only) ( Hendrycks et al., 2021 ; Zeng et al., 2025 ) , and the Knights and Knaves logic puzzle (Logic) ( Xie et al., 2025 ) . Base refers to the model before fine-tuning, and R^{3} represents the curriculum reinforcement learning baseline ( Xi et al., 2024 ) . As shown in the figure, UFT generally outperforms all other algorithms. Furthermore, we provide the evaluation on various benchmarks, and the results are shown in Table 8 . Moreover, we theoretically prove that RFT ( DeepSeek-AI et al., 2025 ; Zeng et al., 2025 ; Liu et al., 2025 ) suffers from an inherent sample complexity bottleneck, which is exponential in the length of the reasoning. In contrast, the unified training paradigm in UFT can improve the sample complexity to a polynomial dependence on the reasoning length, which is an exponential improvement over RFT. 1.1 Contribution We state the contribution of this paper in the following. 1. Integration of Supervision and Reward Signal. UFT provides a general framework that integrates the supervision from SFT and reward from RFT into a single training paradigm. UFT blends reward optimization with log-likelihood maximization on hints (partial solution), and smoothly transitions from fully supervised to fully reinforcement learning. Such integration allows models to explore and learn simultaneously, addressing the trade-off between memorization (SFT) and generalization (RFT) in a principled way. 2. Theoretical Justification. We provide a theoretical analysis of UFT, proving it achieves polynomial sample complexity dependence on reasoning length, compared to the exponential complexity required by standard RFT. This result formally establishes the efficiency gains from unifying learning (cf. Section 4 ). 3. Empirical Validation Across Model Scales and Tasks. We evaluate the algorithms by training Qwen2.5-0.5/1.5/3B ( Qwen et al., 2025 ) and Llama3.2-1/3B ( Grattafiori et al., 2024 ) on Countdown ( Wikipedia contributors, 2025 ; Pan et al., 2025 ) , MATH ( Hendrycks et

Read the complete text