Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

Taojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao, Qiaobo Hao, Zhiguo Yang, Yonghong He · 2026

Read Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning by Taojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao, Qiaobo Hao, Zhiguo Yang, Yonghong He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08926.txt.

Introduction

Zhu and six colleagues propose DYPO, bridging supervised fine tuning and reinforcement learning for robust reasoning. Fine tuning is stable but biased, reinforcement learning explores but suffers gradient variance, and naive loss weighting ignores the clash of gradients. DYPO combines group alignment loss cutting RL gradient variance, multi teacher distillation fixing fine tuning bias via diverse paths, and dynamic gating arbitrating by reward feedback. Theory confirms bias and variance cuts, and tests show average gains of 4.8 percent on complex reasoning benchmarks and over 13 percent on out-of-distribution tasks.

Catalog record

CREATORTaojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao, Qiaobo Hao, Zhiguo Yang, Yonghong He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55268
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08926

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning Taojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao, Qiaobo Hao, Zhiguo Yang, Yonghong He Abstract Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose DYPO (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a Group Alignment Loss (GAL) that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a Multi-Teacher Distillation mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a Dynamic Exploitation-Exploration Gating mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8% on complex reasoning benchmarks and 13.3% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO . 1 Introduction The reasoning capabilities of Large Language Models (LLMs) have become a central focus in artificial intelligence 9 ; 6 ; 27 . While reasoning-guidance techniques like Chain-of-Thought (CoT) prompting have significantly advanced model performance on multi-step tasks 31 , traditional prompting methods relying on static templates struggle with scalability and dynamic adaptability. Consequently, the research focus has shifted toward the post-training stage to enhance robustness and generalization 29 . (a) Data Flow & Training Mechanisms (b) Model performance (c) Learning Curves Figure 1: The SFT-RL Dilemma: Balancing the high-bias stability of SFT against the high-variance exploration of RL. Current mainstream post-training paradigms generally fall into two categories: i) Supervised Fine-Tuning (SFT): SFT offers efficient knowledge injection by learning from high-quality CoT corpora 22 ; 30 . Its low-variance nature ensures stability and rapid fitting, but often at the cost of limited exploratory capacity and restricted Out-of-Distribution (OOD) generalization. ii) Reinforcement Learning (RL): Methods such as RLHF or RLVR allow models to autonomously explore the reasoning space via reward signals, substantially enhancing generalization 18 ; 20 ; 23 ; 24 . Unlike SFT, RL relies on the base model’s intrinsic capabilities; consequently, weaker models often struggle to capture sparse reward signals in complex tasks. To combine these strengths, researchers have widely adopted a “SFT-then-RL” training pipeline 28 ; 36 . However, this sequential approach suffers from bias propagation , where SFT-induced biases misguide subsequent RL exploration 16 , alongside significant computational overhead. Recent research has therefore shifted toward unified optimization , which combines SFT and RL objectives within a single training process 33 ; 5 ; 38 ; 1 . Representative methods include SuperRL 14 , which adopts a binary switching strategy between supervision and reinforcement learning, and CHORD 38 , which harmonizes the two objectives through dynamic soft weighting. These approaches highlight the growing interest in unified SFT-RL post-training, but they still apply a largely uniform optimization recipe across samples whose learning signals differ fundamentally in reliability. Despite the growing interest in unified SFT-RL training, existing fusion strategies predominantly operate at a “surface level” via simple loss weighting 16 . This approach overlooks two fundamental issues. First, it ignores the inherent statistical conflict between the gradient signals: SFT gradients are high-bias (fitting static data) but low-variance 32 , whereas RL gradients are low-bias (reward-driven) but high-variance (due to sampling stochasticity) 20 . Naively aggregating these conflicting vectors is sub-optimal, as RL’s high variance destabilizes training while SFT’s high bias constrains exploration. Second, this uniform approach fails to account for regime-dependent differences in sample difficulty. Specifically, trivial samples provide marginal optimization signals since model performance is already saturated; hard samples yield extremely sparse rewards, rendering RL highly inefficient; and only mid-difficulty samples simultaneously preserve reward discrimination and expose meaningful failure modes. Consequently, globally mixing SFT and RL objectives cannot fully resolve the multidimensional mismatch between stable but biased supervision and exploratory but high-variance policy optimization. In this paper, we first provide a theoretical analysis formally defining this bias-variance trade-off in SFT-RL fusion. We then propose DYPO ( D Y namic P olicy O ptimization), a unified framework that introduces structural solutions to concurrently mitigate both limitations.Unlike binary switching or soft weighting methods, DYPO performs instance-level routing based on rollout outcomes and assigns each regime to a distinct optimization objective. Specifically, DYPO comprises three core components: • Dynamic Difficulty Grading : A mechanism that dynamically categorizes queries based on group rollout outcomes. It effectively arbitrates the optimization pathway: routing complete failures (Hard) to stable SFT for knowledge injection, while directing inconsistent attempts (Mid) to low-bias RL for exploration. • Bias Correction (SFT) : For ‘Hard’ samples, we employ a Multi-Teacher Distillation mechanism to correct the fitting bias inherent in SFT by aggregating diverse reasoning paths from different teacher models. • Variance Reduction (RL) : For ‘Mid’ samples, we introduce a Group Alignment Loss (GAL) 19 that leverages intrinsic group dynamics.By effectively reinforcing winning samples while suppressing losing ones, GAL significantly reduces RL gradient variance compared to standard pairwise losses. Theoretically, we prove that our Dynamic Difficulty Grading mechanism minimizes overall variance by strategically allocating queries based on reward feedback. For ‘Hard’ samples, the triggered multi-teacher strategy linearly reduces fitting bias; for ‘Mid’ samples, the GAL reduces gradient variance by orders of magnitude compared to GRPO. Experimentally, DYPO yields 5–10% performance gains on complex reasoning benchmarks. 2 Preliminaries In this section, we formalize the reasoning trace generation problem and review the two foundational paradigms: SFT and RL. We specifically highlight their respective statistical challenges—fitting bias in SFT and gradient variance in RL—which motivate our proposed approach. 2.1 Problem Formulation We model the reasoning task as a sequential decision-making process. Given an input prompt q sampled from a distribution \mathcal{D} , the LLM functions as a stochastic policy \pi_{\theta}(\tau|q) parameterized by \theta . Here, \tau=(a_{1},a_{2},\dots,a_{T}) represents a reasoning trajectory consisting of a sequence of tokens. The probability of generating a trajectory is factorized autoregressively: \pi_{\theta}(\tau|q)=\prod_{t=1}^{T}\pi_{\theta}(a_{t}|q,a_{<t}) (1) Upon completion, the trajectory \tau is evaluated by a reward function R(q,\tau)\in\mathbb{R} . The objective is to maximize the expected reward J(\theta)=\mathbb{E}_{q\sim\mathcal{D}}\mathbb{E}_{\tau\sim\pi_{\theta}(\cdot|q)}[R(q,\tau)] 18 ; 39 . 2.2 SFT and Fitting Bias Standard SFT adapts the policy by minimizing the negative log-likelihood on a static dataset \mathcal{D}_{\text{sft}} containing gold-standard pairs (q,\tau^{*}) 28 ; 30 : \mathcal{L}_{\text{SFT}}(\theta)=\mathbb{E}_{(q,\tau^{*})\sim\mathcal{D}_{\text{sft}}}\left[-\log\pi_{\theta}(\tau^{*}|q)\right] (2) While SFT provides stable supervision, it inherently suffers from fitting bias . Since the optimization is constrained to the fixed support of \mathcal{D}_{\text{sft}} , the model tends to overfit the specific distribution of the single teacher or dataset. This mimicry limits the model’s ability to explore novel reasoning paths and often leads to sub-optimal local minima where the policy fails to generalize beyond the training examples. 2.3 GRPO and Gradient Variance To enable exploration, we employ GRPO 24 . For each prompt q , GRPO samples a group of trajectories G=\{\tau_{1},\tau_{2},\dots,\tau_{k}\} and optimizes the policy using group-normalized advantages: \begin{array}[]{r@{\,}l}\mathcal{L}_{\text{GRPO}}(\theta)&=-\mathbb{E}_{q\sim\mathcal{D}}\Bigg[\frac{1}{k}\sum_{i=1}^{k}\Big(\text{CLIP}(\rho_{i},\hat{A}_{i},\epsilon)\Big)\\[-4.0pt] &\mspace{25mu}-\beta_{\text{KL}}\mathbb{D}_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})\Bigg]\end{array} (3) where \rho_{i} is the

Read the complete text