Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
Zhiqiang Pu, Yang Liu, Xiaolin Ai · 2026
Read Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner by Zhiqiang Pu, Yang Liu, Xiaolin Ai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18088.txt.
Introduction
Ma and co-authors add an online refiner to reinforcement learning fine-tuning of language models. Fixed constraints keep training stable but block the model from finding better solutions. Their dynamic constraints act only when output turns degenerate. A reference model minimally corrects each response, keeping right content and fixing errors. A supervised loss then trains the model toward the refined text. Tests on dialogue and code beat KL and unconstrained baselines.
Catalog record
| CREATOR | Zhiqiang Pu, Yang Liu, Xiaolin Ai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43998 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18088
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner Zhiqiang Pu, Yang Liu, Xiaolin Ai Abstract Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose dynamic constraints that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an online refiner that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability. 1 Introduction Reinforcement learning fine-tuning (RFT) has emerged as a powerful paradigm for aligning large language models (LLMs) with task-specific objectives, achieving remarkable success across diverse domains including safety alignment ( 12 ; 22 ) , code generation ( 8 ; 32 ) , and reasoning ( 9 ; 34 ) . Recent algorithmic innovations, such as GRPO ( 30 ) , DAPO ( 38 ) , VAPO ( 40 ) , and GSPO ( 41 ) , have continually pushed the performance boundaries of RFT. However, as reinforcement learning (RL) plays an increasingly prominent role in post-training, it suffers from catastrophic forgetting ( 4 ; 14 ) . This phenomenon arises from the inherent difficulty of tuning a model in a high-dimensional exploration space with sparse rewards. The RL algorithm selectively reinforces task-specific knowledge, causing the policy distribution to become increasingly sharp and concentrated ( 36 ) . As the fine-tuned policy \pi_{\theta} deviates substantially from the reference policy \pi_{0} , the general capabilities acquired during pretraining and supervised fine-tuning (SFT) are gradually eroded ( 4 ) . In extreme cases, this catastrophic forgetting can severely impair the model’s expressive capacity, leading to distribution collapse where the model degenerates into generating incoherent or nonsensical outputs ( 14 ; 24 ; 19 ; 25 ) . A natural mitigation strategy is to impose a regularization term that constrains the fine-tuned policy from drifting too far from the reference one. KL divergence regularization, widely adopted in early RFT algorithms ( 14 ; 27 ) , serves precisely this purpose by anchoring \pi_{\theta} to \pi_{0} . However, this approach introduces a fundamental trade-off. While it prevents catastrophic forgetting, it also limits \pi_{\theta} to stay close to \pi_{0} , inherently restricting exploration and preventing the discovery of optimal policies that may reside in distant regions of the solution space ( 38 ) . Our key insight is that if we pursue extreme performance on specific tasks, we should abandon KL regularization, yet an alternative constraint remains essential to prevent the incoherent outputs induced by catastrophic forgetting. The fundamental limitation of KL regularization lies in its property of being a static constraint . In this paper, we propose replacing this static constraint with a dynamic constraint that adapts to the evolving capabilities of the fine-tuned policy. Rather than anchoring to a fixed reference, the dynamic constraint guides \pi_{\theta} toward an adaptive target that progresses alongside the policy. We realize this idea by prompting the reference model \pi_{0} to act as an online refiner . Given an input query and the response from \pi_{\theta} , the refiner generates a refined version of the response. The dynamic constraint is then defined as the cross-entropy teaching \pi_{\theta} to produce the refined response. This design maximally relaxes the constraint on \pi_{\theta} by only intervening when degenerate outputs occur. When \pi_{\theta} produces a coherent response, the refiner reproduces it verbatim and imposes almose no constraint. Only when degenerate outputs occur does the refiner apply targeted and minimal corrections. As illustrated in Figure 1 , while the static constraint pulls the policy backward toward a fixed reference, the dynamic constraint propels it forward toward an adaptive, higher-quality target. From a broader perspective, our approach unifies RL with SFT on a continuously updated dataset. We validate our approach through comprehensive experiments on dialogue and code generation tasks. We first integrate the dynamic constraint into PPO to analyze its training dynamics compared with static KL regularization, then incorporate it into DAPO ( 38 ) , a state-of-the-art RFT algorithm, and evaluate on challenging code generation benchmarks. Our results reveal two compelling advantages. First, the dynamic constraint substantially stabilizes training by reducing variance and preventing distribution collapse. Second, it enables policies to achieve significantly higher rewards by eliminating the performance ceiling imposed by static constraints. These findings underscore the potential of dynamic constraints for advancing RFT. More broadly, our approach establishes a self-elevating cycle between RL and SFT, opening new avenues for improving both paradigms. (a) Static constraint (b) Dynamic constraint Figure 1: An illustration of the insight of dynamic constraint. (a) The left figure illustrates the conventional KL regularization, which constrains the fine-tuned policy \pi_{\theta} to remain close to the reference policy \pi_{0} . However, when the optimal policy lies far from \pi_{0} and requires \pi_{\theta} to deviate significantly, the KL regularization becomes an obstacle to policy optimization. (b) The right figure presents the insight of the dynamic constraint, where the constraint is derived from \pi_{0} ’s refined response based on \pi_{\theta} ’s response and context. When \pi_{\theta} deviates substantially from \pi_{0} , the dynamic constraint not only avoids hindering policy optimization but also provides effective guidance and correction for \pi_{\theta} . 2 Preliminary 2.1 Problem Formulation To describe RFT using standard RL terminology, we formulate next-token prediction in causal language models as a sequential decision-making process. This is formally modeled as a language-augmented Markov Decision Process ( 16 ) , defined as \mathcal{M}=\langle\mathcal{V},\mathcal{S},\mathcal{A},r,P,\gamma\rangle . Here, \mathcal{V} denotes the vocabulary of the language model, consisting of all possible tokens, and w\in\mathcal{V} refers to a token. The state space is defined as \mathcal{S}\subset\mathcal{V}^{M} , where \mathcal{V}^{M} represents the set of all token sequences of length up to M . Similarly, the action space is \mathcal{A}\subset\mathcal{V}^{N} , where \mathcal{V}^{N} denotes token sequences of length up to N . Here, M and N are the maximum token lengths for states and actions, respectively. A state s\in\mathcal{S} is represented as a token sequence s=(w_{1},w_{2},\dots,w_{M}) , and an action a\in\mathcal{A} is defined as a generated sequence a=(w_{1},w_{2},\dots,w_{N}) . For simpliciy, we denote (w_{1},\dots,w_{i},\dots,w_{t-1}) as a_{<t} , and w_{i} as a_{i} . If the sequence length is shorter than the maximum, it is padded with a special token. The reward function r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} assigns a task-specific scalar reward to each state-action pair. The transition function P:\mathcal{S}\times\mathcal{V}\rightarrow\mathcal{S} defines a deterministic transition based on autoregressive generation. At each timestep, the predicted token is appended to the current state: s_{i}=(s_{i-1},a_{i})=(s_{0},a_{<i+1}) , where s_{0} is the tokenized user input and a_{<i+1} is the partial output sequence. The token-level policy of the language model is denoted by \pi(a_{i}\mid s_{0},a_{<i}) , and the sentence-level policy is defined as the product of token-level predictions: \pi(a|s_{0})=\prod_{i=1}^{N}\pi(a_{i}|s_{0},a_{<i}) . The