Reinforcement Learning via Value Gradient Flow

Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, Amy Zhang · 2026

Read Reinforcement Learning via Value Gradient Flow by Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, Amy Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14265.txt.

Introduction

Xu and three colleagues present Value Gradient Flow, an approach to reinforcement learning that stays close to a reference policy. Regularization toward a reference distribution curbs value over-optimization from out-of-distribution extrapolation, but gradient methods scale poorly and rejection sampling stays conservative. Their method recasts the problem as optimal transport taking the reference distribution toward the value-induced optimal policy, solved by gradient flow with value gradients steering particles. Transport budget control supplies regularization, with no policy parameterization, and experiments top methods on benchmarks and LLM tasks.

Catalog record

CREATORHaoran Xu, Kaiwen Hu, Somayeh Sojoudi, Amy Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60228
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforcement Learning via Value Gradient Flow Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, Amy Zhang Abstract We study behavior-regularized reinforcement learning (RL), where regularization toward a reference distribution (the dataset in offline RL or the base model in LLM RL finetuning) is essential to prevent value over-optimization caused by erroneous out-of-distribution extrapolation. Existing methods either rely on reparameterized policy gradient, which are difficult to scale to large generative models, or on reject sampling, which can be overly conservative when attempting to move beyond the behavior support. In this paper, we propose Value Gradient Flow (VGF), a scalable new paradigm for behavior-regularized RL. VGF casts behavior-regularized RL as an optimal transport problem that maps the reference distribution to the value-induced optimal policy distribution. We solve this transport problem via discrete gradient flow, where value gradients guide particles initialized from the reference distribution. Our analysis shows that VGF imposes regularization implicitly by controlling the transport budget. VGF eliminates explicit policy parameterization while remaining expressive and flexible, this enables adaptive test-time scaling by adjusting the transport budget. Extensive experiments demonstrate that VGF significantly outperforms prior methods, achieving state-of-the-art results on offline RL benchmarks (D4RL, OGBench) and LLM RL tasks. Code and runs can be found at https://ryanxhr.github.io/vgf 1 Introduction Figure 1: VGF: Value Gradient Flow . VGF reframes behavior-regularized RL as an optimal transport from the behavior distribution towards the Boltzmann value distribution, with the transport budget as implicit regularization. This scales to large generative models and enables adaptive test-time scaling. Reinforcement learning (RL) has provided a powerful framework for solving sequential decision-making problems in complex environments. These methods have been successfully applied in diverse domains, ranging from robotics ( Levine et al. 2016 ) to game playing ( Mnih et al. 2013 ; Silver et al. 2017 ) , and have recently become instrumental in fine-tuning large language models (LLMs) to align with human preferences and instructions ( Ouyang et al. 2022 ) and enhancing the reasoning capabilities of LLMs ( Shao et al. 2024 ; DeepSeek-AI et al. 2025 ) . While these successes highlight the broad potential of RL, they also expose a key challenge: policies must often be regularized toward a reference distribution to remain stable and reliable. This challenge arises in both offline RL, where erroneous extrapolation beyond fixed datasets can cause severe value overestimation, and RLHF, where deviating too far from the supervised policy risks reward hacking. In these settings, naïve value maximization alone is insufficient. As a result, recent research in both offline RL ( Kumar et al. 2020 ; Fujimoto et al. 2019 ; Wu et al. 2019 ; Xu et al. 2023 ) and RLHF ( Ouyang et al. 2022 ; Wang et al. 2024 ) has increasingly converged on the paradigm of behavior-regularized RL , which balances value maximization with adherence to reliable reference distributions. The most common approach to behavior-regularized RL is to add explicit divergence or distance penalties (e.g., L2 distance or KL divergence) to the RL learning objective ( Ouyang et al. 2022 ; Touvron et al. 2023 ; Gao et al. 2023 ) . While this constrains policies to remain close to the reference distribution, it also introduces several limitations. First, explicit penalty methods typically use a single coefficient to regularize both value learning and policy improvement, even though these two components can prefer different regularization strengths. Tying them together makes the penalty coefficient difficult to tune and can lead to either overly conservative updates or insufficient control of out of distribution drift ( Wu et al. 2025 ; Korbak et al. 2022 ) . Second, scaling these methods to expressive generative policies such as diffusion ( Song et al. 2020 ; Ho et al. 2020 ) and flow models ( Lipman et al. 2022 ) is challenging. Computing policy gradient requires differentiating through multi step sampling procedures, which is unstable and computationally expensive, while distillation the multi step policy into a one step model can sacrifice expressivity ( Park et al. 2025b ) . Behavior-regularized RL without regularization. 1 1 1 Although VGF uses transport budget as an implicit regularization, here we use ”without regularization” to emphassize that VGF doesn’t involve any auxiliary regularization during optimization. We introduce Value Gradient Flow (VGF) , which casts behavior-regularized RL as the optimal transport from an (estimated) reference distribution to the optimal policy distribution induced by the value function. VGF adds no explicit distance or divergence penalty and does not rely on a parameterized policy. Instead, it starts from samples of the reference distribution and gradually nudges them toward higher value regions using a small, fixed number of guidance steps. The distribution after these steps serves as an implicit policy. The transport budget itself (i.e., how far and how often we move) acts as an implicit behavior regularization, limiting deviation from the reference distribution during training while preserving flexibility to enable adaptively scaling at inference. Notably, VGF supports different levels of regularization during training and inference. When the inference transport budget is zero, VGF reduces to reject sampling methods, Conversely, using a larger transport budget at inference than during training brings test-time-scaling improvements. In RLHF, VGF yields inference-time control that steers a supervised finetuned policy using first-order gradient, sidestepping RL-style optimization and reducing both compute and engineering complexity. Across extensive experiments, VGF consistently outperforms strong behavior-regularized baselines, attaining state-of-the-art results on standard offline RL suites (D4RL, OGBench) and delivering substantial gains on RLHF tasks. 2 Preliminaries MDP and value functions. We consider the RL problem presented as a Markov Decision Process (MDP) ( Sutton et al. 1998 ) , which is specified by a tuple \mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{P},d_{0},r,\gamma\rangle . Here \mathcal{S} and \mathcal{A} are state and action space, \mathcal{P}(s^{\prime}|s,a) and d_{0} denote transition dynamics and initial state distribution, r(s,a) and \gamma represent reward function and discount factor, respectively. The goal of RL is to find a policy \pi(a|s) which maximizes expected return J(\pi)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}\cdot r(s_{t},a_{t})] . In the offline setting, interaction with the environment is prohibited and one needs to learn an optimal \pi from a static replay buffer \mathcal{D}=\{s_{i},a_{i},r_{i},s^{\prime}_{i}\}_{i=1}^{N} collected from unknown policies. The dataset can be heterogeneous and suboptimal, we denote the empirical behavior policy of \mathcal{D} as \pi_{\mathcal{D}} , which represents the conditional distribution p(a|s) observed in the dataset. RL methods based on approximate dynamic programming typically maintain an action-value function ( Q -function) and, optionally, a state-value function ( V -function), referred to as Q(s,a) and V(s) respectively ( Haarnoja et al. 2017 ; Nachum et al. 2017 ; Kumar et al. 2020 ; Kostrikov et al. 2021b ) . Define Q^{\pi}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} , where Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a\right] . The value function is learned by satisfying single-step Bellman consistencies. Let \mathcal{T}^{\pi} be the Bellman operator with policy \pi such that (\mathcal{T}^{\pi}Q)(s,a):=r(s,a)+\gamma\mathbb{E}_{s^{\prime}|s,a}\mathbb{E}_{a^{\prime}\sim\pi}\left[Q(s^{\prime},a^{\prime})\right] . Then Q are learned by \min_{Q}J(Q)=\frac{1}{2}\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[(\mathcal{T}^{\pi}Q-Q)(s,a)^{2}\right] . Behavior-regularized RL. In general, behavior-regularized RL considers the following constraint optimization problem with a reference distribution/policy \mu : \pi^{\ast}=\arg\max\limits_{\begin{subarray}{c}\pi\end{subarray}}\ \mathbb{E}_{s\sim\mathcal{D},a\sim\pi(\cdot|s)}\big[R(s,a)\big]\ \ \text{s.t.}\ \ \mathbb{E}_{s\sim\mathcal{D}}\big[M\left(\pi(\cdot|s),\mu(\cdot|s)\right)\big]\leq\epsilon, (1) where R(s,a) is a differentiable function and M is some distance or divergence measure (e.g., KL-divergence, L_{2} distance). \mu could be the offline data distribution \pi_{\mathcal{D}} in offline RL or the base model after pretraining in LLM RL. R(s,a) could be either (multi-step) Q -function or (single-step) reward model in

Read the complete text