Relative Trajectory Balance is equivalent to Trust-PCL

Tristan Deleu ​, Padideh Nouri ​, Yoshua Bengio ​, Doina Precup ​[1ex] Mila – Quebec AI Institute · 2025

Read Relative Trajectory Balance is equivalent to Trust-PCL by Tristan Deleu ​, Padideh Nouri ​, Yoshua Bengio ​, Doina Precup ​[1ex] Mila – Quebec AI Institute (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01632.txt.

Introduction

Fine-tuning generative models with reinforcement learning has grown central. A recent objective, Relative Trajectory Balance, was introduced for that role in flow networks. Tristan Deleu, Padideh Nouri, Yoshua Bengio, and Doina Precup prove a key fact. This objective is exactly equivalent to Trust-PCL. That is an earlier off-policy method with a penalty that keeps new behavior near old.

The bridge rests on known links between flow networks and maximum-entropy learning. The equivalence places the newer objective inside a broader theory. That theory covers KL-regularized methods, which penalize drift from a reference model. The team revisits an example from the original paper. Older methods reach comparable performance there. Two literatures, they clarify, describe the same mechanism.

Catalog record

CREATORTristan Deleu ​, Padideh Nouri ​, Yoshua Bengio ​, Doina Precup ​[1ex] Mila – Quebec AI Institute
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01632

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Relative Trajectory Balance is equivalent to Trust-PCL Tristan Deleu ​, Padideh Nouri ​, Yoshua Bengio ​, Doina Precup ​[1ex] Mila – Quebec AI Institute Abstract Recent progress in generative modeling has highlighted the importance of Reinforcement Learning (RL) for fine-tuning, with KL-regularized methods in particular proving to be highly effective for both autoregressive and diffusion models. Complementing this line of work, the Relative Trajectory Balance (RTB) objective was recently introduced in the context of Generative Flow Networks (GFlowNets) to serve the same role of improving fine - tuning in sequential generative models. Building on prior work linking GFlowNets and maximum - entropy RL, we establish in this paper an equivalence between RTB and Trust-PCL, an off-policy RL method with KL regularization. This equivalence situates RTB within the broader theoretical landscape of KL-regularized RL, and clarifies its relationship to earlier methods. Leveraging this insight, we revisit an illustrative example from the RTB paper and show that KL-regularized RL methods achieve comparable performance, offering an alternative perspective to what was previously reported. 1 Introduction Generative models are playing an increasingly prominent role in artificial intelligence, ranging from language and reasoning ( Brown et al. 2020 ; Wei et al. 2022 ) , to vision ( Ho et al. 2020 ; Rombach et al. 2022 ) and scientific discovery ( Abramson et al. 2024 ) . At the heart of this rapid progress, fine-tuning is central to aligning models with downstream applications while preserving diversity and prior behavior. To complement supervised fine-tuning ( Ouyang et al. 2022 ) , Reinforcement Learning (RL) with KL regularization has proven effective and is widely used for this task ( Jaques et al. 2017 ; Uehara et al. 2024 ) , where reward maximization is traded off against maintaining consistency with a pretrained model. In parallel, Generative Flow Networks ( Bengio et al. 2021 ; Bengio et al. 2023 , GFlowNets;) have been proposed as an alternative paradigm to train generative models with techniques inspired by RL. They have since gained popularity in areas such as combinatorial optimization ( Zhang et al. 2023 ) , causal discovery ( Deleu et al. 2022 ) , and scientific discovery ( Jain et al. 2023 ) . Extending ideas from GFlowNets, the Relative Trajectory Balance objective ( Venkatraman et al. 2024 , RTB;) was recently proposed as an alternative to KL-regularized RL for fine-tuning generative models, framing the problem in terms of Trajectory Balance ( Malkin et al. 2022 ) . In this paper, we show that RTB is exactly equivalent to an existing RL algorithm called Trust-PCL ( Nachum et al. 2018 ) , reinforcing the strong ties existing between GFlowNets and (entropy-regularized) RL ( Tiapkin et al. 2024 ; Mohammadpour et al. 2024 ; Deleu et al. 2024 ; Jiralerspong et al. 2025 ) . In particular, while Venkatraman et al. 2024 suggested that KL-regularized RL methods may perform poorly even on simple tasks, we argue and show on their illustrative example that these issues stem from algorithmic and reward choices rather than from any fundamental limitation of RL with KL regularization itself. This new perspective places RTB within the well-established framework of KL-regularized RL and underscores its actual source of effectiveness. 2 Background We study the problem of training a sequential generative model capable of sampling objects as a sequence of multiple steps. This covers, for example, the case of diffusion samplers ( Sendera et al. 2024 ) that generate images {\bm{x}}_{T} as a sequence of denoising steps {\bm{x}}_{1}\rightarrow{\bm{x}}_{2}\rightarrow\ldots\rightarrow{\bm{x}}_{T} , starting from some Gaussian noise {\bm{x}}_{1}\sim{\mathcal{N}}({\bm{0}},{\bm{I}}) and sequentially sampling from a denoising model {\bm{x}}_{t+1}\sim P_{\phi}({\bm{x}}_{t+1}\mid{\bm{x}}_{t}) . By following this iterative process, we eventually obtain samples from the marginal distribution defined by P_{\phi}^{\top}({\bm{x}}_{T})=\int P_{\phi}({\bm{x}}_{1},{\bm{x}}_{2},\ldots,{\bm{x}}_{T})\,d{\bm{x}}_{1:T-1}=\int\bigg[P({\bm{x}}_{1})\prod_{t=1}^{T-1}P_{\phi}({\bm{x}}_{t+1}\mid{\bm{x}}_{t})\bigg]\,d{\bm{x}}_{1:T-1}. (1) We assume that we have access to a base sequential generative model, with transition probabilities \pi_{\mathrm{prior}}({\bm{x}}_{t+1}\mid{\bm{x}}_{t}) , such as a model pre-trained with a large dataset of observations, and that we are also given a fixed energy function {\mathcal{E}}({\bm{x}}_{T}) that we can query for any object generated by the model. Our objective in this paper is to learn a sequential generative model P_{\phi}({\bm{x}}_{t+1}\mid{\bm{x}}_{t}) such that its marginal distribution is the tilted distribution , modulated by {\mathcal{E}}({\bm{x}}_{T}) : P_{\phi}^{\top}({\bm{x}}_{T})\propto\pi_{\mathrm{prior}}^{\top}({\bm{x}}_{T})\exp(-{\mathcal{E}}({\bm{x}}_{T})/\alpha), (2) for some temperature parameter \alpha>0 . The role of the energy function is to steer generation towards samples with desired properties ( e.g. , generating images based on a description). 2.1 Entropy-regularized Reinforcement Learning One way to approach this problem is to view it from the perspective of Reinforcement Learning ( Korbak et al. 2022 ) . We consider a finite-horizon Markov Decision Process (MDP) {\mathcal{M}}=(\macc@depth\char 1\relax\macc@set@skewchar\macc@nested@a 111{},{\mathcal{A}},r) , where \macc@depth\char 1\relax\macc@set@skewchar\macc@nested@a 111{}={\mathcal{S}}\cup\{\bot\} is a state space augmented by a special state \bot\notin{\mathcal{S}} indicating the end of a trajectory, {\mathcal{A}} is the action space, and r is a reward function that we will detail below. We identify a state s_{0}\in{\mathcal{S}} called the initial state from which all trajectories start; these trajectories are guaranteed to end in \bot since the MDP is finite-horizon. To give a concrete example, we may see the multiple steps of denoising in a diffusion sampler as an MDP with a specific structure ( Fan et al. 2023 ; Black et al. 2024 ) : its states are of the form s_{t}=({\bm{x}}_{t},t) , transitions (actions) correspond to applying one step of denoising ({\bm{x}}_{t},t)\rightarrow({\bm{x}}_{t+1},t+1) , and trajectories must terminate when a state of the form ({\bm{x}}_{T},T) is reached after T steps (termination being indicated by the transition ({\bm{x}}_{T},T)\rightarrow\bot ). Following Deleu et al. 2024 , the reward function r(s,s^{\prime}) obtained when transitioning from s\rightarrow s^{\prime} is defined such that the sum of rewards along a trajectory only depends on the energy of the final state it reaches right before terminating; in other words, for a trajectory \tau=(s_{0},s_{1},\ldots,s_{T},\bot) , we have \sum_{t=0}^{T}r(s_{t},s_{t+1})=-{\mathcal{E}}(s_{T}), (3) where we will use the convention s_{T+1}=\bot throughout this paper. This includes, in particular, the case where the reward is only obtained at the end of the trajectory ( i.e. , r(s_{T},\bot)=-{\mathcal{E}}(s_{T}) , and zero everywhere else, also called outcome-based reward ; Uesato et al. 2022 ). Contrary to standard RL, where the objective is to find a policy \pi(s_{t+1}\mid s_{t}) that maximizes the expected sum of rewards, KL-regularized RL includes an additional KL regularization term between the current policy and an anchor policy \pi_{\mathrm{prior}} we don’t want to deviate too much from: \pi^{\star}_{\mathrm{RelEnt}}=\argmax_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T}r(s_{t},s_{t+1})-\alpha\mathrm{KL}\big(\pi(\cdot\mid s_{t})\,\|\,\pi_{\mathrm{prior}}(\cdot\mid s_{t})\big)\right]. (4) It can be shown that with the particular choice of reward function made in eq. 3 , the marginal distribution associated with this optimal policy is the tilted distribution ( Nachum et al. 2018 ; Korbak et al. 2022 ) \pi^{\star\top}_{\mathrm{RelEnt}}(s_{T})\propto\pi_{\mathrm{prior}}^{\top}(s_{T})\exp(-{\mathcal{E}}(s_{T})/\alpha), (5) where the temperature parameter \alpha naturally emerges from the regularization constant in eq. 4 ; a proof of this is given in appendix A for completeness. This constitutes the foundation of the recent line of work using RL for fine-tuning language models ( Jaques et al. 2017 ; Ouyang et al. 2022 ) and diffusion models ( Fan et al. 2023 ; Black et al. 2024 ; Uehara et al. 2024 ) . 2.2 Relative trajectory balance Taking another perspective, this time rooted in the literature on GFlowNets ( Bengio et al. 2023 ) , Venkatraman et al. 2024 introduced an

Read the complete text