f f -Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Data
Jake Fawkes, Jason Hartford · 2026
Read f f -Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Data by Jake Fawkes, Jason Hartford (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15417.txt.
Introduction
Jake Fawkes and Jason Hartford widen a low-variance loss for generative models. The old loss matched one divergence. Their new family spans all f-divergences. On-policy, each loss has gradients that match its own divergence. Off-policy, it keeps the same best answer. Designers can thus pick traits like wider coverage with no loss of the off-policy pledge. Tests span toy tasks, molecule search with SynFlowNets, and async tuning of large language models.
Catalog record
| CREATOR | Jake Fawkes, Jason Hartford |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98696 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15417
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
f f -Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Data Jake Fawkes, Jason Hartford Abstract In GFlowNets and variational inference, it has been shown that the mean square error between target and model log probabilities is an effective, low variance, surrogate loss for training generative models. This loss has the property that when evaluated on-policy its gradients correspond to those of the KL divergence, while off-policy it remains a valid loss with the same global minimizer. In this work, we demonstrate that this construction can be extended to the whole family of f f -divergences, leading to a family of losses whose on-policy gradients are that of the corresponding f f -divergence, but retain the same global minimizer off-policy. Specifically, we show that the on-policy gradients lead to a one to one correspondence between translation invariant loss functions on the target and model log probabilities, and f f -divergences. This equivalence allows us to design new surrogate loss functions for tuning a wide class of generative models that inherit the properties of the corresponding f f -divergence, such as being more mode covering, whilst being applicable to off-policy data. We apply our losses on a range of tasks, including classic synthetic examples, SynFlowNets for molecule discovery, and asynchronous large language model (LLM) tuning, demonstrating that our models retain their predicted properties on- and off-policy in a wide class of generative models. 1 Introduction Figure 1 : f -divergences include the \alpha -divergence family which contains both Forward KL ( \alpha\!=\!0 ) and Reverse KL / standard Trajectory Balance ( \alpha\!=\!1 ) as special cases. Lower \alpha yields a more mode-covering loss, higher \alpha a more mode-seeking one. The alignment and fine-tuning of generative models—whether they are large language models (LLMs) generating text, or Generative Flow Networks (GFlowNets) generating molecular graphs—relies on minimizing the discrepancy between the model’s distribution. In Reinforcement Learning (RL) fine-tuning, this objective is often framed as maximizing expected reward subject to a KL-divergence penalty, or equivalently, minimizing a divergence between the policy and the target distribution. Traditionally, optimizing these objectives involves gradient estimators with high variance, such as REINFORCE ( Williams 1992 ) , or sophisticated policy gradient methods ( Schulman et al. 2017 , e.g.) . However, recently the GFlowNet literature and large scale LLM tuning in Kimi ( Team et al. 2025a ) has used a simpler approach: minimizing the mean squared error (MSE) between the log-probabilities of the model and the target. This has similarities with the K2 estimator of the \mathbb{KL} divergence proposed in Schulman 2020 , which we term \mathbb{KL}_{sq} . Tang and Munos 2025 , note this “squared KL” loss has a surprising property: whilst it is a biased estimator of the \mathbb{KL} , its gradients on on-policy data are unbiased to the true KL gradients. Further, as it is built on the MSE, it remains a valid loss function with the same global minimizer when applied to off-policy data ( Bartoldson et al. 2025 ; Tang et al. 2025 ) . While the \mathbb{KL}_{sq} estimator offers stability and off-policy compatibility, by construction, it inherits the properties of the reverse KL divergence. In particular, models trained under this objective will typically exhibit “mode-seeking” behavior where the model collapses onto relatively few high-probability modes rather than covering the full diversity of the target distribution. In many generative tasks, such as drug discovery or exploratory agents, “mode-covering” behavior is often preferred because generative models are used to explore the space of high reward candidates (e.g. all possible drug targets), rather than to seek out one (or a few) particularly good candidate(s). In this work, we demonstrate that the effectiveness of the \mathbb{KL}_{sq} loss is not a unique phenomenon restricted to the KL divergence by deriving analogous losses for the entire family of f -divergences. We establish a theoretical equivalence showing that any translation-invariant loss function on log-probabilities corresponds to a specific f -divergence. This allows us to derive a new family of surrogate losses that inherit the specific properties of their parent divergences (e.g., the mode-covering behavior of the Forward KL or Hellinger distance) while retaining the optimization benefits of the \mathbb{KL}_{sq} estimator: low-variance gradients on-policy and validity off-policy. We demonstrate the behaviour and utility of this family of loss functions across a variety of tasks. On the Hypergrid task ( Bengio et al. 2021 ) —a synthetic task explicitly designed to test mode coverage—we show that the Hellinger and forward KL losses discover more modes of the reverse KL loss implied by the trajectory balance loss ( Malkin et al. 2022b ) . We then apply these losses to both molecule generation—by modifying the losses in SynFlowNet ( Cretu et al. 2025 ) —, class conditional sampling in diffusion models and asynchronous reinforcement learning on language models. In both cases, we found that we were able to learn higher entropy policies with mode-covering instances of our family of surrogate loss functions. In summary, our contributions are as follows: • We derive a general family of translation-invariant surrogate loss functions, \mathcal{L}_{f} , whose expected auto-differentiated gradients match those of the corresponding f -divergence on-policy. • We prove the inverse relationship: any convex, translation-invariant loss on log-probabilities corresponds to minimizing a specific f -divergence. • We generalize the “Vargrad” ( Richter et al. 2020 ) and batch-wise normalization techniques to arbitrary f -divergences, addressing the intractable partition function problem in RL fine-tuning. • We apply this framework to GFlowNets, showing that our losses generalize the standard Trajectory Balance objective ( Malkin et al. 2022a ) , and empirically demonstrate on synthetic grids and SynFlowNet molecular discovery tasks that we can control the exploration-exploitation trade-off (mode-seeking vs. mode-covering) simply by changing the surrogate loss function. 2 Related work GFlowNets and GFlowNet losses Bengio et al. 2021 introduce GFlowNets and train them via the flow matching loss, which applies directly to a neural network aiming to learn the optimal Markovian flow for a given GFlowNets and applies on an action level. The detailed balance objective was proposed in Bengio et al. 2023 , which again applies to the action level but parametrises a forward and backward policy distribution. Malkin et al. 2022a propose trajectory balance, which enforces an equality constraint between the forward policy, backward policy, and normalisation constant on the trajectory level and penalises it with the mean square error. It was shown in Malkin et al. 2023 that on-policy training with this loss is equivalent to a form of hierarchical variational inference ( Ranganath et al. 2016 ) with the \mathbb{KL} divergence. Our results extend these to all f -divergences. The Vargrad objective ( Richter et al. 2020 ) is often used in GFlowNet training, which use the same square error form as Malkin et al. 2022a but estimate the normalisation constant in a batch-wise fashion. This Vargrad loss comes from the log variance divergence introduced in ( Nüsken and Richter 2021 ) , where they also propose a variance based estimator built of the Pearson \chi^{2} divergence which is most similar to our work. However, this would not share the same property of matching on-policy gradients as we show in the paper the correct