Robust Reinforcement Learning for Discrete Compositional Generation via General Soft Operators

Marco Jiralerspong, Esther Derman, Danilo Vucetic, Nikolay Malkin, Bilun Sun, Tianyu Zhang, Pierre-Luc Bacon, Gauthier Gidel · 2025

Read Robust Reinforcement Learning for Discrete Compositional Generation via General Soft Operators by Marco Jiralerspong, Esther Derman, Danilo Vucetic, Nikolay Malkin, Bilun Sun, Tianyu Zhang, Pierre-Luc Bacon, Gauthier Gidel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17007.txt.

Introduction

Marco Jiralerspong and co-authors sample top-grade molecules and proteins without drifting off target, in work from 2025. Science often means sifting a vast set of candidates down to a few strong ones. Learning samplers guided by a proxy score can help, but their spread tricks may yield too much range and too few winners.

The team fuses several soft scoring rules into one operator that aims at sharper peaks. They also recast the whole hunt as robust play against proxy error. Their method, named TGM, found better and still varied candidates than rivals on fake and real tasks. Code is shared in public.

Catalog record

CREATORMarco Jiralerspong, Esther Derman, Danilo Vucetic, Nikolay Malkin, Bilun Sun, Tianyu Zhang, Pierre-Luc Bacon, Gauthier Gidel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Reinforcement Learning for Discrete Compositional Generation via General Soft Operators Marco Jiralerspong, Esther Derman, Danilo Vucetic, Nikolay Malkin, Bilun Sun, Tianyu Zhang, Pierre-Luc Bacon, Gauthier Gidel Abstract A major bottleneck in scientific discovery involves narrowing a large combinatorial set of objects, such as proteins or molecules, to a small set of promising candidates. While this process largely relies on expert knowledge, recent methods leverage reinforcement learning (RL) to enhance this filtering. They achieve this by estimating proxy reward functions from available datasets and using regularization to generate more diverse candidates. These reward functions are inherently uncertain, raising a particularly salient challenge for scientific discovery. In this work, we show that existing methods, often framed as sampling proportional to a reward function, are inadequate and yield suboptimal candidates, especially in large search spaces. To remedy this issue, we take a robust RL approach and introduce a unified operator that seeks robustness to the uncertainty of the proxy reward function. This general operator targets peakier sampling distributions while encompassing known soft RL operators. It also leads us to a novel algorithm that identifies higher-quality, diverse candidates in both synthetic and real-world tasks. Ultimately, our work offers a new, flexible perspective on discrete compositional generation tasks. Code: https://github.com/marcojira/tgm . 1 Introduction The task of scientific discovery centers on discovering novel objects with desirable properties such as antimicrobial resistance or binding affinity. These objects are often discrete and structured such that they can be constructed through a sequence of compositional steps, e.g., proteins as sequences of amino acids, molecules composed of various smaller fragments [ 2 , 51 ] . Due to this compositionality, the number of possible objects grows exponentially with object size and becomes too large to search exhaustively. On top of that, experimentally evaluating potential objects can be too technical or expensive. Nonetheless, in many cases, researchers have access to a proxy reward model that quantitatively approximates the value of the property of interest [ 15 , 19 ] . A large body of work leverages this proxy reward function to narrow the set of objects to a small subset of promising candidates, which can then be tested in a lab [ 13 , 25 , 54 , 1 ] . An alternative approach that has demonstrated growing popularity and effectiveness is to pose this problem as an RL task [ 16 , 14 , 7 ] . In what we will denominate discrete compositional generation (DCG), the object construction process is viewed as a Markov decision process (MDP) where terminal states correspond to complete objects and yield a reward corresponding to the evaluation of that object by the proxy reward function. Then, the learned policy can be used as a generative model, ideally sampling promising candidates. Among these approaches, generative flow networks (GFNs) have shown considerable success in various domains [ 5 , 32 , 56 , 12 ] . GFNs learn an amortized policy and, at optimality, sample objects with probability proportional to the exponential of their reward ( 2 ). Tight connections have been established between GFNs and maximum-entropy RL, showing that this objective underpins the diversity of generated samples [ 49 , 41 , 17 ] . While this exact sampling goal is useful in certain applications, e.g., sampling from an intractable Bayesian posterior, it does not necessarily generate the most promising candidates. Intuitively, if we had access to the true reward function from lab evaluations of objects, the standard RL objective, which seeks to maximize reward, could be appropriate. Regularization and the diversity it entails attempt to be robust to the uncertainty of the proxy reward function. In this work, we show an explicit connection between various forms of regularization and robustness to different forms of uncertainty. In particular, we show that regularizing by the Shannon entropy, as is typically done in GFNs, yields an inappropriate reward constraint set that does not contain the proxy reward. Therefore, motivated by the connection between these fields, we introduce alternative operators to address this issue and demonstrate their effectiveness. Our contributions: (1) We propose a compositional model for reward uncertainty in DCG tasks. From this model, we make explicit the connections between regularized RL and reward-robust RL on value functions, allowing us to analyze the reward uncertainty sets associated with various soft operators. (2) Motivated by the robust interpretation of Shannon entropy regularization in RL and the issue it raises for DCG, we introduce a unifying operator called general mellowmax (GM), that smoothly interpolates between the GFN operator and other soft RL operators. We then propose trajectory GM (TGM) as a practical algorithm for this operator. (3) We experimentally show that TGM performs better than GFNs and other soft RL baselines at generating diverse, high-reward objects for both synthetic and real-world sequence design tasks. 2 Background Figure 1 : (Left) A DCG for a protein sequence generation task. (A) Starting from an empty string, amino acids are added sequentially until termination. (B) Then, the full sequence is evaluated by a proxy reward function \Phi(x) whose value is given as reward for the termination action. (C) Over the course of the protein generation, uncertainty accumulates through the \delta_{i} . (D) The true reward depends on \Phi(x) and the accumulated uncertainty . (Right) Illustrated issue with sampling proportional to reward. The much larger reward of the optimal sequence (in green) is drowned out by all the rewards in the combinatorial explosion of longer subsequences. MDPs. An MDP is a tuple ({\mathcal{S}},{\mathcal{A}},\gamma,P,r) where {\mathcal{S}} is a finite state space, {\mathcal{A}} a finite action space, \gamma\in[0,1) a discount factor, P:{\mathcal{S}}\times{\mathcal{A}}\to\Delta_{{\mathcal{S}}} is a transition kernel and r:{\mathcal{S}}\times{\mathcal{A}}\to[-R,R] a bounded reward function. Define \Pi as the set of policy mappings \pi:{\mathcal{S}}\to\Delta_{{\mathcal{A}}} . The goal is to find \pi\in\Pi that maximizes the value function v^{\pi}_{r}(s):=\mathbb{E}^{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid s_{0}=s],\;\forall s\in{\mathcal{S}}. The dependence of the value function on the reward function is made explicit through the subscript r , as clarified in the next paragraph. For any policy \pi\in\Pi , the expected reward is r^{\pi}(s):=\sum_{a\in{\mathcal{A}}}\pi_{s}(a)r(s,a) and the expected transition is P^{\pi}(s,s^{\prime}):=\sum_{a\in{\mathcal{A}}}\pi_{s}(a)P(s,a,s^{\prime}) . The evaluation Bellman operator given by T^{\pi}_{r}v:=r^{\pi}+\gamma P^{\pi}v for all v\in\mathbb{R}^{{\mathcal{S}}} is known to be contracting, admitting v^{\pi}_{r} as a fixed point. Similarly, the optimal value function v^{*}_{r}(s):=\max_{\pi\in\Pi}v^{\pi}_{r}(s) is the fixed point of the optimal Bellman operator T_{r}^{*}v(s):=\max_{\pi\in\Pi}T^{\pi}_{r}v(s),\forall v\in\mathbb{R}^{{\mathcal{S}}},s\in{\mathcal{S}} . An optimal policy can be derived from iterative Bellman updates, which form the building block of RL [ 45 ] . Robust MDPs. The true reward or transition function of an MDP is rarely known in practice. It may be estimated from trajectory data, but a small error on the MDP model can greatly alter policy performance [ 40 ] . The robust MDP setting addresses this issue by assuming that (P,r) is unknown, lying in a given uncertainty set. In our applicative setting of DCG where transitions are fully determined by the performed actions, we reasonably assume that only the reward model is uncertain and r\in{\mathcal{R}} . Then, we aim to maximize performance for the worst-case model, namely: \displaystyle v^{\pi}_{{\mathcal{R}}}(s):=\min_{r\in{\mathcal{R}}}v^{\pi}_{r}(s),\qquad\forall s\in{\mathcal{S}}. To solve this max-min problem, one can resort to robust Bellman operators T^{\pi}_{{\mathcal{R}}}v:=\min_{r\in{\mathcal{R}}}T^{\pi}_{r}v and T^{*}_{{\mathcal{R}}}v:=\max_{\pi\in\Pi}T^{\pi}_{{\mathcal{R}}}v,\quad\forall v\in\mathbb{R}^{{\mathcal{S}}} . Indeed, both are contracting and admit the robust value function v^{\pi}_{{\mathcal{R}}} and the optimal robust value function v^{*}_{{\mathcal{R}}}

Read the complete text