Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning

Xuan Li, Zhanke Zhou, Zongze Li, Jiangchao Yao, Yu Rong, Lu Zhang, Bo Han · 2026

Read Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning by Xuan Li, Zhanke Zhou, Zongze Li, Jiangchao Yao, Yu Rong, Lu Zhang, Bo Han (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05900.txt.

Introduction

Li and team present Ref-guided Policy Tuning for order-based chem tuning with language models. Fine-tuning on refs fails thinking, while checked-reward learning gives sparse feedback under likeness limits from poor search. RePO samples picks with thinking paths, trains with checked rewards for trait meeting under likeness limits, and keeps paths as context while training only the answer with guidance. Across benchmarks, RePO beats fine-tuning and checked-reward baselines and spreads better to unseen order styles.

Catalog record

CREATORXuan Li, Zhanke Zhou, Zongze Li, Jiangchao Yao, Yu Rong, Lu Zhang, Bo Han
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91124
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning Xuan Li, Zhanke Zhou, Zongze Li, Jiangchao Yao, Yu Rong, Lu Zhang, Bo Han Abstract Large language models (LLMs) benefit substantially from supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) in reasoning tasks. However, these recipes perform poorly in instruction-based molecular optimization, where each data point typically provides only a single optimized reference molecule and no step-by-step optimization trajectory. We reveal that answer-only SFT on the reference molecules collapses reasoning, and RLVR provides sparse feedback under similarity constraints due to the model’s lack of effective exploration, which slows learning and limits optimization. To encourage the exploration of new molecules while balancing the exploitation of the reference molecules, we introduce Re ference-guided P olicy O ptimization (RePO), an optimization approach that learns from reference molecules without requiring trajectory data. At each update, RePO samples candidate molecules with their intermediate reasoning trajectories from the model and trains the model using verifiable rewards that measure property satisfaction under similarity constraints in an RL manner. Meanwhile, it applies reference guidance by keeping the policy’s intermediate reasoning trajectory as context and training only the answer in a supervised manner. Together, the RL term promotes exploration, while the guidance term mitigates reward sparsity and stabilizes training by grounding outputs to references when many valid molecular edits exist. Across molecular optimization benchmarks, RePO consistently outperforms SFT and RLVR baselines (e.g., GRPO), achieving improvements on the optimization metric (Success Rate × \times Similarity), improving balance across competing objectives, and generalizing better to unseen instruction styles. Our code is publicly available at https://github.com/tmlr-group/RePO . 1 Introduction LLMs have advanced rapidly on reasoning-intensive tasks through supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) ( 17 ; 39 ; 67 ) . Both paradigms allow LLMs to perform deliberative reasoning before generating answers and solve problems requiring multiple reasoning steps ( 65 ; 70 ; 69 ) . However, how these recipes perform in scientific tasks remains under-investigated. In this paper, we study instruction-based molecular optimization , where the model is required to optimize the molecular property while maintaining structural similarity with the original molecule (Fig. 1 ). This task involves competing objectives: improving a target property requires non-trivial structural edits (e.g., adding functional groups), but those edits will reduce structural similarity to the original molecule and can even break chemical validity ( 32 ; 36 ) . The data point in most datasets, however, provides only a single optimized molecule for reference, without a reasoning trajectory. This supervision mismatch raises a key challenge: how to support (i) effective exploration of the constrained chemical space for molecules with desired properties and (ii) answer-level anchoring to the available references to exploit dataset-provided reference molecules. Figure 1: Molecular optimization aims to optimize the given molecule by modifying its components while maintaining the structural similarity of the original molecule after modification. The molecule is presented as SMILES ( 59 ) , a sequence of symbols representing atoms and bonds. Figure 2: Illustration of RePO. The model generates answers via reasoning; reference guidance anchors to the reference conditioned on the reasoning context, while RLVR optimizes the property under similarity constraints. Figure 3: Performance comparison on molecular optimization tasks. Details of molecular properties can be found in Appendix I.1 . Notably, established RLVR/SFT recipes face substantial limitations in this domain ( 64 ; 16 ; 10 ) . We show that, with only reference molecules as supervision, SFT often collapses into deterministic, answer-only outputs and suppresses the multi-step reasoning needed for exploration. When RL starts from such an SFT initialization, RLVR updates often fail to produce multi-step reasoning trajectories and instead preserve the short-response style. Conversely, direct RL from the base model depends on rare answers that both improve the property and satisfy similarity; early successes are usually tiny edits with marginal gains, which drive conservative optimizations, yield sparse learning signals, and limit final performance. To address these limitations, we propose Reference-guided Policy Optimization (RePO), an optimization approach that provides reference guidance while permitting many valid molecular edits (Fig. 3 ). Concretely, each RePO step samples candidate molecules from the model given an instruction, scores them with a verifiable reward (property satisfaction while enforcing similarity), and updates the policy with three terms: (i) an RLVR term that upweights higher-reward candidates and lowers the low-reward ones, (ii) an answer-level reference-guidance term that increases the likelihood of the reference conditioned on the model’s sampled trajectories, and (iii) a KL regularizer that stabilizes updates. Overall, RePO couples reward-driven exploration in the constrained chemical space with answer-level anchoring to the references, without requiring any labeled intermediate trajectories. We evaluate RePO on instruction-based molecular optimization benchmarks, including TOMG-Bench ( 28 ) and MuMOInstruct ( 11 ) . On TOMG-Bench single-objective tasks, RePO achieves the best Success Rate \times Similarity in four out of six tasks, with up to 17.4% success rate improvement over GRPO (Fig. 3 ). On MuMOInstruct multi-objective tasks, RePO better balances competing objectives and maintains its advantage under unseen instruction styles. RePO also generalizes across model families and benefits from increased inference-time compute. We summarize our contributions as follows: • We reveal a supervision mismatch in instruction-based molecular optimization, which makes answer-only SFT collapse multi-step exploration and limit RLVR’s performance (Sec. 3 ). • We propose RePO, a reference-guided policy optimization approach that combines GRPO-style reward-driven exploration with reference guidance conditioned on the model’s trajectories (Sec. 4 ). • We show that RePO improves optimization metrics, generalizes to unseen instruction styles, and benefits from more generation budget during inference (Sec. 5 ). 2 Preliminaries Task formulation. Molecular optimization aims to modify an input molecule to optimize target properties while preserving structural similarity ( 36 ; 34 ) . As illustrated in Fig. 1 , we denote the query as q\!=\!(x,m_{0}) , where x is the textual instruction and m_{0} is the input molecule. The model outputs a response o=[t;\hat{m}] consisting of reasoning tokens t and an optimized molecule \hat{m} . Given the candidate molecule space \mathcal{M} , target-property function F:\mathcal{M}\to\mathbb{R} , similarity function \mathrm{Similarity}(\cdot,\cdot) , and threshold \delta\in[0,1] , the task is formulated as m^{*}=\arg\max_{m\in\mathcal{M}}F(m)\quad\text{s.t.}\quad\mathrm{Similarity}(m,m_{0})\geq\delta. (1) Here m^{*} denotes an optimized molecule that satisfies both property and similarity constraints. We use m for a generic molecule, m_{0} for the input molecule in q . Each dataset instance also provides a reference molecule, denoted by m_{\mathrm{ref}} , which serves as a reference for m^{*} and is used as the guidance target in Sec. 4 . We provide additional discussions in Appendix G.1 . LLM reasoning in molecular optimization. Despite their merits, conventional molecular design approaches struggle to synthesize molecules with precisely tailored properties ( 29 ; 30 ) and often generalize poorly to novel objectives or constraint combinations ( 11 ; 28 ; 66 ) . These limitations motivate us to leverage LLMs’ ability to follow natural-language instructions ( 9 ) for optimizing molecules. In addition, prior work suggests LLMs can capture non-trivial knowledge of molecular properties ( 18 ) . Nevertheless, LLMs still struggle under chemistry-specific constraints. Benchmarks such as SciBench ( 56 ) and ChemBench ( 38 ) report substantial degradation when tasks require validity preservation or property-preference satisfaction. Accordingly, recent GPT-based molecular optimization

Read the complete text