Diverse Mini-Batch Selection in Reinforcement Learning for Efficient Chemical Exploration in de novo Drug Design
Hampus Gummesson Svensson, Ola Engkvist, Jon Paul Janet, Christian Tyrchan, Morteza Haghir Chehreghani · 2025
Read Diverse Mini-Batch Selection in Reinforcement Learning for Efficient Chemical Exploration in de novo Drug Design by Hampus Gummesson Svensson, Ola Engkvist, Jon Paul Janet, Christian Tyrchan, Morteza Haghir Chehreghani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21158.txt.
Introduction
Hampus Gummesson Svensson and co-authors make molecule-generating software explore more widely by feeding each training step a deliberately varied diet of examples, in a 2025 paper. In de novo drug design, a reinforcement learning agent proposes new molecules and scores them, but scoring is slow and costly, and the agent tends to collapse onto a few similar answers. Learning from diverse batches pushes back against that collapse.
The team introduces mini-batch diversification: each small batch used for an update is chosen to be varied on purpose. Across their experiments in chemical exploration, the method raised the diversity of the molecules found while keeping their quality high. That pairing matters in drug hunting, where several distinct good candidates give medicinal chemists real choices. The authors suggest the same trick can steady exploration in other costly reinforcement learning problems.
Catalog record
| CREATOR | Hampus Gummesson Svensson, Ola Engkvist, Jon Paul Janet, Christian Tyrchan, Morteza Haghir Chehreghani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-21158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diverse Mini-Batch Selection in Reinforcement Learning for Efficient Chemical Exploration in de novo Drug Design Hampus Gummesson Svensson, Ola Engkvist, Jon Paul Janet, Christian Tyrchan, Morteza Haghir Chehreghani Abstract In many real-world applications, evaluating the goodness of instances is often costly and time-consuming, e.g., human feedback and physics simulations, in contrast to proposing new instances. In particular, this is even more critical in reinforcement learning, as new interactions with the environment (i.e., new instances) need to be evaluated to provide a reward signal to learn from. As sufficient exploration is crucial, learning from a diverse mini-batch can have a large impact and help mitigate mode collapse. In this paper, we introduce diverse mini-batch selection for reinforcement learning and propose to use determinantal point processes for this task. We study this framework in the context of a real-world problem, namely drug discovery. We experimentally study how our proposed framework can improve the effectiveness of chemical exploration in de novo drug design, where finding diverse and high-quality solutions is essential. We conduct a comprehensive evaluation with three well-established molecular generation oracles over numerous generative steps. Our experiments conclude that our diverse mini-batch selection framework can substantially improve the diversity of the solutions, while still obtaining solutions of high quality. In drug discovery, such outcome can potentially lead to fulfilling unmet medication needs faster. 1 Introduction Many practical machine learning applications require quality assessment of data and experiences, e.g., via human expert evaluation, computer simulations, field testing, and laboratory experimentation. These assessment methods are frequently resource-intensive, demanding significant time and financial investment. Consequently, the volume of data that can undergo thorough evaluation is often constrained by budgetary limitations. In this paper, we consider this problem in online reinforcement learning, where the training instances are provided solely from the agent’s interaction with the environment. Reinforcement learning (RL) relies on learning from interactions with the environment. Many successful algorithms in environments, e.g., ( Schulman et al. 2017 ; Mnih et al. 2016 ) , run many copies of the environment in parallel to synchronously or asynchronously learn from numerous interactions. For synchronous on-policy algorithms, the experiences are accumulated to compute an average loss to update the agent’s policy. This is also the case in many applications of RL, e.g., de novo drug design ( Olivecrona et al. 2017 ) and text-to-image generation ( Fan et al. 2023 ) . However, in real-world applications, this is necessarily not possible since not all interactions with the environment can be assessed to provide a reward signal. For instance, in recent years, reinforcement learning-based fine-tuning of pre-trained models has shown great success in various applications ( Zhai et al. 2024 ; Fan et al. 2023 ; Olivecrona et al. 2017 ) . In such applications, the goal is often to fine-tune the pre-trained model from human preferences or experiments, while minimizing the number of interactions to be evaluated. To efficiently compute a sufficient solution or find different modes, exploration mechanisms play a vital role in agent performance, especially in tasks with sparse reward or for a reward landscape with a vast number of local optima to explore. One popular approach to enhance exploration in RL is the addition of an exploration bonus to the reward function, known as intrinsic reward ( Burda et al. 2018 ; Badia et al. 2020 ; Seo et al. 2021 ; Tang et al. 2017 ) . Another common approach is maximum entropy RL, where the agent tries to maximize both the reward and entropy simultaneously, i.e., succeeding at a task while still acting as randomly as possible ( Haarnoja et al. 2018 ) . Our work provides a consistent perspective where, while acting diverse, it is important to keep high quality among the interactions used to update the policy. This is accomplished by considering mini-batch diversification in reinforcement learning, where a large number of interactions obtained from running copies of the environment in parallel are summarized in a smaller, diverse set of interactions used for updating the policy. This can provide an effective way to impose additional exploration in the learning process when there is a limited evaluation budget. We argue that providing a diverse mini-batch of interactions makes the agent’s exploration more effective and increases the diversity of the forthcoming interactions. Therefore, we introduce a diverse mini-batch selection framework for reinforcement learning and propose to use determinantal point processes (DPP) for this task. DPPs provide an effective framework to sample a diverse set based on specified similarity information. Previous work has proposed a mini-batch diversification scheme based on DPPs for stochastic gradient descent and shown its effectiveness ( Zhang, Mandt, and Kjellström 2017 ; Huang, Da Xu, and Oppermann 2019 ) . To the best of our knowledge, this is the first effort to study the effects of diverse mini-batch selection in (online) reinforcement learning. Particularly, we study the effects of diverse mini-batch selection in a real-world problem with large social impact, namely the de novo drug design problem. This problem can be formulated as an episodic RL problem, and previous work has shown that this approach provides promising solutions ( Thomas et al. 2022 ; Gao et al. 2022 ) . The reward function is usually complex and has many high-rewarding modes that should be found and subsequently exploited to obtain a diverse set of solutions. Hence, chemical exploration and diversification are of integral importance in drug design. In real-world deployment of this problem, it is often costly and time-consuming to evaluate an instance (i.e., a state-action episode) to obtain a reward. On the other hand, it is usually inexpensive to generate an instance. This makes it a suitable problem to employ diverse mini-batch selection and study its effectiveness. On the other hand, we believe that this methodology can improve RL in general since exploration is a key challenge in RL, and domain-specific information can easily be incorporated. RL , , Evaluation Figure 1: We propose a framework for diverse mini-batch selection in reinforcement learning. The RL agent generates a set of experiences in parallel, e.g., trajectories. A kernel measures the pairwise similarities between trajectories and is used to select a diverse set. The selected set is evaluated and, subsequently, is used to update the RL agent. 2 Background 2.1 De novo Drug Design De novo drug design is the challenge of designing novel drug molecules given a set of predefined constraints. This is a problem where diversity is important since a diverse molecular library increases the likelihood of identifying candidates with unique and favorable pharmacological profiles, thereby enhancing the overall efficiency and success rate of drug development pipelines. Also, it is a problem where real-world applications often use expensive computational simulations and/or human-in-the-loop feedback ( Pitt et al. 2025 ; Sundin et al. 2022 ) . A popular approach for de novo drug design is to use chemical language models to generate string-based representations of molecules ( Arús-Pous et al. 2019 ; Segler et al. 2018 ) . To steer the chemical language model to promising areas of the chemical space, reinforcement learning can be leveraged ( Olivecrona et al. 2017 ) . An action a in this RL problem corresponds to adding one token to the string representation of