The Jungle of Generative Drug Discovery: Traps, Treasures, and Ways Out
Rıza Özçelik, Francesca Grisoni* · 2025
Read The Jungle of Generative Drug Discovery: Traps, Treasures, and Ways Out by Rıza Özçelik, Francesca Grisoni* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05457.txt.
Introduction
Riza Ozcelik and Francesca Grisoni wrote a critical study of how new drugs are made by computer models. Their work, tied to a twenty twenty-five paper on the traps in this field, asks a plain question. How should we judge the molecules that a model dreams up?
The pair trained language-style models for chemistry and looked at close to one billion designs. They found that the size of the set of designs can skew the score and make one model look better than it is. Some common scores, like how new or how alike the designs are, can also mislead. They propose new scores and rules for fair tests.
Catalog record
| CREATOR | Rıza Özçelik, Francesca Grisoni* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05457
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Jungle of Generative Drug Discovery: Traps, Treasures, and Ways Out Rıza Özçelik, Francesca Grisoni* Abstract “How to evaluate de novo designs proposed by a generative model?” Despite the transformative potential of generative deep learning in drug discovery, this seemingly simple question has no clear answer. The absence of standardized guidelines challenges both the benchmarking of generative approaches and the selection of molecules for prospective studies. In this work, we take a fresh – critical and constructive – perspective on de novo design evaluation. We systematically investigate widely used evaluation metrics and expose key pitfalls (‘traps’) that were previously overlooked. In addition, we identify tools (‘treasures’) and strategies (‘ways out’) to navigate the complex ‘jungle’ of generative drug discovery, and strengthen the connections between the molecular and deep learning fields along the way. Our systematic and large-scale results are expected to provide a new lens for evaluating the de novo designs proposed by generative deep learning approaches. Discovering new therapeutics is an adventure as old as human civilization. However, finding new drug molecules is more resource-intensive today than ever [ wouters2020estimated , dimasi2016innovation ] . A key challenge lies in the vastness of the ‘chemical universe’ – estimated to contain more than 10^{60} drug-like molecules – where compounds with desirable biological properties are exceedingly rare [ bohacek1996art ] . Artificial intelligence (AI) has emerged as a transformative technology for drug discovery, to help find the ‘needle in the haystack’. By supporting virtual screening [ stokes2020deep , liu2023deep , wong2024discovery ] and de novo molecule design [ godinez2022design , wan2022deep , moret2023leveraging , li2022generative , isigkeit2024automated , xia2024target ] , AI can narrow down the chemical universe, and it is nowadays widely adopted in academia and industry [ bian2021generative , gangwal2024unlocking , cheng2021molecular , volkamer2023machine , catacutan2024machine ] . Generative deep learning has garnered particular attention for drug discovery. Powered by deep neural networks, these models can learn how to generate molecules with desired properties on-demand, and have already demonstrated success in prospective studies [ yuan2017chemical , merk2018novo , grisoni2021combining , godinez2022design , ballarotto2023novo ] . Generative drug discovery generally involves three stages: train , generate , and evaluate . After almost a decade from its initial introduction [ segler2018generating , gomez2018automatic ] , prolific research has standardized many aspects of model training [ bian2021generative , sousa2021generative , gangwal2024unlocking , cheng2021molecular , grisoni2023chemical , ozccelik2024chemical ] and molecule generation [ gupta2018generative , moret2023leveraging , ozccelik2024chemical , sousa2021generative ] . However, the third stage – evaluation – remains relatively underinvestigated, with choices left to the single practitioners. The evaluation of molecular designs (e.g., in terms of their overall quality, relevance, and ultimately, ranking) holds a crucial role. First, selecting the best candidates from thousands of designs determines the success or failure of follow-up experiments. Second, robust evaluation of the generated molecules is essential to monitor progress in the field, and to compare different approaches. Yes, despite notable efforts to standardize model evaluation [ polykovskiy2020moses , brown2019guacamol , arus2019exploring , nie2024durian ] , no consensus within the community has been reached [ renz2019failure , bender2022evaluation , sousa2021generative , martinelli2022generative ] . Here, we dive into design evaluation, with a critical and constructive perspective. We conduct a systematic study evaluating around 10 9 de novo designs across three state-of-the-art deep learning architectures. Our results uncover a previously overlooked pitfall: the size of a design library can systematically bias the evaluation, and at times even falsify the scientific findings by overshadowing molecular quality. We then draw new connections between drug discovery and deep learning to facilitate molecule selection for prospective studies. Finally, we leverage the tools we develop to dig deeper into the generation stage, and expose unique characteristics of molecule design, compared to other applications of deep learning. To capture the complexity on the landscape of generative drug discovery, we introduce the analogy of a ‘jungle’, where we identify criticalities (‘traps’), useful tools (‘treasures’), and guidelines (‘ways out’). These findings aim to redefine how generative models are evaluated in drug discovery and provide a roadmap for selecting molecules with higher confidence for real-world applications. Results and discussion While many approaches exist to design molecules de novo, ‘chemical language’ models (CLMs) have been among the most succesful [ grisoni2023chemical , skinnider2021chemical , flam2022language ] . CLMs are trained to generate molecules in the form of molecular strings, such as Simplified Molecular Input Line Entry Systems (SMILES) [ weininger1988smiles ] and Self-referencing embedded strings (SELFIES) [ krenn2022selfies ] . Since CLMs can be trained and used to generate molecules in a time-efficient manner [ ozccelik2024chemical ] , and excel at capturing complex molecular properties [ flam2022language ] , they constitute an ideal choice for a large-scale analysis like ours. Here, we use three deep CLM architectures: (i) Recurrent neural networks with long short-term memory cells, (LSTM) [ hochreiter1997long , segler2018generating ] , which learn from and generate chemical sequences one symbol (‘token’) at a time; (ii) Generative Pretrained Transformers (GPT) [ radford2019language , bagal2021molgpt ] , which, via the attention mechanism [ bahdanau2015neural ] learn all pair relationships between input tokens; and (iii) Structured State-Space Sequence models (S4) [ gu2022efficiently , ozccelik2024chemical ] , which were recently introduced, and learn from entire sequence at once, while generating token-by-token. After pre-training the CLMs on 1.5M canonical SMILES strings from ChEMBLv33 [ gaulton2017chembl ] , they were fine-tuned on bioactive molecules of three macromolecular targets [ sun2017excape ] : (a) Dopamine Receptor D3 (DRD3), (b) Peptidyl-prolyl cis/trans Isomerase NIMA-interacting 1 (PIN1), and Vitamine D Receptor (VDR). The fine-tuning was repeated five times for each target, with a different random set of 320 bioactive molecules each. From each fine-tuned CLM, we sampled 1,000,000 molecules in the form of SMILES strings (using multinomial sampling, equation ( 1 )). The size of de novo design libraries can fool you “How many designs should I generate?” Every de novo design study faces this question. Although an arbitrary number of SMILES strings could be generated, 1000 and 10,000 designs are typical choices for model evaluation [ polykovskiy2020moses ] . However, since generative molecule design involves sampling from a learned probability distribution, a minimum library size may be required to ensure a representative overview of the model’s output. Here, we aim to shed light on (a) what library size is sufficient to comprehensively evaluate the quality of designs, and (b) whether the number of chosen designs affects the evaluation outcomes. To this end, we evaluated the following aspects in increasing number of de novo designs: • Similarity between de novo designs and fine-tuning sets . We measured the Frechét ChemNet Distance (FCD) [ preuer2018frechet ] , which captures the biological and chemical similarity of two molecular sets (through the ChemNet [ mayr2018large ] model). Moreover, we computed the Frechét distance [ frechet1957distance ] on five molecular descriptors (Frechét Descriptor Distance, FDD, see Methods). The lower the FDD, the closer the sets of molecules are in terms of the distribution of their physicochemical properties. As controls, we computed the FCD and FDD values of 128 held-out actives and 1280 inactives per each data split, with the hypothesis that active molecules should be closer to the fine-tuning set than the inactive ones. • Internal diversity of