Unified Molecule Generation and Property Prediction
Adam Izdebski, Jan Olszewski, Pankhil Gawade, Krzysztof Koras, Serra Korkmaz, Valentin Rauscher, Jakub M. Tomczak, Ewa Szczurek · 2025
Read Unified Molecule Generation and Property Prediction by Adam Izdebski, Jan Olszewski, Pankhil Gawade, Krzysztof Koras, Serra Korkmaz, Valentin Rauscher, Jakub M. Tomczak, Ewa Szczurek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16559.txt.
Introduction
Adam Izdebski and co-authors present Hyformer in a 2025 paper, one model that both makes new molecules and predicts their traits. The plan rests on learning how samples and their traits are spread out together, so a single model can do both jobs. The pair should help each other in ways a single-job model cannot match.
Such joint models are hard to train, in both their build and their tuning. Hyformer is a transformer that swaps back and forth between the two jobs as it reads. It is trained up front on both jobs at once. Tests show gains in making samples that meet set traits, in calls on traits for cases outside the training spread, and in representation learning. A drug design case, finding new antimicrobial peptides, shows the gain in use.
Catalog record
| CREATOR | Adam Izdebski, Jan Olszewski, Pankhil Gawade, Krzysztof Koras, Serra Korkmaz, Valentin Rauscher, Jakub M. Tomczak, Ewa Szczurek |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16559
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unified Molecule Generation and Property Prediction Adam Izdebski, Jan Olszewski, Pankhil Gawade, Krzysztof Koras, Serra Korkmaz, Valentin Rauscher, Jakub M. Tomczak, Ewa Szczurek Abstract Modeling the joint distribution of the data samples and their properties allows to construct a single model for both data generation and property prediction, with synergistic capabilities reaching beyond purely generative or predictive models. However, training joint models presents daunting architectural and optimization challenges. Here, we propose Hyformer , a transformer-based joint model that successfully blends the generative and predictive functionalities, using an alternating attention mask together with a unified pre-training scheme. We show that Hyformer rivals other joint models, as well as state-of-the-art molecule generation and property prediction models. Additionally, we show the benefits of joint modeling in downstream tasks of molecular representation learning, hit identification and antimicrobial peptide design. 1 Introduction Developing models that simultaneously excel in both generative and predictive tasks is a long-standing challenge in machine learning [ 4 , 26 , 32 ] . Joint models, which unify these tasks, offer synergistic benefits, including learning better representations that are predictive of high-level molecular features and improved predictive robustness towards unseen, e.g., newly generated and OOD data [ 42 , 21 , 8 , 56 ] . These benefits are crucial for applications such as drug design, where success depends on balancing the generation of novel molecules from the unexplored regions of the chemical space, coupled with robust property prediction extrapolating towards the generated molecules [ 22 , 51 , 59 ] . However, molecular generative and predictive tasks are approached primarily in separation, with specialized models developed for each of the tasks [ 2 , 25 , 14 , 69 , 65 ] . Most likely, this is due to the fact that training joint models presents architectural and optimization challenges. First, combining a generative and a predictive part into a single model may over-regularize both parts [ 32 ] . Moreover, optimizing the joint loss biases the gradient flow of the joint model towards the generative task [ 42 ] . In practice, both of the effects may cause joint models to underperform on generative or predictive tasks. Recently, transformer-based models achieved state-of-the art performance in both molecule generation [ 2 ] and property prediction [ 69 ] , indicating the strong potential of transformers for joint molecule modeling. Hence, a natural question arises, whether we can develop a transformer-based joint model with both a high generative and predictive performance, at the same time maintaining the synergistic benefits of joint learning. To address these challenges, we introduce Hyformer , a joint model that combines an autoregressive transformer decoder with a bidirectional transformer encoder in a single model with shared parameters. Upon training, we alternate between using the model as a transformer decoder and as a transformer encoder, with either causal or bidirectional self-attention, alleviating problems typical for the training of joint models. Finally, we evaluate the generative and predictive performance of \HyF across a variety of molecular tasks [ 64 , 7 , 10 ] . To summarize, our contributions are: 1. We propose a novel joint model that unifies the generative and the predictive task in a single set of parameters. 2. We demonstrate that Hyformer rivals the generative and predictive performance of state-of-the-art purely generative or purely predictive models. 3. We show the synergistic benefits of joint modeling with \HyF for molecular representation learning, using a probing procedure based on MoleculeNet benchmark [ 64 ] . 4. We successfully apply \HyF to a real-world use-case of conditionally generating novel antimicrobial peptides. 2 Related Work Previous molecule modeling approaches are commonly either optimized for the task of molecule generation, or molecule prediction, with only few joint models. Molecule Generation Existing approaches for molecule generation can be divided into sequence or graph-based models. Sequence-based models predominantly represent molecules as SMILES [ 63 ] or SELFIES [ 30 ] and process the tokenized sequences [ 28 ] using recurrent and language models [ 49 , 15 , 2 ] . Alternatively, graph-based models represent molecules as molecular graphs, modeled with variational autoencoders [ 36 , 27 , 40 , 23 ] , normalizing flows [ 39 ] , energy-based models [ 35 ] and transformers [ 17 ] . Molecular Property Prediction Similarly to molecule generation, molecular property prediction methods leverage distinct molecular representations. Methods based on pre-trained language models predominantly work with SMILES [ 61 , 13 , 25 , 53 ] . Other predictive approaches represent molecules as molecular graphs modeled with graph neural networks [ 33 , 62 ] . Finally, recent methods leverage the three-dimensional spatial structure of a molecule, either using graph neural networks [ 14 ] or transformers [ 69 ] . Finally, Yang et al. 2019 , Fabian et al. 2020 , Stokes et al. 2020 incorporate pre-computed physicochemical descriptors of molecules into training. Joint Models for Molecules Early joint models for molecules were based on variational autoencoders (VAEs), with predictors trained on latent space [ 20 , 40 ] . We are not aware of any transformer-based models that would be explicitly jointly optimized for both generation and prediction tasks. One transformer model for molecules that could be considered joint is Regression Transformer [ 6 ] , which abstracts away prediction as a conditional sequence generation task, yet without the ability to unconditionally generate molecules. Another, Graph2Seq, is a graph-based encoder-decoder transformer, trained separately as a generative or as a predictive model, but evaluated on both molecule generation and property prediction [ 17 ] . Therefore, the question of whether the transformer architecture can be used to implement a joint model with the high generative performance of a pre-trained generative model and the predictive performance of a fine-tuned one, and which benefits from synergistic effects such as improved data representations, remains open. 3 Background Problem Formulation The aim of joint modeling is to learn the joint distribution of the data and its properties p(\x,y) , i.e., to identify a model that at the same time generates new data and predicts its properties. We are given access to a labeled dataset {\D}=\{(\x_{n},y_{n})\}_{n=1}^{N} , sampled from the joint data distribution p(\x,y) , often accompanied with an unlabeled dataset {\D}_{U}=\{\x_{n}\}_{n=1}^{N_{U}} , sampled from p(\x) , with N_{U}\gg N . Here, examples \x can be thought of as molecules and labels y as molecular properties such as drug-likeliness or toxicity. In the general formulation of Lasserre et al. 2006 , joint modeling aims to learn the joint distribution p(\x,y) by defining a joint model p_{\theta,\phi}(\x,y) that factorizes into a generative model p_{\theta}(\x) and a predictive model p_{\phi}(y\mid\x) such that p_{\theta,\phi}(\x,y)=p_{\phi}(y\mid\x)p_{\theta}(\x), (1) where \theta denotes the parameters of the generative model, and \phi denotes the parameters of the predictive model. Training of the joint model is equivalent to minimizing the negative log-likelihood, i.e., the joint loss \ell_{\lambda}(\theta,\phi)=-\E_{(\x,y)\sim p(\x,y)}[\ln p_{\theta}(\x)+\lambda\ln p_{\phi}(y\mid{\x})], (2) where \lambda\in\mathbb{R} weights the predictive and the generative parts. Choosing the extent to which parameters \theta and \phi are shared and the the way the joint loss is implemented and optimized, is crucial for obtaining a model with both a high generative and predictive performance, at the same time maintaining the synergistic benefits of joint learning [