Generalist Large Language Models for Molecular Property Prediction: Distilling Knowledge from Specialist Models
Khiem Le, Sreejata Dey, Marcos Martínez Galindo, Vanessa Lopez, Ting Hua, Nitesh V. Chawla, Hoang Thanh Lam · 2026
Read Generalist Large Language Models for Molecular Property Prediction: Distilling Knowledge from Specialist Models by Khiem Le, Sreejata Dey, Marcos Martínez Galindo, Vanessa Lopez, Ting Hua, Nitesh V. Chawla, Hoang Thanh Lam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12344.txt.
Introduction
Le and co-authors teach language models molecular property prediction from decision trees. LLMs promise generalist skill here, yet their raw scores fall short of practical use. TreeKD trains a specialist decision tree per property on functional group features. The tree's rule is then turned into words and placed into prompts for LLM training. A random forest variant adds rule-consistency scaling at test time across prompts. On 22 TDC tasks with two LLMs, the method lifts accuracy by a wide margin.
Catalog record
| CREATOR | Khiem Le, Sreejata Dey, Marcos Martínez Galindo, Vanessa Lopez, Ting Hua, Nitesh V. Chawla, Hoang Thanh Lam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45607 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalist Large Language Models for Molecular Property Prediction: Distilling Knowledge from Specialist Models Khiem Le, Sreejata Dey, Marcos Martínez Galindo, Vanessa Lopez, Ting Hua, Nitesh V. Chawla, Hoang Thanh Lam Abstract Molecular Property Prediction (MPP) is a central task in drug discovery. While Large Language Models (LLMs) show promise as generalist models for MPP, their current performance remains below the threshold for practical adoption. We propose TreeKD, a novel knowledge distillation method that transfers complementary knowledge from tree-based specialist models into LLMs. Our approach trains specialist decision trees on functional group features, then verbalizes their learned predictive rules as natural language to enable rule-augmented context learning. This enables LLMs to leverage structural insights that are difficult to extract from SMILES strings alone. We further introduce rule-consistency, a test-time scaling technique inspired by bagging that ensembles predictions across diverse rules from a Random Forest. Experiments on 22 ADMET properties from the TDC benchmark demonstrate that TreeKD substantially improves LLM performance, narrowing the gap with SOTA specialist models and advancing toward practical generalist models for molecular property prediction. 1 Introduction Molecular Property Prediction (MPP) plays a crucial role in the Drug Discovery Wu et al. 2018 ; Liyaqat et al. 2025 , aiming to predict the quantitative characteristics of drug candidates such as ADMET properties (Absorption, Distribution, Metabolism, Excretion, and Toxicity) Ferreira and Andricopulo 2019 . Accurate prediction enables reliable screening of potential drug candidates at early stages, lowering experimental costs and optimizing efficiency. While traditional specialist models achieve strong performance on individual properties, they lack the ability to interact via natural language, provide explanations, or engage in scientific conversations Chaves et al. 2024 ; Wang et al. 2025 ; Xia et al. 2025 . Recently, the success of Large Language Models (LLMs) Achiam et al. 2023 ; Comanici et al. 2025 across diverse text-based tasks Hendrycks et al. 2021 ; Suzgun et al. 2023 has motivated increasing efforts to develop LLM-based generalist models that can predict multiple properties while offering these interactive capabilities Zhang et al. 2025 . However, despite training LLMs on the textual modality of molecules (SMILES notation), their current performance on MPP remains far below that of specialist models, limiting their practical adoption for MPP Guo et al. 2023 ; Zhong et al. 2024 . A key reason for this performance gap is that LLMs struggle to recognize functional groups (FGs), the substructures within a molecule that largely determine its properties Andrews et al. 1984 , when provided only SMILES strings Castro Nascimento and Pimentel 2023 ; Runcie et al. 2025 . To address this limitation, several methods have been proposed that integrate additional structural modalities, such as 2D/3D graphs and fingerprints, into LLMs Liu et al. 2023 ; Li et al. 2024 ; Le et al. 2024 ; Cao et al. 2025 . However, these methods require a large-scale dataset, a complex adaptor, and an extra pretraining phase to align these modalities with LLMs Li et al. 2023 . This alignment process is resource-intensive and increases the risk of data leakage. Furthermore, LLMs have been shown to have difficulty in capturing the concept of property cliffs Xia et al. 2023 , which means situations where small changes in the structure of a molecule lead to significant changes in its properties Maggiora 2006 ; Stumpfe and Bajorath 2012 . In contrast, traditional tree-based specialist models have been proven to learn differently from crafted features and are better at capturing this concept Xia et al. 2023 . In this work, we propose TreeKD, a novel knowledge distillation method that improves LLM performance on MPP by transferring complementary knowledge from tree-based specialist models. Instead of forcing LLMs to recognize FGs implicitly, we extract and provide all FGs found in each molecule (from a library of 40K), reducing the semantic gap between SMILES strings and natural language Wellawatte and Schwaller 2025 . Building on the extracted features, we train a specialist decision tree for each property. Decision trees learn from crafted features differently than LLMs and are more sensitive to property cliffs, with their knowledge encoded in learned predictive rules. We then verbalize these rules as natural language and integrate them into the training context, enabling knowledge transfer from decision trees to LLMs. To further enhance performance, we introduce rule-consistency, a test-time scaling technique inspired by bootstrap aggregating Breiman 1996 . Instead of using a single decision tree, we train a Random Forest of N decision trees for each property and incorporate one randomly selected predictive rule into each training example. The randomness mirrors the inherent diversity in ensemble methods, preventing the LLM from over-relying on any single rule. At inference, rule-consistency ensembles N predictions from N different prompts, each augmented with a different predictive rule from the Random Forest, unlike self-consistency Wang et al. 2023 which ensembles predictions from the same prompt. Our main contributions are summarized as follows: 1) We propose TreeKD, a knowledge distillation approach that transfers complementary knowledge from tree-based specialist models to LLMs by verbalizing decision tree rules and integrating them into the training context. 2) We introduce rule-consistency, a test-time scaling technique that leverages Random Forest diversity to ensemble predictions across different rules, providing more robust predictions than sampling-based approaches. 3) Through extensive experiments on 22 ADMET properties from the TDC benchmark using Gemma-2-2B and Granite-3.3-2B, we demonstrate that TreeKD significantly improves over baseline LLMs on MPP, achieving competitive performance with specialist models while retaining the interpretability and natural language interaction capabilities unique to LLM-based methods. Figure 1: An overview of our proposed method TreeKD, consisting of the following four steps: ① Extract FGs, ② Construct Specialist Models, ③ Distill Knowledge from Specialist Models, and ④ Test-Time Scaling. 2 Related Work Generalist LLMs for MPP . A major practice in developing generalist models for MPP based on LLMs is curating a massive instruction dataset to train LLMs on Zhang et al. 2025 . Fang et al. 2024 introduce Mol-Instructions, a dataset of 2M samples stemming from various sources such as PubChem and MoleculeNet, and show that training LLMs on this dataset improves their performance. Similarly, Yu et al. 2024 create a larger dataset, SMolInstruct, which contains 3.3M samples, and establish LlaSMol models by fine-tuning LLMs on it. Beyond straightforward training, Liu et al. 2024 ; Wang et al. 2025 aim to unleash the In-context Learning (ICL) Brown et al. 2020 power of LLMs during inference by employing Meta-ICL Min et al. 2022 during training. In addition, Liu et al. 2023 ; Li et al. 2024 ; Le et al. 2024 ; Cao et al. 2025 propose methods to assist LLMs in perceiving structural information of molecules by integrating additional structural modalities, such as 2D/3D graphs and fingerprints, though these methods are resource-intensive and exacerbate data leakage risk. Our goal is to boost LLMs’ performance by leveraging knowledge from tree-based specialist models. This approach is complementary to existing methods and does not require additional modalities or complex alignment procedures. Knowledge Distillation with LLMs . Knowledge Distillation (KD) Hinton et al. 2015 is a widely used method to train a neural network model (student) with the knowledge of another neural network model (teacher). Originally, this is done by minimizing the Kullback–Leibler (KL) divergence between their logit distributions.