How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset

Sunil Mohan, Theofanis Karaletsos · 2025

Read How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset by Sunil Mohan, Theofanis Karaletsos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06418.txt.

Introduction

Sunil Mohan and Theofanis Karaletsos build a test of whether language models truly understand how drugs work. Drug development and personalized medicine both need models that recall mechanism facts and reason with them in new situations. A drug's mechanism is a chain of interactions running from the drug to the disease it targets.

Their dataset probes stored knowledge and fresh reasoning side by side, using counterfactual twists the models are unlikely to have memorized. One leading small model matched or beat bigger rivals, and a compact reasoning model came close behind. Reasoning with knowledge supplied in the prompt proved easier than recalling it, and twists inside the chain were hardest.

Catalog record

CREATORSunil Mohan, Theofanis Karaletsos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06418

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset Sunil Mohan, Theofanis Karaletsos Abstract Two scientific fields showing increasing interest in pre-trained large language models (LLMs) are drug development / repurposing, and personalized medicine. For both, LLMs have to demonstrate factual knowledge as well as a deep understanding of drug mechanisms, so they can recall and reason about relevant knowledge in novel situations. Drug mechanisms of action are described as a series of interactions between biomedical entities, which interlink into one or more chains directed from the drug to the targeted disease. Composing the effects of the interactions in a candidate chain leads to an inference about whether the drug might be useful or not for that disease. We introduce a dataset that evaluates LLMs on both factual knowledge of known mechanisms, and their ability to reason about them under novel situations, presented as counterfactuals that the models are unlikely to have seen during training. Using this dataset, we show that o4-mini outperforms the 4o, o3, and o3-mini models from OpenAI, and the recent small Qwen3-4B-thinking model closely matches o4-mini’s performance, even outperforming it in some cases. We demonstrate that the open world setting for reasoning tasks, which requires the model to recall relevant knowledge, is more challenging than the closed world setting where the needed factual knowledge is provided. We also show that counterfactuals affecting internal links in the reasoning chain present a much harder task than those affecting a link from the drug mentioned in the prompt. 1 1 1 This is a slightly expanded version of our paper How Well Does ChatGPT Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset , published in FLLM 2025 ( 15 ) . \hypersetup colorlinks=true, linkcolor=black, urlcolor=blue, citecolor=gray, \organization CZI {}^{\text{a}} Chan Zuckerberg Initative, Redwood City, CA 1 Introduction Scientific research requires analysing novel situations based on deep knowledge and understanding of the domain. For general purpose pre-trained large language models (LLMs) to be useful tools in such endeavors, they need to be able to bring both factual knowledge and a domain-specific reasoning ability to the tasks. There have been several efforts to investigate and chart the reasoning ability of LLMs, both from a task-specific perspective as well as understanding the core capabilities ( 24 ; 30 ; 16 ) . Researchers have observed that LLMs tend to perform better at recalling factual information than on reasoning (e.g. ( 11 ) ). Researchers have also recognized the importance of using counterfactuals (e.g. ( 27 ) ) and other variants of known problems ( 23 ) , which describe scenarios the model is unlikely to have seen during training, for obtaining more robust and generalizable metrics of LLM reasoning. As LLMs show improving capabilities, they are increasingly being investigated as tools in various research and clinical tasks in medicine, e.g. clinical decision support ( 2 ) , personalized medicine ( 8 ) , and drug development and repurposing ( 1 ; 10 ) . We focus on these fields where, for LLMs to be useful, they need to be able to recall factual knowledge about the interactions of drugs and diseases in the body, and also exhibit a deep enough understanding of the language these are described in to be able to reason about them, especially under novel situations. We developed the Drug Mechanisms Counterfactuals Dataset to investigate this capability. DrugBank defines a Mechanism of Action 2 2 2 https://dev.drugbank.com/guides/terms/mechanism-of-action (MoA) as: “the specific chemical and/or physical effects, activities, processes, etc. that a particular drug either elicits or participates in so as to fulfill one or more medical indications.” In pharmaceutical databases like DrugBank, MoAs are presented as descriptive text as part of that drug’s information 3 3 3 https://go.drugbank.com/drugs/DB0104#mechanism-of-action describes the MoA for the drug Abacavir in DrugBank. . These describe a series of interactions between biomedical entities like chemicals, genes, proteins and biological processes. Taken together, they form one or more interlinked chains of effects and relationships that counteract the corresponding effects or causes (if the cause is internal) of the disease that drug is an indication for (see fig. 1 for an example). One can take any such set of interactions, check to see if it forms an interlinked path between a drug and a disease, compose them into an inference chain, and reason about whether that chain forms a potential MoA where the drug affects the disease in a therapeutic manner, or exacerbates it. This is the representation that the Drug Mechanisms Counterfactuals Dataset is based on. The counterfactuals are derived by altering links in such inference chains, or initiating new ones, and the LLM is asked about the effect of that change. In the rest of the paper, we introduce the dataset, and use it to evaluate some recent OpenAI LLMs. We show that the reasoning models o3, o3-mini and o4-mini outperform GPT-4o at these tasks, with o4-mini showing the best results despite being a smaller model than o3. We test our queries in both the closed world setting, where the needed factual interactions and MoA knowledge is included in the prompt, and the open world setting where the model is expected to recall all relevant knowledge for the task. Results show that open world is the more challenging setting. Our evaluation also reveals that ‘deep’ counterfactuals that are based on internal links in the MoA inference chain present a much harder problem than ‘surface’ counterfactuals that are based on a direct interaction with the drug mentioned in the prompt. This paper is a slightly expanded version of our FLLM 2025 paper ( 15 ) , in that it includes some additional figures, and also an evaluation of the Qwen3-4B Thinking LLM. The Drug Mechanisms Counterfactuals Dataset and evaluation framework is available for download 4 4 4 https://github.com/czi-ai/DrugMechCounterfactuals . 2 Drug Mechanisms and Counterfactuals 2.1 Drug Mechanisms of Action (MoA) Drug:Abacavir | increases abundance of | Chemical:carbovir triphosphate Chemical:carbovir triphosphate | decreases activity of \ | Protein:HIV-1 reverse transcriptase Protein:HIV-1 reverse transcriptase | participates in \ | BiologicalProcess:Viral DNA replication BiologicalProcess:Viral DNA replication | causes | Disease:HIV infection Figure 1 : Two representations of the MoA for Abacavir on HIV Infection. The Drug Mechanisms Counterfactuals dataset is derived from DrugMechDB ( 9 ) , a database of graphical representations of the Mechanisms of Action of 4,664 Drug-Disease pairs, manually curated ‘‘from free-text descriptions found on DrugBank, Wikipedia and within literature’’. The curators encoded 14 entity types following specific ontologies 5 5 5 https://sulab.github.io/DrugMechDB#curation , and 66 interaction relation (edge) labels, which don’t have formal definitions. Each interaction is directed from a source to a target entity. Its label describes how the source relates to (e.g. “located in”) or affects (e.g. direction-of-effect like “increases abundance of”, “negatively regulates”) the target entity. DrugMechDB curators followed the convention that each MoA is a directed acyclic graph (DAG), and the Drug entity serves as the single root-node (no incoming edges) and the Disease entity as the single sink-node (no outgoing edges), such that all paths between the two are directed from the Drug to the Disease. An example is shown in fig. 1 . 2.2 Evaluating Consistency between two MoA Graphs Some of our experiments ask

Read the complete text