Tracing Pharmacological Knowledge in Large Language Models

Basil Hasan Khwaja, Dylan Chen, Guntas Toor, Anastasiya Kuznetsova · 2026

Read Tracing Pharmacological Knowledge in Large Language Models by Basil Hasan Khwaja, Dylan Chen, Guntas Toor, Anastasiya Kuznetsova (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03407.txt.

Introduction

Basil Hasan Khwaja and three colleagues trace where drug knowledge lives inside a language model. Models perform well on pharmacology tasks, but nobody knew how that knowledge is stored or fetched. Their 2026 study opens up Llama-based biomedical models with two tools. One is activation patching, which swaps internal signals to find what causes an answer. The other is linear probes, which test what each internal signal encodes.

Early layers turn out to matter most for drug-group knowledge. The strongest causal effects come from middle tokens inside a drug group's name, not its final token. Probes tell a matching story: single-token signals are weak, while pooled signals across tokens reach top accuracy, and traces exist already in the embedding layer. Drug meaning, the authors conclude, is spread across tokens rather than filed in one place.

Catalog record

CREATORBasil Hasan Khwaja, Dylan Chen, Guntas Toor, Anastasiya Kuznetsova
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23687
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03407

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tracing Pharmacological Knowledge in Large Language Models Basil Hasan Khwaja, Dylan Chen, Guntas Toor, Anastasiya Kuznetsova Abstract Large language models (LLMs) have shown strong empirical performance across pharmacology and drug discovery tasks, yet the internal mechanisms by which they encode pharmacological knowledge remain poorly understood. In this work, we investigate how drug-group semantics are represented and retrieved within Llama-based biomedical language models using causal and probing-based interpretability methods. We apply activation patching to localize where drug-group information is stored across model layers and token positions, and complement this analysis with linear probes trained on token-level and sum-pooled activations. Our results demonstrate that early layers play a key role in encoding drug-group knowledge, with the strongest causal effects arising from intermediate tokens within the drug-group span rather than the final drug-group token. Linear probing further reveals that pharmacological semantics are distributed across tokens and are already present in the embedding space, with token-level probes performing near chance while sum-pooled representations achieve maximal accuracy. Together, these findings suggest that drug-group semantics in LLMs are not localized to single tokens but instead arise from distributed representations. This study provides the first systematic mechanistic analysis of pharmacological knowledge in LLMs, offering insights into how biomedical semantics are encoded in large language models. Meaningfulness Statement A meaningful representation of life captures biologically grounded concepts in a structured and interpretable form. In biomedical LLMs, this entails encoding pharmacological knowledge that can be localized and intervened. Our work advances this goal by mechanistically characterizing how drug groups are represented in Llama-based models, identifying early-layer representations that are causally predictive of drug-group associations. We further show that pharmacological semantics are distributed across tokens and are stable across layers. 1 Introduction Large language models (LLMs) have recently emerged as a powerful foundation for a wide range of tasks in pharmacology 20 and drug discovery 10 , driven by their ability to model complex biomedical knowledge from large-scale text corpora. LLMs have been applied to target identification 9 , drug–drug interaction prediction 18 , and automated hypothesis generation 24 , particularly when deployed within agentic frameworks. Recent work further demonstrates that LLMs can be integrated with structured resources such as knowledge graphs and biomedical databases to support drug repurposing 23 and literature-driven drug discovery 7 . However, despite promising empirical results, the internal mechanisms by which LLMs encode pharmacological concepts, such as drug classes, functional groups, or therapeutic actions, remain poorly understood. This has motivated growing interest in mechanistic interpretability and representation analysis of biomedical LLMs, with the goal of localizing where and how pharmacological semantics emerge across layers and components of the model. Such understanding is critical not only for improving model reliability and generalization, but also for establishing LLMs as scientifically trustworthy tools in high-stakes biomedical domains. In this work, we use activation patching to causally demonstrate that early layers in Llama-based models play a key role in storing knowledge of drug groups. Interestingly, the strongest causal effects are associated with intermediate tokens within the drug-group span, rather than the final token, contrary to prior findings on general factual knowledge in LLMs 11 . In addition, linear probing of drug-group activations reveals that pharmacological semantics are distributed across tokens rather than localized to individual positions, and that such semantics are already present in the embedding space. Pharmacological knowledge representation in LLMs remains understudied. This work provides a systematic investigation of how such knowledge is encoded and retrieved, establishing a foundation for mechanistic understanding of biomedical LLMs. 2 Methods 2.1 Dataset construction for benchmarking and activation patching Although many benchmarks evaluate large language models (LLMs) on biomedical tasks, existing benchmarks do not explicitly target drug 1 1 1 By ”drugs” we refer to all compounds listed in the U.S. National Library of Medicine 14 , including approved drugs, compounds with known biological activity, and research compounds. -class relationships. We therefore constructed a dataset by parsing drug names from pharmacological actions categories curated by the U.S. National Library of Medicine 14 . We excluded categories not directly related to pharmacological mechanisms (e.g. aerosol propellants). Using a curated dictionary of drugs and drug groups, we next generated a dataset for evaluation and activation patching experiments. We used two-choice question–answering dataset for two reasons. First, drug names are differently tokenized, so the first or last tokens may not preserve sufficient information to reliably identify the drug, making token-level evaluation impractical. Second, the correct answer is not unique, as multiple drugs may belong to the same pharmacological class. Therefore, we reformulated each query as a two-choice question by randomly sampling distractor options and varying the position of the correct answer. Predictions were obtained by selecting the answer choice with the maximum logit value among the two options. Model performance was evaluated using accuracy, as the dataset is balanced. 2.2 Activation Patching The goal of this work is to investigate where the knowledge of drug classes is stored and if we can locate it in modern LLMs. To this end, we employed activation patching, a method widely used in mechanistic interpretability research to identify the specific model components responsible for particular functions and knowledge storage 11 . The method aims to causally identify activations that affect the model’s output. Concretely, it consists of three forward passes through the model: (1) a pass on a clean prompt, during which latent activations are cached, (2) a pass on a corrupted prompt, and (3) a pass on the corrupted prompt in which the activation of a selected model component is replaced with its corresponding activation from the clean cache. Specifically, we define a clean prompt with a known correct answer, for example: Question: Which compound is categorized as vasoconstrictor agents? A) ergotamine B) araldite Answer: where the correct answer is A. We then construct a corresponding counterfactual prompt in which the different drug group is chosen such that the correct answer becomes B: Question: Which compound is categorized as bronchoconstrictor agents? A) ergotamine B) araldite Answer: We performed activation patching by replacing selected activations from the counterfactual run with those from the clean run. We utilized symmetric token replacement in all experiments, as Gaussian noise may impair internal model mechanisms by introducing out-of-distribution inputs 25 . To evaluate the patching effect, we used a normalized logit difference metric, originally introduced for the indirect object identification (IOI) task 22 , defined as \mathrm{metric}(r,r^{\prime})\;=\;\frac{\mathrm{LD}_{\mathrm{pt}}(r,r^{\prime})-\mathrm{LD}_{*}(r,r^{\prime})}{\mathrm{LD}_{\mathrm{cl}}(r,r^{\prime})-\mathrm{LD}_{*}(r,r^{\prime})} where \mathrm{LD} denotes the logit difference between the correct and incorrect answers, and the subscripts \mathrm{cl} , * , and \mathrm{pt} correspond to the clean, counterfactual, and patched runs, respectively. Model layers are indexed from 0 to 31. All patching experiments were performed using Transformer Lens package 13 . 2.3 Linear probing To test whether MLP modules in Llama-based models are critical for semantic representation, we constructed custom datasets containing paired drug groups with opposing semantic meanings. As a first group, we selected adrenergic receptor alpha-agonists and alpha-antagonists: agonists enhance the activity of a biological target, whereas antagonists inhibit it. To ensure that any observed effects reflect genuine semantic structure rather than token-specific artifacts, we included a second group consisting of central nervous system stimulants and depressants. Specifically, we used a fixed two-answer prompt structure while varying the position of the drug group within the

Read the complete text