Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks
Andrew D. Blevins, Ian K. Quigley · 2025
Read Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks by Andrew D. Blevins, Ian K. Quigley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20924.txt.
Introduction
Andrew Blevins and Ian Quigley test whether drug-discovery models are learning chemistry or learning chemists. They link ChEMBL assay records to publication authors and train a classifier to name the chemist from a molecule's fingerprint alone. It succeeds often enough to matter, reaching sixty percent top-five accuracy across more than eighteen hundred chemists under a strict scaffold-based split.
An activity model fed only a protein identifier and the inferred author probabilities, with no molecular descriptors at all, then matches a baseline that sees the structure. The models are exploiting who made a compound and what that chemist usually targets, a Clever Hans failure mode that inflates public benchmark scores. The authors propose author-disjoint splits and dataset practices that separate chemist intent from biological effect.
Catalog record
| CREATOR | Andrew D. Blevins, Ian K. Quigley |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20924
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks Andrew D. Blevins, Ian K. Quigley Abstract Can machine learning models identify which chemist made a molecule from structure alone? If so, models trained on literature data may exploit chemist intent rather than learning causal structure-activity relationships. We test this by linking ChEMBL assays to publication authors and training a 1,815-class classifier to predict authors from molecular fingerprints, achieving 60% top-5 accuracy under scaffold-based splitting. We then train an activity model that receives only a protein identifier and an author-probability vector derived from structure, with no direct access to molecular descriptors. This author-only model achieves predictive power comparable to a simple baseline that has access to structure. This reveals a “Clever Hans” failure mode: models can predict bioactivity largely by inferring chemist goals and favorite targets without requiring a lab-independent understanding of chemistry. We analyze the sources of this leakage, propose author-disjoint splits, and recommend dataset practices to decouple chemist intent from biological outcomes. 1 Introduction In a meeting years ago, an experienced chemist glanced at a panel of compounds in front of us and said “that’s a Stuart Schreiber molecule.” The observation has haunted us: if humans can recognize chemist style from structure alone, could our machine learning models be doing the same? And if so, are we training models that learn chemistry and biology, or models that simply exploit intent leakage—the stylistic regularities that arise as chemists pursue specific goals? This failure mode is well documented in computer vision, where it goes by names like “Clever Hans” or shortcut learning [ 20 , 9 ] . Models latch onto dataset signatures and spurious correlations that work in aggregate but encode the wrong mechanism [ 33 ] , resulting in impressive benchmark numbers that collapse once out of distribution. We show that analogous shortcut behavior appears in medicinal chemistry. We test this hypothesis through a two-stage experiment (Figure 1 ). First, we link ChEMBL assays to publication authors, creating an author–molecule graph spanning 1{,}815 prolific authors. We train a classifier to predict who made a molecule from structure alone and find that chemist styles are highly distinctive and readily learnable. Second, we train an activity model that receives only a protein identifier and the predicted author probabilities, with no direct access to molecular structure. This author-only model achieves competitive performance (Table 2 ), comparable to a simple ECFP+protein baseline, showing that activity can be predicted from chemist intent proxies even when direct structural descriptors are withheld. In this work we use the term chemist style to refer to regularities in how molecules are designed within a lab or author group: preferred scaffolds, functional groups, reaction motifs, and target families. Operationally, however, we do not observe individual molecule designers. Instead, we infer “style” from ChEMBL document authorship metadata, treating each author identifier as a proxy for a lab- or PI-level group. This authorship-derived notion is imperfect—papers may list many authors, some of whom are biologists or non-synthetic contributors—but it is sufficient to ask whether authorship-linked patterns are strong enough to confound activity prediction. The problem runs deeper than known curation pitfalls [ 3 , 8 ] . Community datasets aggregate heterogeneous literature under uneven experimental intent, and even rigorous standardization efforts [ 5 , 32 ] leave residual signals that modern models readily exploit. We show that dataset construction choices encode chemist style signals that confound evaluation, analogous to known shortcuts in computer vision benchmarks, and we point toward leakage-aware splits, adversarial debiasing, and cross-lab replication as mitigations. Typical Activity Model Molecule Structure Protein ID Activity Model Active / Inactive Stage 1: Author Identification Molecule Structure Author ID Model (GBDT) Author Probabilities (1,815-dim) Stage 2: Activity Prediction Author Probabilities Protein ID Activity Model (GBDT) Active / Inactive No direct molecular features used! Figure 1: Overview of the two-stage chemist style leakage test. Stage 1: We train a model to predict which of 1,815 authors synthesized a molecule from its structure alone, achieving 60% top-5 accuracy. Stage 2: We use these author probabilities (plus protein ID) to predict activity, without providing molecular features. The strong performance (validation AUROC around 0.65) reveals a shortcut: models can predict activity by inferring chemist intent without explicitly modeling structure–activity relationships. 2 Related Work A large body of protein-ligand interaction work trains and/or validates on ChEMBL-derived bioactivity or related resources (e.g., BindingDB, KIBA): examples include DeepDTA, GraphDTA, MolTrans, and DeepAffinity [ 24 , 23 , 13 , 14 ] . Earlier ligand-based target prediction on ChEMBL demonstrated strong performance from classical ML [ 16 , 21 ] , and the ChEMBL resource itself continues to evolve toward higher-quality deposition and tooling [ 4 ] . Recent curation efforts (e.g. ExCAPE-DB, Papyrus, MF-PCBA) provide large, cleaned, binarized activity sets meant to reduce assay noise and ease fair benchmarking [ 17 , 32 , 6 ] . We situate our work within this lineage but show that author/style signals remain a strong confound. Structure-first models now predict complex biomolecular assemblies and sometimes claim binding improvements. AlphaFold 3 introduced a diffusion architecture that models proteins, nucleic acids, ligands, and modifications in a unified framework [ 1 ] . Boltz-2 extends this line by modeling both complex structures and (approximate) binding affinities as a joint task [ 25 ] . In molecular property prediction, foundation pretraining on massive unlabeled chemical corpora (e.g., Recursion’s MolE) can yield strong ADMET and activity baselines [ 22 ] . Our results complement these advances by isolating a distinct failure mode: chemist style shortcuts that may inflate performance irrespective of the underlying biological mechanism. Extended-Connectivity Fingerprints (ECFPs) [ 28 ] combined with boosted trees or random forests remain competitive for many ChEMBL-style tasks, particularly under leakage-prone splits [ 27 , 26 , 12 , 36 ] . We use a gradient boosting model on ECFPs as a deliberately simple probe to measure the strength of chemist style signals and to emphasize how non-causal shortcuts can dominate performance in public benchmarks. Figure 2: Classic example of a Clever Hans predictor, reproduced from Lapuschkin et al. [ 20 ] . The left panel shows an image correctly classified as containing a horse; the right panel shows a relevance heatmap revealing that the classifier bases its decision primarily on the photographer’s corner watermark rather than on the horse itself. By analogy, in this work we argue that molecules in ChEMBL carry chemist-specific “watermarks” in their structure, which even simple models can exploit, confounding attempts to learn genuine causes of binding Computer vision has long documented dataset–specific signals and cross–dataset failures, with a prominent example being 33 . Subsequent work formalized shortcuts —decision rules that exploit unintended cues and fail under distribution shift [ 9 ] —and exposed failure modes via explanation methods [ 20 ] . Robustness benchmarks such as ImageNet-C/A [ 10 , 11 ] and the WILDS benchmark for real distribution shifts [ 15 ] have driven better evaluations alongside algorithmic responses (e.g., GroupDRO and IRM) [ 29 , 2 ] . Together, these literatures motivate our focus on chemist style leakage as a chemistry–specific shortcut. Beyond well-known scaffold leakage and congeneric series effects, several works have documented systematic issues in public chemoinformatics