Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling
Thor Klamt, Wolfgang Nejdl, Ming Tang, [0.6em] L3S Research Center, Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany, [0.4em] [email protected] · 2026
Read Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling by Thor Klamt, Wolfgang Nejdl, Ming Tang, [0.6em] L3S Research Center, Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany, [0.4em] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11764.txt.
Introduction
Thor Klamt, Wolfgang Nejdl, and Ming Tang dissect why machine learning predictors of PROTAC drug activity look strong on random splits but falter on new protein targets. They trace most of the gap to differences between laboratories' measurements, which cap leave-one-target-out accuracy near 0.67 across eight architectures despite heavy tuning. Few-shot retraining with ADMET features lifted that score to 0.705, and they release PROTAC-Bench with 10,748 measurements to anchor future evaluation.
Catalog record
| CREATOR | Thor Klamt, Wolfgang Nejdl, Ming Tang, [0.6em] L3S Research Center, Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany, [0.4em] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 109059 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling Thor Klamt, Wolfgang Nejdl, Ming Tang, [0.6em] L3S Research Center, Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany, [0.4em] [email protected] Abstract Machine-learning predictors of biochemical activity often exhibit large random-split-to-leave-one-target-out generalisation gaps that have been documented but not decomposed. We frame this as an evaluation-science question and use targeted protein degradation as the empirical test bed. PROTACs (proteolysis-targeting chimeras) are heterobifunctional small molecules that induce targeted protein degradation, with more than forty candidates currently in clinical trials; published predictors of PROTAC activity report AUROC values of 0.85 0.85 to 0.91 0.91 under random-split cross-validation, while the leave-one-target-out (LOTO) protocol of Ribes et al. reduces performance to approximately 0.67 0.67 . Random-split cross-validation rewards within-target interpolation, whereas LOTO measures the novel-target prediction that de-novo design depends on. We decompose this gap and identify inter-laboratory measurement variance as the dominant component, anchored by a within-target cross-laboratory cascade that bounds the inter-laboratory contribution at 0.124 0.124 AUROC, substantially above the 0.05 0.05 contribution from binarisation-threshold choice across four labelling schemes. Across eight published architectures and ESM-2 protein language models scaled to 3B parameters, LOTO AUROC plateaus near 0.67 0.67 under the canonical evaluation protocol, with a comparable plateau under SMILES-level deduplication of the training partition; a 21-dimensional, 2000-trial hyperparameter optimisation across head architectures, molecular encoders, and protein encoders cannot break this ceiling, and the rank-1 single-seed configuration regresses by 0.161 0.161 AUROC under multi-seed validation, matching a closed-form selection-bias prediction ( 2 ; 3 ) . Few-shot k = 5 k{=}5 stratified per-target retraining combined with ADMET features lifts 65-target LOTO AUROC from 0.668 0.668 to 0.7050 0.7050 , and post-hoc Platt scaling recovers raw output to within the 0.05 0.05 well-calibrated threshold. We release PROTAC-Bench, a curated benchmark of 10,748 PROTAC degradation measurements across 173 targets and 65 LOTO folds, alongside the variance-decomposition framework, the per-target calibration protocol, and the evaluation code, providing a methodological template for variance attributions in other small-data therapeutic settings. 1 Introduction PROTACs are heterobifunctional small molecules that bind a target protein and an E3 ubiquitin ligase to induce the target’s polyubiquitination and proteasomal degradation. By eliminating the target rather than inhibiting its catalytic function, this mechanism in principle expands the druggable landscape to scaffold proteins, transcription factors, and other classes that lack the active-site geometry classical small-molecule inhibitors require, with more than forty PROTAC candidates currently in clinical trials and a medicinal-chemistry pipeline that depends on activity predictors for compound-library triage. Published PROTAC predictors 30 ; 33 ; 9 report AUROCs of 0.85 to 0.91 under random-split cross-validation. 43 show that approximately 80 percent of random-split test entries share their UniProt target with a training entry, and that the leave-one-target-out (LOTO) protocol drops macro performance to approximately 0.67 against a matched random-CV macro of 0.755 and pooled 0.902 . The pooled-to-macro aggregation thus separates from cross-target generalisation, with the latter dominated by cross-laboratory variance (Section 5 ). The gap is not in dispute. What remains unanswered is whether it reflects models that cannot extrapolate to novel targets, or measurement variance that no binary-label predictor can resolve. Under the second, the gap reflects inter-laboratory measurement noise and labelling heterogeneity rather than learning failure, with cross-domain precedent in molecular-property and pair-input prediction 18 ; 38 . Our analysis identifies inter-laboratory measurement variance as the dominant component, with the reproducibility floor as the operational ceiling at current dataset scale; the 65 LOTO-eligible targets reflect a deliberate trade-off in favour of within-target measurement density and multi-publication replicate structure over corpus size (Section 3.1 ). We report four contributions: (i) a variance-decomposition framework with three convergent ceiling estimates at 0.668 to 0.678 AUROC; (ii) an architecture-invariant performance plateau across eight published predictors and ESM-2 protein language models scaled to 3 billion parameters, with the rank-1 of 2,000 hyperparameter-optimisation trials regressing by 0.161 AUROC under multi-seed validation in agreement with the Bailey-López de Prado closed-form selection-bias prediction 2 ; 3 ; 7 ; (iii) a per-target recovery analysis where few-shot k{=}5 stratified retraining lifts 65-target LOTO AUROC from 0.668 to 0.7050, and a four-factor factorial decomposition that collapses an apparent +0.042 AUROC warhead-transfer effect to within seed noise once few-shot calibration is included as a co-factor; and (iv) PROTAC-Bench (10,748 measurements across 173 targets, 65 LOTO folds), evaluation code, Croissant metadata with MLCommons RAI extension, and a reusable dual-LLM metadata-enrichment pipeline. 2 Related Work PROTAC and targeted-degrader machine learning. Computational PROTAC activity prediction has been approached through deep architectures combining target-pocket encoders with PROTAC molecular encoders. DeepPROTACs 30 introduced a pocket graph convolutional network coupled to a BiLSTM over PROTAC fragment SMILES (0.847 random-split AUROC); DegradeMaster 33 extended this with an equivariant graph neural network over the ternary complex (approximately 0.94 random-split AUROC under supervised setting per their Table 2); PROTAC-STAN 9 introduced ternary-attention over Morgan fingerprints, ESM-650M target embeddings, and an E3-ligase one-hot vector (0.883 random-split). Per-architecture replication notes appear in Appendices F and O . 43 are the only prior authors to evaluate under LOTO, reporting an AUROC drop to 0.604; the random-CV-to-LOTO gap was documented but not decomposed. Generative approaches address PROTAC design rather than activity prediction 19 ; 31 ; 29 and assume a reliable activity oracle that the present work shows is currently unavailable. Ternary-complex structure prediction has advanced rapidly 13 ; 44 ; 40 ; 53 ; the structural approaches evaluated here predate AlphaFold-3 PROTAC accessibility, and Section 4.3 demonstrates that structure-prediction quality under currently-evaluable tools does not translate to activity-prediction performance under held-out-target evaluation. 1 1 1 Concurrent published predictors and benchmarks not included in the canonical-evaluation sweep (AiPROTAC, DeepPROTACs 2.0, DeepPSA, Protap 54 ) require pipeline-specific reimplementation outside the canonical scope and are deferred to future work. Cold-split evaluation protocols. Distinguishing random-split from cold-split evaluation has cross-domain precedent. 39 introduced the C1, C2, C3, and C4 evaluation classes for protein-protein interaction prediction, formalised by 38 for drug-target interaction prediction (PROTAC-Bench’s LOTO protocol corresponds to the C3 setting), and 4 demonstrated that current sequence-based PPI predictors collapse to chance under leakage-free evaluation. Cold-split evaluation under scaffold or similarity-controlled protocols reveals performance degradation that random splits hide 34 ; 47 ; 15 ; 1 ;the continuous similarity-resolution sweep in Appendix C integrates this framing with the binary LOTO protocol. 50 report up to 60 percent performance degradation under real-world molecular OOD; 24 introduce DataSAIL for principled split construction with quantitative leakage scoring; 45 established block cross-validation for ecological models in the presence of spatial autocorrelation, the ecological analogue of inter-laboratory measurement variance. Methodological lineage. The decomposition in Section 5 extends a methodological lineage of attributing apparent generalisation gaps to measurement-variance components: Cronbach generalizability theory ( 11 ) , pharmacogenomic and cross-source IC50 variance ( 21 ; 28 ) , train-test redundancy in molecular-property benchmarks ( 18 ) , hyperparameter-optimisation selection bias ( 7 ; 2 ; 3 ; 5 ) , the transportability framework where LOTO is an explicit S-node intervention ( 41 ) , architecture-invariance under cold-split evaluation ( 52 ; 12 ; 20 ) , accidental-taxonomy and shortcut-learning interpretations of inverted PLM scaling ( 22 ; 10 ; 17 ; 26 ) , simple-baseline outperformance of parameterised meta-learning ( 49 ;