Can We Trust In-Distribution Success? Locked Evaluation Reveals Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation Prediction
Mehrdad Shoeibi, Niloofar Yousefi · 2026
Read Can We Trust In-Distribution Success? Locked Evaluation Reveals Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation Prediction by Mehrdad Shoeibi, Niloofar Yousefi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00152.txt.
Description
Shoeibi and Yousefi ask whether in-distribution success can be trusted in CRISPRi prediction. A benchmark win inside one distribution may fail under shift. An endpoint may also entangle with a design factor. They study both risks in perturbation-effect prediction. A frozen Geneformer representation is tested under a locked protocol.
Heads and model choice were frozen before test evaluation. Outcome labels stayed withheld until final unblinding. In-distribution, the representation beat a random-feature control by 0.1645 in R squared. It then failed zero-shot transfer on both external screens. Its correlations there were negative, below the control on each. A count-only model explained the benchmark endpoint strongly. Much of the magnitude signal overlapped with cell count, they find.
Catalog record
| CREATOR | Mehrdad Shoeibi, Niloofar Yousefi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 149 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2608.00152