Can We Trust In-Distribution Success? Locked Evaluation Reveals Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation Prediction

Mehrdad Shoeibi, Niloofar Yousefi · 2026

Read Can We Trust In-Distribution Success? Locked Evaluation Reveals Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation Prediction by Mehrdad Shoeibi, Niloofar Yousefi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00152.txt.

Description

Shoeibi and Yousefi ask whether in-distribution success can be trusted in CRISPRi prediction. A benchmark win inside one distribution may fail under shift. An endpoint may also entangle with a design factor. They study both risks in perturbation-effect prediction. A frozen Geneformer representation is tested under a locked protocol.

Heads and model choice were frozen before test evaluation. Outcome labels stayed withheld until final unblinding. In-distribution, the representation beat a random-feature control by 0.1645 in R squared. It then failed zero-shot transfer on both external screens. Its correlations there were negative, below the control on each. A count-only model explained the benchmark endpoint strongly. Much of the magnitude signal overlapped with cell count, they find.

Catalog record

CREATORMehrdad Shoeibi, Niloofar Yousefi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE149 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2608.00152

Browse the full library · About the Tower of Babel