scBench-Long:
Verifiable Benchmarking · 2026
Read scBench-Long: by Verifiable Benchmarking (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26563.txt.
Introduction
Ian Diks, Zhen Yang, and three colleagues at LatchBio introduce scBench-Long, a benchmark for long scientific work by AI agents. Single-cell biology studies individual cells rather than tissue averages. Analysts turn raw cell measurements into specific biological claims through many steps. Existing benchmarks test broad knowledge or single analysis steps. This one tests whether an agent can recover published conclusions from near-raw data, with no prescribed method.
The benchmark holds twenty-one evaluations. Topics include melanoma immune cells, lung tumours, and severe COVID-19 lung damage. Tasks mix several data types, from gene activity to immune receptor sequences. Candidate claims were checked and converted into controlled answer sets with fixed grading. Over a thousand agent runs were completed. The strongest model and harness pairing passed about a quarter of its runs. The authors argue real scientific agency needs this end-to-end test.
Catalog record
| CREATOR | Verifiable Benchmarking |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26563
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
scBench-Long: Verifiable Benchmarking Abstract Ian Diks 1,* , Zhen Yang 1,* , Arjun Banerjee 1 , Tim Proctor 1 , Kenny Workman 1 1 LatchBio, San Francisco, CA, USA * Equal first authors Correspondence: [email protected] ABSTRACT Single-cell studies require analysts to convert raw measurements into specific biological claims through multi-step workflows and integration of metadata, assay context, and auxiliary evidence. Existing AI-biology benchmarks largely measure broad knowledge, executable workflows, or local analysis steps. We introduce scBench-Long, a benchmark for long-horizon single-cell biology in which agents must recover scientific conclusions from raw or near-raw data without prescribed methods. The benchmark contains 21 evaluations spanning melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human–monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Tasks cover paired scRNA/TCR sequencing, RNA and chromatin profiling, cross-species transcriptomics, combinatorial scRNA-seq, single-nucleus RNA-seq, immune repertoires, ortholog maps, ligand–receptor resources, and validation evidence. Candidate claims are reproduced, reviewed, and converted into controlled answer vocabularies with deterministic grading and trajectory rubrics. Across 1,068 completed trajectories, the strongest model–harness pair passes 16/63 runs (25.4%). scBench-Long evaluates whether agents can move beyond local analysis steps and make complex scientific claims that are supported by single-cell data. Topline Benchmark Performance We ran the current evaluation set across frontier model families and agent harnesses. Passing requires exact recovery of the graded structured answer for an evaluation attempt. Present systems show low but nonzero success rates: the strongest model–harness pair passes 16/63 completed runs across a 1,068-trajectory benchmark matrix, while most pairs solve only a small minority of tasks. This is consistent with the benchmark targeting long-horizon scientific workflow control rather than isolated tool execution. Figure 1: Topline benchmark performance. Completed-run pass rates and task-level robustness counts across 21 final evaluations and 17 model–harness pairs. Wilson confidence intervals are reported in Table 2 rather than plotted because runs are grouped within a shared evaluation set. Figure 2: Efficiency frontiers for completed trajectories. Pass rate versus mean turns and mean cost per completed run. Points are model–harness pairs; dotted lines mark Pareto frontiers. Turn metadata were complete; cost metadata were available for 1,050/1,068 trajectories. Introduction Scientists use single-cell data to construct biological claims about cell states, perturbations, development, immunity, and disease. Raw single-cell measurements do not directly encode these claims. They must be processed through multi-step workflows, interpreted against experimental design, and integrated with assay-specific context, metadata, prior literature, and often additional assay data such as immune repertoires, chromatin accessibility, or functional validation experiments [ 17 , 18 , 19 ] . AI agents are beginning to show utility in biological data analysis, but current benchmarks largely test broad biology knowledge without deep treatment of the diverse task types specific to single-cell biology. Existing single-cell benchmarks focus on local analysis steps and do not evaluate realistic long-horizon science [ 34 , 35 , 36 , 37 , 38 , 27 , 28 ] . Verifiable ground truth is difficult to define in long-horizon single-cell benchmarking. The same dataset can support multiple valid claims, and published results do not always independently reproduce. A trustworthy evaluation must therefore carefully constrain the set of admissible answers without penalizing valid and potentially unanticipated analysis paths [ 39 , 40 , 41 , 43 , 28 , 29 , 38 ] . We introduce scBench-Long, a benchmark for long-horizon single-cell biology. scBench-Long contains 21 evaluations across human melanoma CD8 T-cell reactivity, RNA+ATAC regulatory inference, human–monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology [ 44 , 45 , 46 , 47 , 48 ] . Tasks provide raw or near-raw data, calibrated experimental context, auxiliary resources when needed, and compact scientific questions. Agents must recover structured biological conclusions that are graded deterministically over controlled vocabularies and symbols, with companion rubrics used to diagnose progress through key analysis chokepoints [ 27 , 28 , 29 ] . scBench-Long tests whether agents can move beyond accurately performing local analysis steps to reasoning about the kinds of realistic scientific tasks found in published studies or used in drug programs. We hope it serves as a useful measuring device for evaluating frontier agent systems as they improve in capability and are integrated into practical research and engineering. Benchmark Design Agents are evaluated on final scientific conclusions from 21 long-horizon single-cell evaluations spanning five study systems: human melanoma tumor-reactive CD8 T cells, RNA+ATAC regulation of CD8 tissue residency, human–monkey chimeric embryo development, age-dependent KRAS-driven lung tumorigenesis, and lethal COVID-19 lung pathology. The evaluations combine single-cell and single-nucleus transcriptomes, paired TCR or BCR repertoires, chromatin accessibility, cross-species ortholog mapping, ligand–receptor databases, developmental references, and functional validation data [ 44 , 45 , 46 , 47 , 48 ] . Benchmark Construction Workflow Figure 3: Benchmark construction workflow. scBench-Long tasks are built by selecting single-cell study systems, calibrating candidate claims through reproduction and review, packaging anonymized data and controlled answer surfaces, and pairing deterministic final-answer grading with hidden trajectory diagnostics. scBench-Long is structured around biological studies rather than independent datasets. Each study contributes multiple evaluations that revisit the same experimental system from different scientific perspectives, requiring agents to answer distinct questions about cell identity, state, regulation, composition, and mechanism. This mirrors how single-cell datasets are analyzed in practice: the same experiment is repeatedly interrogated to support different biological claims [ 17 , 18 ] . The benchmark builds on prior claim-conditioned benchmark construction rather than introducing a new general grading framework. Each evaluation includes raw or near-raw data, calibrated experimental context, scrubbed labels where needed, a compact scientific question, a controlled final-answer surface, reproduction notes, and trajectory rubrics. Paper claims are used as sources for candidate evaluations, not as automatic ground truth. Candidate tasks are retained only when the target conclusion can be reproduced from the staged data and hardened through review, distractor design, and inspection of trajectories from multiple model families [ 28 , 27 , 36 , 29 ] . The single-cell-specific contribution is the choice of study systems, evidence layers, answer surfaces, and failure probes. The 21 evaluations span human melanoma tumor-reactive CD8 T cell conjugates, RNA+ATAC regulation of CD8 tissue-resident-like programs, human–monkey chimeric embryo development, age-dependent KRAS-driven lung tumorigenesis, and lethal COVID-19 lung pathology. The underlying data include 10x Chromium 5 \prime single-cell RNA sequencing with paired TCR sequencing, 10x Multiome RNA and chromatin profiling, Smart-seq2 cross-species transcriptomics, Parse Biosciences combinatorial single-cell RNA sequencing, and 10x Chromium 3 \prime single-nucleus RNA sequencing. Several studies additionally require integrating T or B cell receptor repertoires, chromatin accessibility, ortholog mappings, ligand–receptor databases, developmental references, and immunohistochemistry [ 20 , 21 , 22 , 23 , 19 , 24 , 25 ] . Controlled answer surfaces are built around scientific objects that single-cell analysts reason with: cell-state identities, donor compositions, regulator and direction labels, clonotype or repertoire claims, ligand–receptor interpretations, and mechanism-caution conclusions [ 28 , 29 ] . Distractors are chosen to separate data-supported conclusions from common shortcuts, including canonical marker priors, raw-abundance rankings, single-modality answers, and causal interpretations not supported by the data. We use a small set of terms consistently throughout the paper. A final evaluation is one graded task with a controlled answer surface. A replicate is an independent attempt at that evaluation. A trajectory is the completed