Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark

Jasper Götting a, Pedro Medeiros a, Jon G Sanders a, Nathaniel Li †, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks, Seth Donoughe ∗ · 2025

Read Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark by Jasper Götting a, Pedro Medeiros a, Jon G Sanders a, Nathaniel Li †, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks, Seth Donoughe ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16137.txt.

Introduction

Jasper Gottling and co-authors benchmark language models on virology lab skills in a 2025 paper. Their test asks models to troubleshoot real lab protocols, a skill built from deep hands-on knowledge. Questions came from dozens of working virologists at doctoral level.

The set holds over three hundred multimodal questions spanning basic, tacit, and visual knowledge. Human experts scored only about twenty-two percent on questions inside their own specialty. The best model reached nearly forty-four percent, beating almost all experts tested. The authors warn that such skill is dual-use and raises governance questions. They propose care in how these growing abilities are shared and overseen.

Catalog record

CREATORJasper Götting a, Pedro Medeiros a, Jon G Sanders a, Nathaniel Li †, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks, Seth Donoughe ∗
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16137

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark Jasper Götting a, Pedro Medeiros a, Jon G Sanders a, Nathaniel Li †, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks, Seth Donoughe ∗ Abstract We present the Virology Capabilities Test (VCT), a large language model (LLM) benchmark that measures the capability to troubleshoot complex virology laboratory protocols. Constructed from the inputs of dozens of PhD-level expert virologists, VCT consists of 322 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories. VCT is difficult: expert virologists with access to the internet score an average of 22.1 % 22.1\% on questions specifically in their sub-areas of expertise. However, the most performant LLM, OpenAI’s o3, reaches 43.8 % 43.8\% accuracy, outperforming 94 % 94\% of expert virologists even within their sub-areas of specialization. The ability to provide expert-level virology troubleshooting is inherently dual-use : it is useful for beneficial research, but it can also be misused. Therefore, the fact that publicly available models outperform virologists on VCT raises pressing governance considerations. We propose that the capability of LLMs to provide expert-level troubleshooting of dual-use virology work should be integrated into existing frameworks for handling dual-use technologies in the life sciences. † † footnotetext: a Primary authors † † footnotetext: † Work conducted while at Center for AI Safety † † footnotetext: ∗ Correspondence to [email protected] 1 Introduction As AI development accelerates, evaluations have become essential for quantifying the capabilities of large language models (LLMs), particularly in scientific reasoning [ 33 , 23 , 22 ] . However, commonly used benchmarks, such as MMLU [ 23 ] or GPQA [ 50 ] , have important limitations. They often rely on multiple-choice questions with one correct answer among four choices. Although straightforward to create, evaluate, and grade, such benchmarks do not capture rare, tacit, and search-proof knowledge. Moreover, they do not test for reasoning over images, despite multimodality becoming a standard LLM capability with clear real-world applications [ 68 ] . Furthermore, many existing benchmarks suffer from false ground-truth labels and rapid saturation [ 42 , 5 , 40 , 51 ] . These limitations are particularly consequential for scientific applications and biosecurity risk assessment. Although there is a broad consensus that it is important to evaluate frontier AI models for biosecurity risks [ 39 , 13 , 3 ] , existing biosecurity-focused assessments have relied primarily on few benchmarks and isolated human red-teaming studies, which are time-consuming and costly [ 38 ] . Therefore, current efforts to measure the biological risks posed by general-purpose AIs remain nascent and speculative [ 47 ] , with existing evaluations focusing primarily on simple textbook knowledge or limited subsets of the complex challenges posed by real-world laboratory work [ 35 , 25 , 33 ] . Thus, there is great need for the ability to meaningfully assess the capacity of LLMs to assist in practical biology. It is an open question whether current or next-generation AI models could usefully assist a non-expert in successfully performing wet-lab experiments. Resolving this uncertainty is particularly important for virology: although advanced assistance for virology laboratory work is valuable for scientists working to cure and prevent infectious diseases, it is an inherently dual-use capability since viruses have been responsible for most historical pandemics and are likely to cause future ones as well [ 1 ] . We developed VCT to measure practical knowledge about virology, with a special focus on troubleshooting experiments. VCT targets virology methods with dual-use potential as well as other closely related methods. It excludes general molecular and cellular biology methods, as well as—out of safety concerns—material that is unambiguously hazardous, as illustrated in Figure 2 . Example VCT Question: Troubleshooting a low-contrast plaque assay I am analyzing my influenza virus plaque assay. MDCK cells were infected with low pathogenic avian influenza A virus for 1 h at 37 °C & 5\% CO 2 in a medium with 4\% bovine serum albumin for virus adsorption. The infected cell monolayer was incubated with a 0.45\% agarose overlay in growth medium supplemented with 1 µg/mL of L-(tosylamido- 2 -phenyl) ethyl chloromethyl ketone (TPCK)-treated trypsin for 48 hours at 37 °C & 5\% CO 2 . However, my well looks like this and quantification is difficult. What has likely occurred? Figure 1: A representative VCT question. The question text describes a scenario in detail. If the situation can only be resolved with visual information, the question also includes an image. To correctly answer the question, one must properly interpret the image, and then either determine which statements are true from a provided set of 4 – 10 answer statements (multiple-response format) or provide an open-ended answer that is graded against a rubric (free-text format). The full question will all answer components and additional examples are included in Appendix A8 . Concretely, the benchmark comprises questions that are important, difficult, validated, and—to represent real-world usage—multimodal: • Important: Questions test essential virology knowledge required for competent lab work. Without this knowledge, researchers would struggle to implement and troubleshoot protocols. The selected protocols include those with dual-use applications. • Difficult: Questions are deliberately challenging and “Google-proof”—i.e., answers cannot be easily found through web searches. Success requires deep domain expertise, as demonstrated by poor performance from non-experts with internet access who attempted to solve VCT questions. • Validated: Questions have answers that were validated by multiple experts. These reviewers were instructed to approve only questions they considered to be objective. While the practice of virology constantly evolves, we aim to only include questions with clear consensus answers in VCT. • Multimodal: Questions incorporate images to reflect real laboratory scenarios, where visual analysis is crucial for troubleshooting. High performance on VCT requires the interpretation of experimental results, setups, and readouts from images. Our evaluation reveals that frontier language models achieve performance comparable to, or exceeding, human experts on VCT ( Section 5 ). The best-performing model scores 43.8\% on VCT. PhD-level virologists, assigned VCT question-sets specifically tailored to each individual’s sub-area of specialization, scored only 22.1\% . These results raise urgent questions about the potential for current and future LLMs to provide actionable advice on dual-use virology methods to malicious actors. The scientific capabilities of frontier models will doubtless accelerate beneficial research in the life sciences, but the demonstrated ability to match or exceed expert performance in troubleshooting dual-use virology lab work warrants careful consideration (see Section 6 for further discussion). Figure 2: Schematic of material included in VCT. The horizontal axis represents increasing potential for misuse, from general molecular biology knowledge (left) to unambiguously dual-use topics (right). The vertical axis indicates knowledge abstraction level, from highly conceptual (top) to highly practical (bottom). The VCT benchmark (blue dashed box) focuses on practical, field-specific virology knowledge while excluding both foundational topics shared across biological disciplines and unambiguously dual-use content. Examples illustrate the classification of virology topics across these two dimensions. 2 Related Work Evaluating LLMs. Benchmarks are frequently used to evaluate LLMs’ scientific knowledge and mathematical reasoning [ 23 , 35 , 50 , 58 , 69 , 24 , 10 , 56 ] . Recent benchmarks have expanded beyond text-only formats to measure how effectively models can interpret and

Read the complete text