BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

Dionizije Fa ∗, Marko Čuljak ∗, Bruno Pandža, Mateo Čupić · 2026

Read BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics by Dionizije Fa ∗, Marko Čuljak ∗, Bruno Pandža, Mateo Čupić (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21800.txt.

Introduction

Dionizije Fa and co-authors introduce a test suite for artificial intelligence agents that do bioinformatics work. Modern agents promise to run whole analyses on request. But the field lacks shared ways to measure how well they perform and how robust they are. BioAgent Bench, presented in this 2026 preprint, is built to fill that gap.

The benchmark holds curated end-to-end tasks, including RNA sequencing analysis, variant calling, and metagenomics. Each prompt names the concrete output files the agent must produce, which allows automated checking of results. The suite also stress-tests agents under controlled disturbances. Shared benchmarks like this let labs compare agents fairly and spot failures before the tools reach real research pipelines.

Catalog record

CREATORDionizije Fa ∗, Marko Čuljak ∗, Bruno Pandža, Mateo Čupić
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics Dionizije Fa ∗, Marko Čuljak ∗, Bruno Pandža, Mateo Čupić Abstract This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) with prompts that specify concrete output artifacts to support automated assessment, including stress testing under controlled perturbations. We evaluate frontier closed-source and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that frontier agents can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, because bioinformatics workflows may involve sensitive patient data, proprietary references, or unpublished IP, closed-source models can be unsuitable under strict privacy constraints; in such settings, open-weight models may be preferable despite lower completion rates. We release the dataset and evaluation suite publicly. bioagent-bench/bioagent-bench bioagent-bench/bioagent-experiments † † affiliationnotice: Equal contribution 1 Introduction Figure 1: An overview of BioAgent Bench. Inputs to LLM agents consist of a task prompt, input data, and reference data. While solving the provided task, an agent can use general-purpose packages or specialized bioinformatics tools. After the agent finishes generation, LLM judge compares its outputs against ground truth and produces evaluation results. In addition to the standard ”vanilla” inputs, we also experiment with different perturbations to stress-test the agents. We focus our evaluation on 10 tasks in bioinformatics (each designed around a different organism, virus, or an entire ecosystem), and 10 models (5 open-weight and 5 closed-weight models). AI agents have become a central component of developing computational workflows, including those in the life sciences. Bioinformatics is a particularly compelling domain for such agentic systems, as many routine analyses involve chaining together command-line tools, managing heterogeneous file formats, and interpreting intermediate outputs in a way that is both structured and highly subfield-specific. Rigorously evaluating such analyses is challenging due to their complexity and the existence of a wide range of plausible outputs. Further, parameter and algorithm choices at early stages of a pipeline can drastically influence final results, thereby complicating ablation studies and weakening the robustness of drawn conclusions. These limitations make some analysis steps difficult or impossible to label with strict pass/fail criteria. Consequently, existing evaluations tend to collapse rich workflows into simplified question-answering or code-generation problems that fail to faithfully capture realistic agent behavior. At the same time, bioinformatics pipelines can operate on highly sensitive data, such as patient tumor sequencing samples or clinical metadata, where both privacy and intellectual-property constraints prevent sharing raw datasets with third-party model providers or releasing them publicly ( 28 ) . In these settings, practical deployment often requires models that can be run entirely within the institution’s secure environment. To address these gaps, we introduce a benchmark dataset of common bioinformatics tasks specifically designed to evaluate large language model (LLM) agents in realistic workflows that require tool use. The benchmark covers a diverse set of tasks that reflect everyday usage scenarios for bioinformatics practitioners. Using the benchmark, we compare the performance of frontier closed-source models with that of state-of-the-art open-source models when used as agents. We show that frontier proprietary models can successfully solve the majority of benchmark tasks without extensive custom scaffolding, suggesting that current LLM technology is already capable of supporting many routine bioinformatics workflows. In general, our work provides three main contributions: (i) a benchmark dataset for evaluating AI agents on practical bioinformatics tasks; (ii) a systematic comparison of closed and open-source models in this setting; and (iii) an evaluation suite and an evaluation harness that executes tasks, records intermediate steps, grades outputs, and tests robustness with perturbation tests. 2 Related Work General LLM agent benchmarks. The evaluation of large language models as autonomous agents has emerged as a critical research direction, moving beyond traditional static benchmarks to assess models in dynamic, interactive environments. AgentBench ( 12 ) provides a comprehensive evaluation framework spanning eight distinct environments, including operating systems, databases, knowledge graphs, and web browsing. Aiming to evaluate the abilities of LLMs to effectively utilize external tools and APIs 11 design a benchmark comprising 73 APIs and 314 tool-use dialogues, and decompose evaluation into three progressive levels: API call accuracy, retrieval capability, and end-to-end task completion with planning. ToolBench ( 22 ; 21 ; 6 ) further broadens this scope by covering 16k real-world APIs across 49 categories. Our work is most closely related to SWE-bench ( 8 ) , which focuses on software engineering and requires the models to generate patches that resolve problems reported in GitHub issues. Compared to software engineering or general tool-use benchmarks, biomedical tasks are harder to evaluate automatically because many candidate solutions ultimately require wet-lab experiments and clinical validation. Moreover, biological systems exhibit substantial noise and heterogeneity (e.g., batch effects, protocol variation), so multiple analysis pipelines or hypotheses can be reasonable, and success is better characterized by decision quality under constraints than by a single correct output. These properties complicate benchmark construction, where the tasks must balance realism with automatic evaluation. Biomedical LLM benchmarks. A growing body of work focuses on building benchmarks for LLMs and LLM-based agents for drug discovery and biomedical data analysis tasks. For example, BioML-bench ( 15 ) evaluates tasks where agents must parse a task description, build a pipeline, implement models, and submit predictions graded by established metrics across domains such as protein engineering, single-cell omics, biomedical imaging, and drug discovery. Broader capability-oriented evaluations include LAB-Bench ( 10 ) , a large multiple-choice benchmark covering practical biology research skills, including literature reasoning, database navigation, figure interpretation, and sequence manipulation. 13 propose an evaluation framework to assess agent capabilities in single-cell omics analysis. Finally, BixBench ( 16 ) focuses on data analysis in computational biology, presenting agents with real-world scenarios that require dataset exploration, multi-step analysis, and nuanced interpretation. In contrast to benchmarks centered on data analysis and broad research text response-based capabilities, our benchmark emphasizes multi-step bioinformatics pipelines across different bioinformatics domains and computational environments. 3 Preliminaries A bioinformatics pipeline is a workflow that transforms raw biological data, such as DNA/RNA sequencing reads, into analyzed outputs (e.g., alignments, expression estimates, or variant calls). Within a pipeline, it is useful to distinguish two kinds of data: input data and reference data. Input data is the sample material being processed (for example, reads from a particular experiment or individual). Reference data is the shared baseline used across many samples for steps like alignment or annotation (for example, it can be a reference genome, gene annotation, or a known variant database). Separately, in an evaluation setting, we use a different set of terms. A task is a single instance with defined inputs and a clear success criterion. A trial is one execution of that task. A grader is the evaluation logic that scores a trial. During a trial, we record a transcript (also called a trace or trajectory), meaning the full log of what happened, including messages, tool calls, and intermediate

Read the complete text