BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents

Jinge Wu, Hongjian Zhou , Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Sean Wu,[5pt] Honghan Wu, Fenglin Liu, David A. Clifton, [15pt] University of Oxford University College London Technical University of Munich · 2026

Read BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents by Jinge Wu, Hongjian Zhou , Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Sean Wu,[5pt] Honghan Wu, Fenglin Liu, David A. Clifton, [15pt] University of Oxford University College London Technical University of Munich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06177.txt.

Introduction

Jinge Wu and ten co-authors release BioMedArena, an open source toolkit for building and fairly evaluating biomedical research agents. Comparing such agents today is unreliable, since the same model on the same benchmark reports different scores under different harnesses and tool sets. Rebuilding a comparable setup for a new model can take weeks of custom engineering.

The toolkit separates six layers of evaluation, from benchmark loading to scoring, and exposes 166 biomedical benchmarks and 75 tools in nine families. New models, benchmarks, or tools plug in through a few lines of adapter code. It also ships reference agent harnesses and context strategies that users can equip on any backbone. Equipped backbones beat prior best results on eight representative benchmarks by about fifteen points on average.

Catalog record

CREATORJinge Wu, Hongjian Zhou , Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Sean Wu,[5pt] Honghan Wu, Fenglin Liu, David A. Clifton, [15pt] University of Oxford University College London Technical University of Munich
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06177

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents Jinge Wu, Hongjian Zhou , Mingde Zeng, Jiayuan Zhu, Junde Wu, Jiazhen Pan, Sean Wu,[5pt] Honghan Wu, Fenglin Liu, David A. Clifton, [15pt] University of Oxford University College London Technical University of Munich Abstract Building a deep research agent today is an exercise in glue code: the same backbone evaluated on the same benchmark can report different accuracies in different papers because harness and tool registry all differ, and integrating a new foundation model into a comparable evaluation surface costs weeks of model-specific engineering. We call this the per-paper engineering tax and release BioMedArena, an open-source toolkit that not only alleviates it but also provides an arena for fair comparison of different foundation models when evaluating them as deep-research agents. BioMedArena decouples six layers of biomedical agent evaluation—benchmark loading, tool exposure, tool selection, harness mode, context management, and scoring—and exposes 147 biomedical benchmarks and 75 biomedical tools across 9 functional families. Adding a new model, benchmark, or tool reduces to registering a few-line provider adapter. We further provide 6 agent harnesses (including our proposed Mutual-Evolve ) with 6 context-management strategies, which provide 12 backbones with competitive research capabilities and significantly improved performance, achieving state-of-the-art (SOTA) results on 8 representative biomedical benchmarks, with an average lift of +15.03 percentage points over prior SOTA 1 1 1 The toolkit, configurations, and per-task traces are available at https://github.com/AI-in-Health/BioMedArena . . Figure 1: Performance gains under BioMedArena across 8 representative biomedical benchmarks, boosting backbones and surpassing prior SOTA by +15.03 percentage points (pp) on average. 1 Introduction Recently, deep research agents have emerged as a focal point of LLM research, with benchmarks such as Humanity’s Last Exam (HLE) [ 32 ] drawing widespread attention as a proving ground for the research capability of LLMs with tools. Yet a concerning pattern has emerged: even when evaluated on the same benchmarks with the same backbone model, reported scores diverge across papers, and it is also difficult to efficiently reproduce the competitive performance described in those papers [ 20 ] . A core reason is that most existing studies do not open-source their evaluation harness and tool implementations, resulting in building a deep research agent today is, in practice, an exercise in glue code. Concretely, a researcher who wants to evaluate whether Claude Opus 4.6 [ 4 ] or Gemini 3.1 Pro [ 14 ] is the stronger backbone for agentic literature retrieval grounded in PubMed — or whether a newly released open-weight protein model is competitive with closed-weight backbones on variant interpretation — faces a problem that is not really about the models themselves: the candidate systems have never been evaluated under the same tool registry, the same iteration cap, or the same harness strategies. Assembling a comparable evaluation surface for a single new model takes days to weeks of model-specific engineering before a single accuracy number can be produced, let alone results that match those reported for closed-source systems. We call this the per-paper engineering tax : a cost that deep research agent researchers pay separately, repeatedly, and rarely amortize across papers. This tax operates on 3 concrete axes: Harnesses are not comparable : The agent harness is one of the most important components for enabling LLMs to perform deep research. However, existing deep research agent systems each implement their own loop (e.g., ReAct-style [ 41 ] , plan-execute, self-consistency (majority voting) [ 39 ] or dialog-turn [ 40 , 27 ] ), and the same backbone can report substantially different accuracies across them. Tool registries are not shared : each system [ 17 , 19 , 36 ] bundles its own typed tools with its own schema, and a tool implemented once is not shared or reusable for different models. Scoring is not unified : deterministic match, LLM-as-judge [ 42 ] , and code-execution paradigms are mixed inconsistently, and the choice of judge model itself shifts reported accuracy. The downstream consequence is that adding a new foundation model to a comparable evaluation surface requires re-implementing model-specific code in each of these 3 axes separately. There is no shared environment under which a model trained today can be compared head-to-head against a model trained tomorrow. To address this gap, we introduce BioMedArena, an open-source toolkit that decouples 6 layers of biomedical deep research agent evaluation — benchmark loading, tool exposure, tool selection, harness mode, context management, and scoring — and exposes 147 biomedical benchmarks, 75 biomedical tools across 9 functional families, and a benchmark-aware scoring router. BioMedArena provides a direct practical benefit: evaluating a new foundation model on over a hundred evaluation datasets requires no benchmark-specific glue code, no tool-integration work, and no scoring logic on the part of the model developer—a few-line, usually < 100-line , provider adapter is sufficient for comparison with previous models in deep-research capability under the same evaluation environment and settings. Meanwhile, BioMedArena implements 6 agent harnesses and 6 context-management strategies, any of which can be equipped on any backbone for fair and comparable evaluation against every other model. Meanwhile, Mutual-Evolve , a harness in which parallel solvers share intermediate findings through a typed Global Workspace, is presented to boost the deep-research capability of agents. We validate BioMedArena on 12 backbones over 8 benchmarks run under the same evaluation environment, in which the harness, tool registry, and scoring policy are fixed for fair comparison. The 12 backbones include 5 open-source and 7 closed-source models, and the 8 benchmarks cover medicine, biology, chemistry, genomics, laboratory research, and multidisciplinary reasoning. To our knowledge, this is the first regularly comparable biomedical agent evaluation at this scale. Figure 1 shows that BioMedArena improves every backbone’s research capability and helps them surpass prior best results on benchmarks, with an average gain of +15.03 percentage points in accuracy. BioMedArena provides not only a fairer evaluation surface but also a directly competitive one: a future model can be integrated with a few-line provider adapter and evaluated head-to-head under the same harness and tools used to produce these results. The per-paper engineering overhead is alleviated, and results become reproducible by construction. Overall, our main contributions are: • We build an open-source biomedical deep-research agent toolkit, BioMedArena, which decouples 6 layers of biomedical agent evaluation and registers 147 benchmarks, 75 typed tools across 9 functional families, 6 agent harnesses, and 6 context-management strategies. • We propose Mutual-Evolve , a deep-research agent harness in which multiple parallel agent solvers share key intermediate findings through a typed Global Workspace of four banks (errors, skills, tools, and guides), keeping the robustness of widely-used majority-vote self-consistency while letting solvers share what they learn and aggregate answers based on the evidence behind them, not only the final choice. • We present the first regularly comparable biomedical agent evaluation study at scale: 12 backbones over 8 benchmarks under a fixed and fair evaluation environment. We establish state-of-the-art results on the benchmarks, showing that BioMedArena provides a competitive, not merely standardized, evaluation environment for foundation models. Figure 2: Overview of the BioMedArena toolkit: a unified biomedical benchmark interface, a tool registry organized by functional family, multiple agent harnesses including our Mutual-Evolve , and composable context-management strategies, all unified behind a provider abstraction over open-source

Read the complete text