BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation
Fuyi Yang, Chenchen Ye, Mingyu Derek Ma, Yijia Xiao, Matthew Yang, Wei Wang · 2025
Read BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation by Fuyi Yang, Chenchen Ye, Mingyu Derek Ma, Yijia Xiao, Matthew Yang, Wei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08866.txt.
Introduction
Fuyi Yang and five co-authors introduce BioVerge, a benchmark and testing ground for AI agents that propose new biomedical hypotheses. Finding hidden links in the literature has long driven discovery. Older methods stick to single data types or fixed extraction patterns, though. That caps the novelty of what they can find.
The dataset blends structured records with text drawn from historical hypotheses and the PubMed literature. It is arranged so agents can explore at the edge of current knowledge. The companion agent works in a generate-and-judge loop: one module drafts hypotheses while another scores them, round after round. Experiments show architecture choices change how widely the agent explores, structured and textual sources each add unique context, and self-evaluation measurably improves the novelty and relevance of the proposals.
Catalog record
| CREATOR | Fuyi Yang, Chenchen Ye, Mingyu Derek Ma, Yijia Xiao, Matthew Yang, Wei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08866
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation Fuyi Yang, Chenchen Ye, Mingyu Derek Ma, Yijia Xiao, Matthew Yang, Wei Wang Abstract Hypothesis generation in biomedical research has traditionally centered on uncovering hidden relationships within vast scientific literature, often using methods like Literature-Based Discovery (LBD). Despite progress, current approaches typically depend on single data types or predefined extraction patterns, which restricts the discovery of novel and complex connections. Recent advances in Large Language Model (LLM) agents show significant potential, with capabilities in information retrieval, reasoning, and generation. However, their application to biomedical hypothesis generation has been limited by the absence of standardized datasets and execution environments. To address this, we introduce BioVerge , a comprehensive benchmark, and BioVerge Agent , an LLM-based agent framework, to create a standardized environment for exploring biomedical hypothesis generation at the frontier of existing scientific knowledge. Our dataset includes structured and textual data derived from historical biomedical hypotheses and PubMed literature, organized to support exploration by LLM agents. BioVerge Agent utilizes a ReAct-based approach with distinct Generation and Evaluation modules that iteratively produce and self-assess hypothesis proposals. Through extensive experimentation, we uncover key insights: 1) different architectures of BioVerge Agent influence exploration diversity and reasoning strategies; 2) structured and textual information sources each provide unique, critical contexts that enhance hypothesis generation; and 3) self-evaluation significantly improves the novelty and relevance of proposed hypotheses. Our dataset and code are accessible here: https://github.com/Fuyi-Yang/BioVerge/ \workshoptitle LAW and AI4Science 1 Introduction Figure 1 : The ABC principle as a method to explore Literature Based Discovery (LBD). Traditional hypothesis generation tasks centered around finding hidden relationships among concepts within the scientific domain. A widely recognized method was Literature Based Discovery (LBD), which highlights that undiscovered knowledge exist within scientific literature and are obtainable through automated extraction systems 4 . Importantly, the ABC principle depicted in Figure 1 tackles the LBD task 36 , stating that if distinct literature connects concepts A/B and concepts B/C , then the intersections of concept B infers potential links between concepts A/C . This technique harnessed many novel relationships that were proven experimentally, including connections between magnesium deficiency and migraine, Raynaud’s disease and fish oil, significantly contributing to scientific research 35 ; 4 . Despite earlier successes, these traditional co-occurrence, semantic, and graph-based methods often relied on a single data type, limiting their ability to discover subtler connections 33 ; 7 ; 13 ; 46 ; 41 ; 38 . Furthermore, with little reasoning or refinement, these approaches fail to develop intricate reasoning paths that lead to more discoveries 10 . Recently, Large Language Models (LLM) agents have garnered significant interest from its performance on exploration, reasoning, and generation tasks 39 ; 42 ; 49 . LLM-based agents can utilize tools to gather information, perform thoughtful reasoning, and create novel proposals 29 ; 2 ; 51 ; 15 ; 34 . However, despite its potential, LLM agents have limited applications in the biomedical hypothesis generation, due to a lack of standardized benchmarks and databases. To address this gap, we introduce BioVerge , a benchmark that stores biomedical knowledge as structured hypotheses triplets from PubTator3 and textual PubMed literature 45 ; 28 . The knowledge base contains data published before Jan. 1, 2024 to prevent test set data contamination, and both datasets perform cleaning and filtering to ensure experiment and evaluation fairness. We rank candidate test hypotheses by the impact factor (IF) of their publishing journals to select a credited, reliable test set 32 . We also propose BioVerge Agent , an LLM agent framework leveraging BioVerge to perform biomedical hypothesis generation. We construct an API interface to support agent tool calling to extensively explore and retrieve context in natural language. By alternating between Generation and Evaluation modules, BioVerge Agent can propose, self-evaluate, and iteratively refine hypothesis. We further compare Single and Double Agent architectures, which differ in memory sharing between the modules, to explore their impacts on hypothesis generation reasoning and performance. Finally, we perform extensive experiments on our constructed test dataset. Our results reveal that Single and Double Agent clearly present different reasoning strategies and suggests a tradeoff between exploration and accuracy performance. Additionally, the access to diverse structural and textual sources, and self-evaluation of past proposals, all encourage BioVerge Agent to refine its reasoning trajectory and notably improve the novelty and relevance of hypothesis proposals. Overall, our main contributions are three-fold: 1. We introduce the BioVerge benchmark, which enables tool-augmented reasoning over both structured and unstructured biomedical data. 2. We demonstrate the capability of BioVerge Agent to generate hypotheses that push the boundaries of existing biomedical knowledge. 3. We present a systematic study of self-evaluating agent architectures and information sources, identifying key design choices that enhance biomedical hypothesis generation. 2 BioVerge benchmark Figure 2 : The cutoff date separation between the historical knowledge base and test dataset candidates for dataset construction. The knowledge base contains article corpus, knowledge graph, and triplet representations. The test dataset is cleaned to ensure all test queries are proposed after Jan. 1, 2024, related to Diabetes Mellitus, and ranked by the impact factor (IF) score of its publishing journal. We define hypothesis as a triplet (s,r,o) , where s,o\in\mathcal{E} are biomedical entities and r\in\mathcal{R} is one of twelve relation types in Table 2 , all defined in the PubTator3 ontology 45 . We denote hypothesis description d as a natural language explanation for (s,r,o) . Biomedical hypothesis generation task. Given entities s,o\in\mathcal{E} , propose relation r\in\mathcal{R} and description d such that (s,r,o)\notin H (knowledge base), (s,r,o)\in T (test set), d is distinct from historical articles with hypotheses of the same s,o entities and aligns with ground truth articles. Table 1 : Dataset statistics for the knowledge base and test set. Dataset Triplets Publications Unique Entities Knowledge Base 10,566,798 8,896,514 558,006 Test Set 177 135 174 Table 2 : The relations, descriptions, and valid entity pairs for each relation defined in Pubtator3. Relation Description Valid Entity Pairs Associate Complex or unclear relationships (Chemical, Disease), (Chemical, Gene), (Chemical, Variant), (Disease, Gene), (Disease, Variant), (Variant, Variant) Cause Triggering a disease by a specific agent (Chemical, Disease), (Variant, Disease) Compare Comparing the effects of two chemicals or drugs (Chemical, Chemical) Cotreat Simultaneous administration of multiple drugs (Chemical, Chemical) Drug interact Pharmacodynamic interactions between two chemicals (Chemical, Chemical) Inhibit Reduction in amount or degree of one entity by another (Chemical, Variant), (Gene, Disease) Interact Physical interactions, such as protein-binding (Chemical, Gene), (Chemical, Variant), (Gene, Gene) Negative correlate Increases in the amount or degree of one entity decreases the amount or degree of the other entity (Chemical, Gene), (Chemical, Variant), (Gene, Gene) Positive correlate The amount or degree of two entities increase or decrease together (Chemical, Chemical), (Chemical, Gene), (Gene, Gene) Prevent Prevention of a disease by a genetic variant (Variant, Disease) Stimulate Increase in amount or degree of one entity by another (Chemical, Variant), (Gene, Disease) Treat Treatment of a disease using a chemical or drug (Chemical, Disease) 2.1 Dataset construction In following sections we describe the construction and cleaning of the historical knowledge base and test set, along with evaluation metrics for experiment studies. Dataset statistics are shown in Table 1 . 2.1.1 Knowledge base BioVerge contains