BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedicine

Yunsoo Kim, Yusuf Abdulle, Honghan Wu · 2025

Read BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedicine by Yunsoo Kim, Yusuf Abdulle, Honghan Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22240.txt.

Introduction

Answering a medical question often means hopping along chains of links: a drug touches a protein, the protein drives a disease. Yunsoo Kim, Yusuf Abdulle, and Honghan Wu build a test for that kind of reasoning in this 2025 paper. Their benchmark, BioHopR, draws one-step and two-step questions from a large biomedical knowledge graph, including questions with many right answers.

Top models stumble as the hops grow. The best, a reasoning-focused commercial model, scored about thirty-eight percent precision on one-step questions but under fifteen percent on two-step ones, and open models, including ones tuned for medicine, trailed further. Every model dropped sharply on the second hop, where the needed links stay implicit. The benchmark gives the field a yardstick for a skill that real biomedical work demands.

Catalog record

CREATORYunsoo Kim, Yusuf Abdulle, Honghan Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22240

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedicine Yunsoo Kim, Yusuf Abdulle, Honghan Wu Abstract Biomedical reasoning often requires traversing interconnected relationships across entities such as drugs, diseases, and proteins. Despite the increasing prominence of large language models (LLMs), existing benchmarks lack the ability to evaluate multi-hop reasoning in the biomedical domain, particularly for queries involving one-to-many and many-to-many relationships. This gap leaves the critical challenges of biomedical multi-hop reasoning underexplored. To address this, we introduce BioHopR , a novel benchmark designed to evaluate multi-hop, multi-answer reasoning in structured biomedical knowledge graphs. Built from the comprehensive PrimeKG, BioHopR includes 1-hop and 2-hop reasoning tasks that reflect real-world biomedical complexities. Evaluations of state-of-the-art models reveal that O3-mini, a proprietary reasoning-focused model, achieves 37.93% precision on 1-hop tasks and 14.57% on 2-hop tasks, outperforming proprietary models such as GPT4O and open-source biomedical models including HuatuoGPT-o1-70B and Llama-3.3-70B. However, all models exhibit poor capabilities in the multi-hop reasoning, underscoring the challenges of resolving implicit reasoning steps in the biomedical domain. By addressing the lack of benchmarks for multi-hop reasoning in biomedical domain, BioHopR sets a new standard for evaluating reasoning capabilities and highlights critical gaps between proprietary and open-source models while paving the way for future advancements in biomedical LLMs. BioHopR is available at https://huggingface.co/datasets/knowlab-research/BioHopR. 1 Introduction Recent advances in large language models (LLMs) and Question Answering (QA) systems have shifted the focus from simple factoid retrieval tasks to more sophisticated reasoning capabilities Huang and Chang 2022 ; Plaat et al. 2024 ; OpenAI 2025 . Among these, multi-hop reasoning has emerged as a critical area of research, where answering a question requires traversing multiple interconnected reasoning steps Misra et al. 2023 ; Yang et al. 2024 ; Schnitzler et al. 2024 . For example, to answer “Who is the wife of the president of the United States?” , a LLM must first identify the president (step 1) and then determine their spouse (step 2). This type of reasoning, referred to as multi-hop reasoning, is especially vital in domains where information is highly interconnected, such as the biomedical field. In the biomedical domain, knowledge is often structured in ontologies and knowledge graphs (KGs), where entities like drugs, diseases, proteins, and phenotypes are represented as nodes, and their relationships as edges Himmelstein et al. 2017 ; Sung et al. 2021 ; Chandak et al. 2023 . Biomedical queries frequently demand multi-step reasoning over these graphs Sung et al. 2021 ; Su et al. 2024 ; Matsumoto et al. 2025 . For instance, identifying diseases associated with a drug might require a single-hop relation, while determining proteins targeted by that drug through its associated disease involves two reasoning steps. Furthermore, biomedical reasoning often involves one-to-many or many-to-many relationships. For example, a single question might have several correct answers - like a drug that works on multiple proteins Liang et al. 2019 . This complexity highlights the need for specialized benchmarks that rigorously evaluate models’ ability to reason across multiple steps while generating comprehensive, multi-answer responses. Existing benchmarks for multi-hop reasoning, such as Hetionet Himmelstein et al. 2017 and other biomedical QA datasets Rao et al. 2022 , have laid the groundwork for evaluating multi-hop capabilities in the biomedical domain. However, these benchmarks primarily focus on single-hop tasks or utilize pre-defined templates that fail to fully capture the intricacies of multi-step reasoning. Similarly, general-domain benchmarks like TWOHOPFACT Yang et al. 2024 test models’ latent multi-hop reasoning ability but lack the domain-specific challenges of biomedical reasoning, such as reasoning over structured relationships and handling multi-answer outputs. As a result, the unique challenges of biomedical multi-hop reasoning remain underexplored. To address these limitations, we introduce BioHopR , a new benchmark specifically designed to test the multi-hop reasoning capabilities of LLMs in the biomedical domain. Unlike general-domain benchmarks that rely on reasoning across disconnected documents, BioHopR focuses on reasoning within a single, structured biomedical knowledge graph. Our benchmark systematically constructs 1-hop (e.g., Drug–Disease) and 2-hop (e.g., Drug–Disease–Protein) question-answer pairs from the PrimeKG knowledge graph Chandak et al. 2023 . Questions are designed to evaluate models’ abilities to reason step-by-step, explicitly requiring the inference of intermediate entities, and generate multi-answer responses reflective of real-world biomedical complexity. Contributions. Our main contributions are as follows: • A New Benchmark for Multi-Hop Reasoning: We propose BioHopR , the first publicly available benchmark explicitly designed to evaluate multi-hop, multi-answer reasoning within structured biomedical knowledge graphs. We will release the dataset and the code for evaluation. • Multi-hop Knowledge Curated from a Comprehensive Up-To-Date Knowledge Graph: Leveraging the comprehensive and up-to-date PrimeKG knowledge graph, we systematically construct a dataset of 1-hop and 2-hop biomedical questions and their answers, ensuring real-world relevance. • Evaluation and Analysis of LLMs in Biomedical Multi-hop Reasoning: We evaluate state-of-the-art LLMs on our benchmark, highlighting their strengths and, more importantly, limitations in handling biomedical multi-hop reasoning tasks. By introducing BioHopR, we aim to fill a critical gap in multi-hop QA research and advance the development of LLMs capable of robust and interpretable reasoning in structured, high-stakes domains like biomedical research and healthcare. Dataset Domain Reasoning Answer-Level MedQA Jin et al. 2021 Biomedical No Single Answer Hetionet QA Himmelstein et al. 2017 Biomedical Graph-based Reasoning (MH) Single Answer MedExQA Kim et al. 2024 Biomedical Explanation Generation Single Answer TWOHOPFACT Yang et al. 2024 General Implicit Reasoning (MH) Single Answer BioHopR (Ours) Biomedical Implicit Reasoning (MH) Multi Answers Table 1: Comparison of BioHopR with existing datasets. Key differentiators include domain focus, reasoning type (MH is tagged for multi-hop reasoning supported dataset), and answer-level, such as multi-answer capability. 2 Related Works Biomedical Question Answering. Research in medical LLMs has been facilitated by the development of question-answering (QA) datasets that benchmark models’ understanding of medical domain knowledge Hendrycks et al. 2020 ; Jin et al. 2021 ; Pal et al. 2022 . These datasets typically consist of multiple-choice questions (MCQs) focused on single-hop reasoning tasks, providing a straightforward way to evaluate LLMs’ ability to comprehend and respond to diverse medical inquiries. While these benchmarks have driven significant progress, they primarily measure classification accuracy, which is insufficient for capturing the nuanced reasoning required for medical expertise. Medical QA often involves interconnected concepts where reasoning over multiple steps is crucial. However, current benchmarks rarely go beyond single-hop tasks and do not evaluate models’ ability to provide explanations for their answers or justify their reasoning process. Recently, MedExQA introduced an evaluation framework with detailed explanations for assessing the reasoning capabilities of LLMs Kim et al. 2024 . While this is a step forward, it remains constrained to single-hop reasoning and does not address the need for multi-hop reasoning or the generation of multiple valid answers—a common requirement in biomedical inquiries. Knowledge Graph Question Answering. Knowledge Graph Question Answering (KGQA) systems leverage structured knowledge graphs to answer questions that require reasoning over graph-based relationships. In the biomedical domain, Hetionet Himmelstein et al. 2017 introduced a knowledge graph containing entities like genes, drugs, and diseases, enabling structured reasoning. Extensions of Hetionet have been used for multi-hop QA tasks Rao et al. 2022 , but these datasets often rely on fixed templates and predefined reasoning paths, limiting their ability to evaluate the nuanced multi-hop reasoning required in real-world biomedical applications.

Read the complete text