Notebook for the BioASQ Lab at CLEF 2025
Shashank Verma, Fengyi Jiang, Xiangning Xue · 2025
Read Notebook for the BioASQ Lab at CLEF 2025 by Shashank Verma, Fengyi Jiang, Xiangning Xue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05577.txt.
Introduction
Shashank Verma, Fengyi Jiang, and Xiangning Xue describe their entry in a 2025 biomedical question-answering contest, the BioASQ challenge at CLEF. Doctors and researchers face a flood of medical papers, and a system that answers questions from that flood could save hours. Their system searches PubMed, the big medical library, and builds answers from what it finds.
The pipeline first turns papers into dense number codes for fast matching. It then re-ranks the best hits with a blend of fine-tuned cross-encoders and large language model rerankers before drafting a reply. Blending several rankers tends to beat any single one on hard questions. Contest runs like this show what retrieval-augmented systems can do for medical search today.
Catalog record
| CREATOR | Shashank Verma, Fengyi Jiang, Xiangning Xue |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05577
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Notebook for the BioASQ Lab at CLEF 2025 Shashank Verma, Fengyi Jiang, Xiangning Xue Abstract Biomedical semantic question answering rooted in information retrieval can play a crucial role in keeping up to date with vast, rapidly evolving and ever-growing biomedical literature. A robust system can help researchers, healthcare professionals and even layman users access relevant knowledge grounded in evidence. The BioASQ 2025 Task13b Challenge serves as an important benchmark, offering a competitive platform for advancement of this space. This paper presents the methodologies and results from our participation in this challenge where we built a Retrieval-Augmented Generation (RAG) system that can answer biomedical questions by retrieving relevant PubMed documents and snippets to generate answers. For the retrieval task, we generated dense embeddings from biomedical articles for initial retrieval, and applied an ensemble of finetuned cross-encoders and large language models (LLMs) for re-ranking to identify top relevant documents. Our solution achieved an MAP@10 of 0.1581, placing 10th on the leaderboard for the retrieval task. For answer generation, we employed few-shot prompting of instruction-tuned LLMs. Our system achieved macro-F1 score of 0.95 for yes/no questions (rank 12), Mean Reciprocal Rank (MRR) of 0.64 for factoid questions (rank 1), mean-F1 score of 0.63 for list questions (rank 5), and ROUGE-SU4 F1 score of 0.29 for ideal answers (rank 11). [ orcid=0009-0003-1478-4328, [email protected], ] *1 1 [ orcid=0009-0007-9420-5259, [email protected], ] 1 [ orcid=0000-0002-5775-1193, [email protected], ] 1 keywords Biomedical question answering ,Information Retrieval (IR) ,Retrieval Augmented Generation (RAG) ,Dense Retrieval ,Re-Ranker ,Large Language Model (LLM) ,Semantic Question Answering ,Prompt Engineering † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2025 Working Notes, 9 – 12 September 2025, Madrid, Spain † † address: Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332 † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Answering biomedical questions with precise, well supported responses requires intensive searching, reading and synthesizing information from dozens of research articles - a time consuming task demanding both domain expertise and contextual understanding. It poses unique challenges due to domain-specific jargon and compositional question structure. Solving this problem will enable access to an accurate biomedical assistant for researchers and clinicians, supporting their decision-making and work efficiency and could even open doors for access to general public for self education and to help fight biomedical misinformation. BioASQ Task 13b Biomedical Semantic QA Nentidis et al. 2025a describes this exact challenge, dividing it into two phases: Phase A is a retrieval task that identifies the most relevant list of documents and snippets for the given question, while Phase B focuses on generating both a formatted exact answer and a natural language ideal answer using the documents & snippets as context. Figure 1: Distribution of question types in BioASQ training data (left) and number of golden documents & snippets per question (right). The BioASQ challenge (Nentidis et al. Nentidis et al. 2025b ) has been continuously evolving to advance biomedical question answering, and we are participating in its 13th edition. We were provided a BioASQ-QA dataset Krithara et al. 2023 with 5,389 questions, each accompanied with gold responses for most relevant documents, snippets, exact answers, and ideal answers that were curated by field experts. The number of gold documents and snippets varied but were generally in the order of tens (Fig. 1 ). Questions were divided into four types as shown in Fig. 1 : yes/no (n=1,459), factoid (n=1,600), list (n=1,047), and summary (n=1,283). Question types define the format of exact answers: “yes” or “no” for yes/no questions; factoid questions expect a ranked list of alternative correct answers ordered by confidence, whereas list questions require a complete set of items without ranking. In addition to the exact answers, each of the question types also contain an ideal answer which is a natural language response to the question. In this paper, we present our automated Retrieval-Augmented Generation (RAG) system, which aims to solve this problem. While prior systems have largely relied on sparse retrieval using BM25 Robertson et al. 1995 , which rely on word matching or entity based searching through PubMed API, our approach utilizes modern dense retrieval techniques and neural re-rankers with advanced LLM-based generation, allowing for strong performance evidenced by consistent top10 placement in the leaderboard across phases and question types. In summary, our work makes the following contributions: 1. We built a dense retrieval system using bge-large-en embeddings on the full PubMed corpus, significantly improving document recall. 2. We designed an ensemble of finetuned cross-encoders and GPT-based re-rankers to maximize MAP@10 for retrieved documents. 3. We implemented a few-shot prompting pipeline with instruction-tuned LLMs to generate both exact and ideal answers. 2 Related Work 2.1 Retrieval-Augmented Generation (RAG) Retrieval-augmented generation (RAG) has emerged as a pivotal framework in natural language processing, particularly for enhancing the performance of large language models on knowledge-intensive tasks. The seminal work by Lewis et al Lewis et al. 2020 , introduced this paradigm, demonstrating how combining retrieval-based and generation-based approaches can improve accuracy by grounding model outputs in external documents. This approach is especially relevant for domains requiring up-to-date and domain-specific knowledge, such as biomedical question and answering. Subsequent work by Izacard et al Izacard and Grave 2021 further refined RAG by introducing Fusion-in-Decoder (FiD), which enhances generation by aggregating information from multiple retrieved documents. 2.2 Vector Databases for Scalable Search Storing and querying large sets of embeddings, requires scalable and efficient data structures. Malkov et al Malkov and Yashunin 2016 introduced Hierarchical Navigable Small World (HNSW) algorithm and provided the foundation of many vector database implementations, offering logarithmic complexity scaling for similarity searches. Johnson et al Johnson et al. 2017 further explored billion-scale similarity search, emphasizing GPU acceleration. Additionally, Annoy by Bernhardsson Bernhardsson 2018 offers an alternative ANN approach with tree-based indexing, though we prioritize HNSW for its balance of speed and accuracy. 2.3 Retrieval with Bi-Encoders and Cross-Encoders Efficient and accurate retrieval is central to RAG pipeline. Common ranking methods include traditional statistical models such as BM25 Robertson et al. 1995 , which rely on word matching, and more parameterized deep learning models like transformers based cross-encoders and bi-encoders, examples including PubMedBERT Gu et al. 2021 , MiniLM Wang et al. 2020 . Bi-encoders, such as Sentence-BERT (Reimers and Gurevych Reimers and Gurevych 2019 ), encode queries and documents independently into dense vectors, enabling efficient similarity searches over large datasets. However, bi-encoders often trade precision for speed. Rosa et al Rosa et al. 2022 demonstrated the superiority of cross-encoders in zero-shot retrieval tasks while Nogueira and Cho Nogueira and Cho 2019 explored their use in passage re-ranking with BERT. Retrieving large number of documents with a bi-encoder and re-ranking to the top-k with a cross-encoder mirrors strategies evaluated in the BEIR benchmark by Thakur et al Thakur et al. 2021 . 2.4 LLMs for Re-ranking Large Language Models (LLMs) are state of the art in many natural language processing tasks and have been employed in various aspects of Information Retrieval including passage re-ranking. Recent work by Ma et al. Ma et al. 2023 and Pradeep et al. Pradeep et al. 2023 has shown