Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach

Dimitra Panou, Alexandros C. Dimopoulos, Manolis Koubarakis, Martin Reczko · 2025

Read Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach by Dimitra Panou, Alexandros C. Dimopoulos, Manolis Koubarakis, Martin Reczko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01480.txt.

Introduction

Dimitra Panou and co-authors pool many language models to answer biomedical questions in a 2025 paper. Biomedical literature grows fast, which makes text mining and question answering both vital and demanding. Their entry to a long running biomedical challenge uses open source large models as retrieval augmented generators for two question tasks.

Thirteen top open models were evaluated, and all combinations were explored to build tailored pipelines per question type. Majority voting settles yes or no questions, while list and factoid answers merge the union of model outputs. In the 2025 challenge rounds their system placed first for ideal answers in one task round, among other top placings.

Catalog record

CREATORDimitra Panou, Alexandros C. Dimopoulos, Manolis Koubarakis, Martin Reczko
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01480

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach Dimitra Panou, Alexandros C. Dimopoulos, Manolis Koubarakis, Martin Reczko Abstract Biomedical text mining and question-answering are essential yet highly demanding tasks, particularly in the face of the exponential growth of biomedical literature. In this work, we present our participation in the 13th edition of the BioASQ challenge, which involves biomedical semantic question-answering for Task 13b and biomedical question-answering for developing topics for the Synergy task. We deploy a selection of open-source large language models (LLMs) as retrieval-augmented generators to answer biomedical questions. Various models are used to process the questions. A majority voting system combines their output to determine the final answer for Yes/No questions, while for list and factoid type questions, the union of their answers in used. We evaluated 13 state-of-the-art open source LLMs, exploring all possible model combinations to contribute to the final answer, resulting in tailored LLM pipelines for each question type. Our findings provide valuable insight into which combinations of LLMs consistently produce superior results for specific question types. In the four rounds of the 2025 BioASQ challenge, our system achieved notable results: in the Synergy task, we secured 1st place for ideal answers and 2nd place for exact answers in round 2, as well as two shared 1st places for exact answers in rounds 3 and 4. [ orcid=0000-0002-9824-4489, [email protected] ] [ orcid=0000-0002-4602-2040, [email protected], url=https://dsit.di.uoa.gr/dimopoulos-cv/ ] [ orcid=0000-0002-1954-8338, [email protected], url=https://cgi.di.uoa.gr/ koubarak/ ] [ orcid=0000-0002-0005-8718, [email protected], url=https://www.fleming.gr/research/ifbr/staff-scientists/reczko-lab, ] *1 keywords Biomedical Question Answering,BioASQ ,Large Language Models ,Retrieval-Augmented Generation † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2025: Conference and Labs of the Evaluation Forum, September 09–12, 2025, Madrid, Spain † † address: Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece † † address: Institute for Fundamental Biomedical Science, Biomedical Sciences Research Center ”Alexander Fleming”, Greece † † address: Department of Informatics & Telematics, School of Digital Technology, Harokopio University, Greece † † address: Archimedes, Athena Research Center, Greece † † corresp: Corresponding author. 1 Introduction Large Language Models (LLMs) are transforming numerous fields, but their development and application face distinct challenges tied to accessibility and capabilities. Closed-source models, such as GPT-models OpenAI et al. 2024 , often maintained by large corporations, demonstrate advanced capabilities but lack public accessibility and transparency. In contrast, open-source LLMs, grant accessibility, transparency and facilitate fine-tuning and integration into customizable pipelines. Despite the remarkable progress in LLMs, which are increasingly capable of tackling complex tasks, biomedical QA remains a uniquely challenging field. Effective QA systems must not only retrieve relevant information handling domain-specific terminology, but also discern when to recommend a single ”best” option and when to present multiple perspectives. Avoiding mistakes is critical in this field, as decisions often have direct consequences on human health. Reliable question-answering systems must support experts in exploring these critical issues with accuracy and depth. In fast evolving fields such as drug discovery and molecular biology, where new findings appear constantly and may contradict earlier work, robust tools help professionals stay informed, avoid errors, and make evidence-based decisions that truly advance science and healthcare. 1.1 The BioASQ Challenge The BioASQ challenge has played a central role in advancing biomedical question answering (QA), particularly through its tasks. Task B requires systems to retrieve relevant documents and snippets and then generate precise answers to biomedical questions, while the Synergy track adds further complexity by introducing an interactive, feedback-based QA setting, simulating real-world clinical scenarios. These tasks push the limits of current Retrieval-Augmented Generation (RAG) systems, demanding high precision in both information retrieval and generation. Although large language models (LLMs) are becoming increasingly efficient today, the BioASQ challenge demonstrates that achieving accurate results relies on well-structured and carefully designed QA pipelines. Effective systems use hybrid retrievers Mandikal and Mooney 2024 , domain-specific encoders Lee et al. 2019 ; Gu et al. 2020 , and fine-tuned generators tailored for biomedical text. Pipelines often include re-ranking steps, prompt tuning, and targeted post-processing to handle subtasks like yes/no classification or list generation. These components are critical to ensure relevance, factuality and clarity, something end-to-end LLMs still struggle with in complex domains. Our lab has participated in BioASQ Challenge for three consecutive years. During this period, we experimented with various methodologies to enhance the document selection task. We began by developing our own model, ELECTROLBERT Reczko 2022 , and later fine-tuned a GAN Panou and Reczko 2023 combined with sparse BM25 for document ranking. In our most recent iteration, we transitioned to leveraging existing models for document retrieval, systematically exploring and comparing sparse, dense, and hybrid approaches Panou et al. 2024 . Despite improvements in our pipelines, we observed that the Mean Average Precision (MAP) in Phase A remains relatively low for all participants. This is mainly because selecting the right documents has become more difficult as the document collection keeps growing. Matching the retrieved documents with the small set chosen by experts remains a challenge. On the other hand, there is still room to improve how answers are generated from the retrieved documents. That’s why in this work, we focus on improving the generation of ‘ideal’ and ‘exact’ answers in Phase B. 2 Methodology 2.1 Synergy For the Synergy challenge, we used the same methods as for the final submissions of the BioASQ12 competition, with the notable addition of a DeepSeek-R1 model variant for the generation of exact and ideal answers DeepSeek-R1-Distill-Llama-70B-GGUF 2025 . The improved language generation skills of this model led to a notable improvement in the free text required for the ideal answers. 2.2 Task 13b, phase A: Document Retrieval & Snippet Identification In Phase A and Aplus of the BioASQ challenge, the organizers release biomedical questions curated by experts Nentidis et al. 2024 ; Nentidis et al. 2025 that have to be processed within a strict 24-hour interval. For Phase A participants have to retrieve and submit up to 10 relevant documents per question, utilizing abstracts sourced from the PubMed 1 1 1 https://pubmed.ncbi.nlm.nih.gov/ database. Based on the retrieved documents, participants must then identify and extract the most relevant snippets. For document retrieval in Phase A, we adopted a standard approach shown to deliver strong performance in previous work Panou et al. 2024 , specifically using the BM25 Robertson and Zaragoza 2009 ; Robertson et al. 2000 ranking algorithm enhanced with pseudo-relevance feedback from RM3 Lavrenko and Croft 2001 . From this setup, we initially retrieved the top 50 candidate documents and subsequently re-ranked them based on the relevance of their associated snippets. Snippet prediction, which extracts the most semantically relevant snippet from each of the top 10 retrieved documents, is performed as described in Panou et al. 2024 . 2.3 Task13b, phase A+ / phase B: exact answer generation In Phase A+ participants will submit exact and/or ideal answers before the expert selected (gold) documents and snippets (released in Phase B) are known. It serves as a baseline to compare with Phase B, where feedback is provided to guide system improvement. Each participant must rely on their own predicted documents and snippets for subsequent processing. They have 24

Read the complete text