VILLA: V ersatile I nformation Retrieval From Scientific L iterature Using Large La nguage Models Conference: ; ;
Blessy Antony, Amartya Dutta, Sneha Aggarwal, Vasu Gatne, Ozan Gökdemir, Samantha Grimes, Adam Lauring, Brian R. Wasik, Anuj Karpatne, T. M. Murali · 2026
Read VILLA: V ersatile I nformation Retrieval From Scientific L iterature Using Large La nguage Models Conference: ; ; by Blessy Antony, Amartya Dutta, Sneha Aggarwal, Vasu Gatne, Ozan Gökdemir, Samantha Grimes, Adam Lauring, Brian R. Wasik, Anuj Karpatne, T. M. Murali (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23849.txt.
Introduction
Antony and co-authors present VILLA, a system that pulls virus mutation facts from scientific papers with language models. Good ground-truth datasets are scarce, and past extraction work leaned on short, neat text. VILLA uses multi-step retrieval to handle open-ended extraction. The team built a test set of 629 mutations in ten influenza A proteins from 239 papers. VILLA beats plain retrieval and other agent tools on this task, they report. The work targets virology, a field past studies largely ignored.
Catalog record
| CREATOR | Blessy Antony, Amartya Dutta, Sneha Aggarwal, Vasu Gatne, Ozan Gökdemir, Samantha Grimes, Adam Lauring, Brian R. Wasik, Anuj Karpatne, T. M. Murali |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80377 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VILLA: V ersatile I nformation Retrieval From Scientific L iterature Using Large La nguage Models Conference: ; ; Blessy Antony, Amartya Dutta, Sneha Aggarwal, Vasu Gatne, Ozan Gökdemir, Samantha Grimes, Adam Lauring, Brian R. Wasik, Anuj Karpatne, T. M. Murali Abstract . The lack of high-quality ground truth datasets to train machine learning (ML) models impedes the potential of artificial intelligence for science (AI for science) research. Scientific information extraction (SIE) from the literature using LLMs is emerging as a powerful approach to automate the creation of these datasets. However, existing LLM-based approaches and benchmarking studies for SIE focus on broad topics such as biomedicine and chemistry, are limited to choice-based tasks, and focus on extracting information from short and well-formatted text. The potential of SIE methods in complex, open-ended tasks is considerably under-explored. In this study, we used a domain that has been virtually ignored in SIE, namely virology, to address these research gaps. We design a unique, open-ended SIE task of extracting mutations in a given virus that modify its interaction with the host. We develop a new, multi-step retrieval augmented generation (RAG) framework called VILLA for SIE. In parallel, we curate a novel dataset of 629 629 mutations in ten influenza A virus proteins obtained from 239 239 scientific publications to serve as ground truth for the mutation extraction task. Finally, we demonstrate VILLA’s superior performance using a novel and comprehensive evaluation and comparison with vanilla RAG and other state-of-the art RAG- and agent-based tools for SIE. Figure 1. (A) We study an open-ended task in scientific information extraction (SIE) in the context of virology. (B) We develop VILLA, a novel two-stage method for retrieval-augmented generation (RAG). (C) VILLA outperforms zero-shot prompting, RAG- and agent-based baselines, and other state-of-the-art methods for SIE in our novel task of viral mutation extraction. 1. Introduction Artificial Intelligence (AI) for science is accelerating research through discovery of patterns and candidate hypotheses from large-scale datasets. However, a major barrier in AI for Science is the lack of high-quality ground truth datasets to train machine learning (ML) models ( 5 ; 43 ) . In many fields, the relevant information is spread across the literature as unstructured text ( 26 ; 20 ) . It is infeasible to manually review relevant publications ( 24 ; 6 ; 13 ; 41 ) , especially with the exponential growth in scientific literature ( 17 ; 16 ) . Hence, there is an urgent need for automated mechanisms to extract the findings hidden in the scientific corpus to advance AI for science. Scientific information extraction (SIE) is the process of identifying and retrieving the desired data from the unstructured text in research publications. Advances in SIE focus on mining important data from text to enable auto-generation of tabular databases of the extracted information ( 13 ) . In recent years, the promising potential of LLMs in understanding general language and reasoning tasks have led to the development of LLM-powered tools for SIE ( 58 ) . Various approaches adopt domain specific LLMs ( 49 ; 22 ; 40 ; 31 ) and general purpose LLMs for SIE ( 19 ; 10 ; 3 ; 52 ) using zero-shot and few-shot prompting, retrieval-augmented generation (RAG), and supervised fine-tuning (SFT) (please see Appendix A for brief descriptions). Multiple studies ( 29 ; 51 ; 10 ; 52 ) demonstrate and compare the ability of open and closed LLM-driven tools to mine information from abstracts and full text of scientific publications. Despite the success of these methods, there are two issues. (1) Standard information extraction tasks are inadequate in capturing the complexity of SIE Existing studies evaluate the capabilities of LLMs in SIE through tasks such as named entity recognition (NER) ( 29 ; 51 ; 10 ) , relation extraction (RE) ( 29 ; 10 ) , question answering (QA), and document classification (DC). All these tasks except NER are formulated in a manner that includes the desired information in the prompt. QA explicitly provides two or more (one correct and remaining incorrect) options. IR, RE, and DC implicitly include a list of possible labels in the prompt for the LLMs to choose from. Moreover, all these tasks involve extracting information from short, clean, and well-formatted scientific text, such as a sentence or an abstract. These tasks do not realize the complexity of parsing the full text of publications and extracting the correct information. (2) The potential of SIE methods in complex, open-ended tasks is unknown Prior benchmarking pipelines evaluate the ability of several general purpose LLMs in SIE using zero-shot ( 10 ) and few-shot prompting ( 29 ; 10 ) . Additionally, many studies report and compare the performance of LLMs fine-tuned for SIE ( 29 ; 10 ) . Though the fine-tuning approach yields the best performance, the compute resources requirement for SFT of LLMs is high ( 29 ; 10 ) . Several RAG-based tools such as HiPerRAG ( 20 ) and agentic frameworks such as OpenScholar ( 2 ) , and PaperQA2 ( 54 ) that exploit the advantages of RAG and SFT ( 2 ) have demonstrated superior performance in SIE. However, the evaluation of these methods was limited to choice-based tasks. There are no studies that evaluate and compare the performance of RAG-based methods in open-ended tasks. In this study, we have addressed both these major research gaps. We used a highly under-explored domain in SIE, namely virology, to evaluate and demonstrate the potential of our contributions. Akin to other domains, the literature in virology has rapidly expanded (particularly since the COVID-19 pandemic) from 40,000 papers annually in 2019 to over 80,000 in subsequent years. It is imperative to extract pertinent information from this vast corpus using SIE methods and convert them into suitable formats for computational analysis to accelerate research in virology. However, there are few ground-truth datasets for SIE tasks in virology due to the paucity of annotated datasets in this domain. The standard datasets used to benchmark SIE methods span a wide range of topics such as drugs ( 12 ; 45 ; 27 ; 25 ) , diseases ( 15 ; 23 ; 9 ) , molecular biology ( 55 ; 11 ; 7 ) , and general biomedical concepts ( 44 ; 30 ) . See Appendix B for details on the datasets. Barring LitCovid ( 9 ) , the proportion of questions pertaining virology in these datasets is low ( 0.34\% to 5.32\% ). LitCovid is a virology dataset focusing only on SARS-CoV-2. In this study, (1) we designed an open-ended SIE task, i.e., the prompt we created did not ask LLMs to select from a set of pre-defined labels or answers. This task prompted the LLM to extract mutations in a given viral protein. The query did not include any concise or formatted text containing the potential answer. (2) We proposed a novel framework called ‘ VILLA’ for SIE. This two-stage RAG-based framework includes a new technique to construct the context by using abstracts to identify relevant papers and then performing an exhaustive search using the full-text of the selected publications. (3) Our team of domain experts (virologists) manually curated a novel dataset in