Toward Total Recall: Enhancing FAIRness through AI-Driven Metadata Standardization
Sowmya S. Sundaram, Mark A. Musen · 2025
Read Toward Total Recall: Enhancing FAIRness through AI-Driven Metadata Standardization by Sowmya S. Sundaram, Mark A. Musen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05307.txt.
Introduction
Sowmya Sundaram, Rafael Gonçalves, and Mark Musen chase total recall in science data search by fixing metadata with AI. Their paper is from two thousand twenty-five; the pack row gives no title or year, so the text head is the source. Metadata records are often patchy or malformed, which hides data sets from search.
They pair GPT-4 with strict templates from the CEDAR store, which name each field and its allowed values. The model then cleans records in bulk to fit the template. On two big biology stores, recall of apt data sets leaps from under a fifth to near two thirds. The template path beats a plain data-dictionary path, and rival models trail the GPT-4 pairing.
Catalog record
| CREATOR | Sowmya S. Sundaram, Mark A. Musen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Toward Total Recall: Enhancing FAIRness through AI-Driven Metadata Standardization Sowmya S. Sundaram, Mark A. Musen Abstract Current metadata often suffer from incompleteness, inconsistency, and incorrect formatting, hindering effective data reuse and discovery. Using GPT-4 and a metadata knowledge base (CEDAR), we devised a method that standardizes metadata in scientific data sets, ensuring the adherence to community standards. The standardization process involves correcting and refining metadata entries to conform to established guidelines, significantly improving search performance and recall metrics. The investigation uses BioSample and GEO repositories to demonstrate the impact of these enhancements, showcasing how standardized metadata lead to better retrieval outcomes. The average recall improves significantly, rising from 17.65% with the baseline raw datasets of BioSample and GEO to 62.87% with our proposed metadata standardization pipeline. This finding highlights the transformative impact of integrating advanced AI models with structured metadata curation tools in achieving more effective and reliable data retrieval. keywords metadata, FAIR, NLP Introduction Effective data sharing can be improved by adhering to FAIR standards Wilkinson et al. 2016 —ensuring data is Findable, Accessible, Interoperable, and Reusable. An important aspect of making data FAIR is high quality metadata Musen 2022 , which refers to descriptive information about the data. Metadata play a pivotal role in organizing, categorizing, and enhancing the discoverability of data by including details such as keywords, formats, and contextual information about the data’s origin, purpose, and usage. However, manually improving metadata quality is a complex and time-consuming process due to lack of consensus, variability in community standards, etc. To address this challenge of metadata quality, we propose an automated method for standardizing metadata. In this research endeavor, we investigate the use of Natural Language Processing (NLP) techniques for automated metadata correction and assess their impact on recall performance, a quantitative metric for data retrieval. The focal point of our investigation is on improving the metadata, as high-quality metadata enhance the searchability of public datasets, ultimately benefiting scientific research (Figure 1 ). Figure 1: The Big Picture Metadata comprise lists of field name–value pairs that may be augmented reporting guidelines. In Figure 2 —a snapshot of a record sourced from the BioSample repository of the National Center for Biotechnology Information (NCBI) Barrett et al. 2012 —the black box highlights an example of metadata. In this example, the field name is tissue and the field value is lung cancer, which is inaccurate as lung cancer is not a type of tissue. Searches for scientific datasets primarily involve querying metadata. Consequently, a researcher querying for the name–value pair of tissue would overlook this potentially valuable record during their search. As important as metadata are, current metadata are usually incomplete, inconsistent, and incorrectly formatted Gonçalves and Musen 2019 . Standardizing metadata is therefore a necessary process for ensuring that datasets are properly described and accessible, so that they they can be more easily reused and integrated with other datasets, thus facilitating secondary use and analysis of scientific data.. Figure 2: Record from BioSample where the black box highlights a field name–value pair. In this example, the field name tissue is wrongly associated with field value lung cancer . Orange boxes mask identifying information The process of metadata standardization involves the analysis of the language used to specify name–value pairs. To the best of our knowledge, there have been a lack of concerted efforts in this research area, underscoring the relevance of our lab’s consistent work in automated metadata standardization. In a previous attempt to enhance BioSample records using NLP techniques Gonçalves et al. 2019 , our team employed the earliest available embeddings (vectorial representations of words learned by training over text)—specifically word2vec Mikolov et al. 2013 and GloVe Pennington et al. 2014 —to represent metadata terms. These embeddings are powerful tools for capturing semantic relationships among words. However, we encountered significant scalability challenges because these embeddings were limited to terms present in the training set, thereby neglecting the specialized and niche terms often found in medical and scientific data. To address these limitations, we expanded our approach by incorporating Generative Pre-trained Transformer 4 (GPT-4) Achiam et al. 2023 embeddings S Sundaram and Musen 2023 , which provide a more comprehensive and flexible representation of language. Unlike traditional embeddings, GPT-4 can accommodate a broader range of terms, including those unseen during training or those that are less common or domain-specific. We subsequently analyzed how GPT-4 can improve the metadata of a small dataset of 200 metadata records from BioSample Sundaram et al. 2024 through a peer review board. Our results indicated that the domain experts preferred GPT-4 augmented with cues from a metadata knowledge source in its prompts. In our current study, we expand our investigation to a large dataset of 2,400 samples each from BioSample and the Gene Expression Omnibus (GEO) Edgar et al. 2002 , evaluating whether the necessity of a metadata knowledge source remains valid across different LLMs, including GPT-4, Large Language Model Meta AI (LLaMA-3) Touvron and et al. 2023 , and MedLLaMA-2 Alvi and et al. 2023 . Instead of a peer review board, we measure the impact on search outcomes. While many authors highlight the theoretical benefits of the FAIR principles, the absence of quantitative analysis complicates the assessment of the tangible effects of FAIR data. A key contribution of our work is the quantification of the retrieval of BioSample records following metadata standardization and its effect on findability of scientific data. The Search Experiment Our experiment is to improve metadata standardization of BioSample records and measure the impact of metadata improvements on search metrics. We begin by detailing the dataset utilized in our experiment. Dataset For the experiment, we utilized records from two separate datasets: one from BioSample and another from GEO. Each dataset contains 2,400 samples related to three types of cancer: lung cancer, liver cancer, and ovarian cancer. The queries used to retrieve the records were as follows: lung cancer (query: lung cancer[All Fields] AND ”human 1 0”[filter]) 1 1 1 This query translates to searching all the fields of a record to match with lung cancer and filtering the result set to only contain records from human samples , liver cancer (query: liver cancer[All Fields] AND ”human 1 0”[filter]) , and ovarian cancer (query: ovarian cancer[All Fields] AND ”human 1 0”[filter]) . For each query, we randomly sampled 800 records from both datasets. We initially sampled 1,000 records for each query, removed those with formatting errors, and selected the maximum uniform number of well-formatted records across all datasets, which happened to be 800. Therefore, our total dataset comprises 4,800 records: 2,400 from BioSample and 2,400 from GEO (Table 1 ). Dataset Description BioSample (2400) LungCancer (800) LiverCancer (800) OvarianCancer (800) GEO (2400) LungCancer (800) LiverCancer (800) OvarianCancer (800) Total - 4800 records Table 1: Dataset Composition Method Manual standardization of metadata involves a human who typically interprets applicable metadata standards and examines the metadata record to assess adherence, subsequently suggesting corrections. Similarly, for a given record, we sought to replicate this process by directing the LLM to utilize the data dictionary NCBI , which includes the allowed list of metadata name–value pairs, their formats, and descriptions, to correct the record. This process is illustrated in Figure 3 for GPT-4. Input to