Supplementary Information
A General Pipeline for Digesting Scientific Literature into a Shared Scientific Knowledge Base · 2026
Read Supplementary Information by A General Pipeline for Digesting Scientific Literature into a Shared Scientific Knowledge Base (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27384.txt.
Introduction
Black describes a pipeline that digests published papers into a shared, queryable knowledge base. Published papers hold validated measurements in clashing formats and vocabularies. The Materials Explorer Pipeline extracts structured records from those papers, each carrying its measurements, source details, and a confidence score. Records stay portable and citable, so other scientists can build on them directly. Interactive browsing surfaces correlations and candidate hypotheses for expert review in this systems paper of 2026.
A test on recent superconducting qubit papers yields a corpus of two hundred thirty-three samples spanning ten material classes. The sources come from a national quantum research centre's community. Papers stop being a pile of PDFs and become a database. Cross-group comparison and hypothesis testing gain the shared resource they lacked. Extraction quality, not access, becomes the bottleneck.
Catalog record
| CREATOR | A General Pipeline for Digesting Scientific Literature into a Shared Scientific Knowledge Base |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27384
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supplementary Information A General Pipeline for Digesting Scientific Literature into a Shared Scientific Knowledge Base Abstract A General Pipeline for Digesting Scientific Literature into a Shared Scientific Knowledge Base C.T. Black ∗ a The published scientific literature is a rich, continuously growing record of measurements, correlations, and observations that modern AI tools can now make accessible in new ways. The Materials Explorer Pipeline digests collections of scientific papers into a structured, queryable database, producing sample records with full provenance and confidence, making them interactively explorable, and surfacing hypothesis candidates for scientist review. Each extracted record is a self-contained, portable unit of knowledge, carrying the measurements, research details, and source citations needed to use and cite the data appropriately. The Pipeline is demonstrated on recent superconducting qubit materials literature of the Co-design Center for Quantum Advantage, a DOE National Quantum Information Science Research Center, producing a corpus of 233 samples across 10 material classes. The Pipeline architecture is domain-agnostic and designed to be readily portable to other scientific domains. † † footnotetext: a Co-design Center for Quantum Advantage, Brookhaven National Laboratory, Upton, New York 11973, USA. E-mail: [email protected] † † footnotetext: † Supplementary Information available. See DOI: 00.0000/00000000. Introduction The published scientific literature has always been the primary record of what a field knows. The measurements and observations reported in peer-reviewed publications are a rich scientific resource that is already validated and citable. Word embedding studies have shown that latent useful scientific knowledge is encoded in this published record, pointing toward the possibility of extracting it systematically and at scale. 22 For scientists, working with the vast scientific literature in aggregate is a growing challenge. Results reported across hundreds of papers from dozens of groups, using varied terminology and formats, are difficult to compare, and no shared resource exists to support cross-group comparisons or systematic hypothesis testing against the full published record. Here, we describe the Materials Explorer Pipeline, which gives scientists a new way to visualize and make use of their collective knowledge, presenting measurements and derived quantities for further use, and surfacing correlations and hypothesis candidates to evaluate. In this way, the Pipeline is part of a broader opportunity for AI tools to extend scientific cognition. 26 The Pipeline sidesteps an impediment to community database adoption, which requires scientists to contribute data separately from publication, a task that most do not take on. Computationally-derived databases such as the Materials Project 12 have shown the high value of structured, queryable materials data at scale, but experimental property databases of comparable scope remain scarce. The Materials Explorer Pipeline can draw from papers already produced. A schema encoding the measurements of interest, paired with an automated ingestion pipeline, populates a structured database from the existing body of literature, with no burden on the contributors. Human review is built into every stage where AI makes an inference, ensuring that the digested corpus reflects scientific judgment. Throughout, data extraction is performed via repeated calls to a large language model. Early approaches to automated extraction of structured data from scientific publications framed the problem as named entity recognition on sentence fragments, achieving strong benchmark performance on materials science text but not addressing assembly of complete, multi-field records from full-length documents. 24 A significant advance later came from showing that fine-tuned language models can extract schema-defined records from materials science texts, 8 with the limitation that fine-tuning required hundreds of manually annotated examples per task and locked the schema at training time. Subsequent work showed that zero-shot prompting of frontier models can achieve comparable extraction quality without labeled training data. 17 , 1 A comprehensive review of this approach for chemistry and materials science, covering prompting strategies, multimodal methods, and evaluation, is available in the literature. 20 Most recently, a tiered extraction hierarchy using lightweight relevance triage before full structured extraction was applied to experimental records in shock physics, 19 an approach similar to the one employed in this work. The Materials Explorer Pipeline builds on this research progression, introducing a mandatory catchall block to the schema that captures knowledge not yet formalized into named fields, which allows the schema to grow more capable as the corpus accumulates. The Pipeline has four components that work together to build a shared corpus of scientific knowledge and make it useful (Fig. 1 ). The database schema guides knowledge encoding from the publications, determining the most important information to capture and enabling the downstream comparisons that give the data value. The Ingester analyzes each paper and populates the database. The browser-based Explorer visualizes the corpus and renders it interactively accessible. The corpus mining tool operates over the complete set of records, surfacing candidate scientific hypotheses together with supporting and complicating evidence. The corpus database is the central artifact the Pipeline produces, fully traceable to the source publications and growing continually as new papers are ingested. This Pipeline is demonstrated using superconducting qubit materials literature published in the last year by the Co-design Center for Quantum Advantage (C2QA) 7 , a DOE National Quantum Information Science Research Center, though the Pipeline architecture is domain-agnostic, as described below. Fig. 1: Materials Explorer Pipeline architecture . Publications are digested by the Ingester, which analyzes each paper and populates the Corpus. The Explorer makes the Corpus interactively accessible for visualization, search, and browsing. Corpus Mining surfaces candidate scientific hypotheses with supporting and complicating evidence. Human reviews and approvals appear after ingestion and AI reasoning in corpus mining. Schema Design The database schema is the intellectual heart of the Pipeline, encoding domain knowledge such that the sample information captured from each publication becomes immediately useful. The schema defines the initial determinations of what information gets recorded and thereby which comparisons across publications are enabled. Each database record corresponds to a single sample characterized in a publication and has a six-block structure: metadata, sample description, measurements, derived quantities, catchall, and similarity profile. Single papers can thus contribute many sample records. Together, these six blocks make each record a self-contained Portable Unit of Knowledge (PUK) (Fig. 2 ), which carries everything needed to understand, use, and appropriately cite a sample’s data independently of the Pipeline that produced it, and enabling records to move freely between institutions and research groups. A complete example PUK is in the Supplementary Information (S1). The metadata and sample description establish the identity and provenance of the record, including what material was studied, how it was made, and where the data came from. The measurements block records reported values, with derived quantities calculated from reported values using standard formulas. The Block 3 field definitions are provided in the Supplementary Information (S2). A catchall block holds relevant information that does not fit any current schema field. Each record also contains a similarity profile that enables corpus-wide search and comparison. Prior systems typically discard information that does not fit the schema at design time. The Pipeline’s catchall block is an architecture for capturing author-stated correlations, anomalous observations, and additional measurements. Fields appearing in the catchall across more than 5% of corpus samples are presented for review as potential candidates for promotion to named schema columns, after human approval. In this way, the schema evolves to better represent what the corpus contains as it grows.