Managing, Analyzing and Sharing Research Data with Gen3 Data Commons

Craig Barnes, Kyle Burton, Michael S. Fitzsimons, Hara Prasad Juvvala, Brienna Larrick, Ao Liu, Clint Malson, Noah Metoki-Shlubsky, Christopher Meyer, Andrii Prokhorenkov, Jawad Qureshi, Radhika Reddy, Pauline Ribeyre, L. Philip Schumm, Mingfei Shao, Trevar Simmons, Alexander VanTol, Peter Vassilato · 2025

Read Managing, Analyzing and Sharing Research Data with Gen3 Data Commons by Craig Barnes, Kyle Burton, Michael S. Fitzsimons, Hara Prasad Juvvala, Brienna Larrick, Ao Liu, Clint Malson, Noah Metoki-Shlubsky, Christopher Meyer, Andrii Prokhorenkov, Jawad Qureshi, Radhika Reddy, Pauline Ribeyre, L. Philip Schumm, Mingfei Shao, Trevar Simmons, Alexander VanTol, Peter Vassilato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04944.txt.

Introduction

Barnes and co-authors lay out Gen3, a free tool set for shared data banks. A data bank is a cloud home where a group holds, works, and shares data. Gen3 has built more than a dozen such banks to date. In sum they hold past 28 PB of data in store. Near 64 million fair-use data parts sit in those banks. To start a bank, a team first sets its data shape.

Gen3 then builds three parts on its own from that shape. One part lets users search and browse the bank holdings. One part lets users send new data in clean form. One part gives fair code paths for machine reads of data. A small set of stock parts sits under all banks. That base lets banks trade data with peer tools.

Catalog record

CREATORCraig Barnes, Kyle Burton, Michael S. Fitzsimons, Hara Prasad Juvvala, Brienna Larrick, Ao Liu, Clint Malson, Noah Metoki-Shlubsky, Christopher Meyer, Andrii Prokhorenkov, Jawad Qureshi, Radhika Reddy, Pauline Ribeyre, L. Philip Schumm, Mingfei Shao, Trevar Simmons, Alexander VanTol, Peter Vassilato
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Managing, Analyzing and Sharing Research Data with Gen3 Data Commons Craig Barnes, Kyle Burton, Michael S. Fitzsimons, Hara Prasad Juvvala, Brienna Larrick, Ao Liu, Clint Malson, Noah Metoki-Shlubsky, Christopher Meyer, Andrii Prokhorenkov, Jawad Qureshi, Radhika Reddy, Pauline Ribeyre, L. Philip Schumm, Mingfei Shao, Trevar Simmons, Alexander VanTol, Peter Vassilatos, Aarti Venkat, Robert L. Grossman Abstract Abstract Gen3 is an open-source data platform for building data commons. A data commons is a cloud-based data platform for managing, analyzing, and sharing data with a research community. Gen3 has been used to build over a dozen data commons that in aggregate contain over 28 PB of data and 64 million FAIR data objects. To set up a Gen3 data commons, you first define a data model. Gen3 then autogenerates 1) a data portal for searching and exploring data in the commons; 2) a data portal for submitting data to the commons; and 3) FAIR APIs for accessing the data programmatically. Gen3 is built over a small number of standards-based software services, which are designed to support current and future Gen3 components so that Gen3 can interoperate with other data platforms and data ecosystems. Despite the well understood importance and benefits of sharing research data, it is still a challenge for most research projects and communities to share the data that their researchers generate in a way that is integrated with the tools, applications, and software services used by their community. A data commons is a cloud-based data platform with a governance structure to manage, analyze and share data to support a research community [ 1 ] . Data commons generally clean and curate submitted data, align it with a data model, and sometimes harmonize it by processing it with a common set of workflows. A data commons also supports the interactive exploration of its data by integrating the analysis and visualization tools and services that its community uses. In this article, we describe Gen3, which is an open-source data platform for building and operating data commons [ 2 ] . Gen3 has been used to develop and operate over 15 data commons, which in aggregate, make over 28 PB of data available to the research community, spanning more than 64 million FAIR data objects. Gen3 has been used as part of the infrastructure for large-scale data commons, such as the NIH National Cancer Institute (NCI) Cancer Research Data Commons (CRDC) [ 3 , 4 ] , the NIH National Heart, Lung, and Blood Institute (NHLBI) BioData Catalyst data platform [ 5 ] , and the NIH National Institute of Biomedical Imaging and Bioengineering Medical Imaging and Data Resource Center (MIDRC) [ 6 ] . Importantly, Gen3 has also been used to set up smaller scale data commons to support particular research communities, such as the liquid biopsy research community [ 7 ] , the Connecting Health Outcomes Research Data Systems (CHORDS) research community ( chordshealth.org/discovery ), and a data commons to support COVID-19 research during the pandemic for the Chicago region [ 8 ] , which is now focused on the study of Long COVID. See Figure 1 for an example of a Gen3 data commons from a user perspective. Gen3 supports cloud-based data objects, structured data, and semi-structured data. All three types of data are managed by Gen3 in a FAIR manner [ 9 ] . High-level features of a Gen3 system include: 1. Native FAIR [ 9 ] support for structured, semi-structured, and unstructured data. Gen3 provides APIs so that structured data from databases, semi-structured JSON-based data, and unstructured data objects are all managed as findable, accessible, interoperable, and reusable (FAIR) data [ 9 ] . 2. Data model driven architecture. Gen3 is a data-model driven architecture in the sense that once a data commons administrator defines a YAML-based data model, the Gen3 software auto-generates FAIR APIs [ 9 ] for the data, as well as a web-based portal for discovering, exploring data, and submitting data to the commons. Whenever the data model is updated, the FAIR APIs and data portals are auto-generated again. This approach reduces the effort required to build data commons and keep them up to date as their data changes over time. This is especially important for smaller projects, which often do not have a dedicated data scientist for data curation and ingestion. 3. Cloud-native with elastic scalability. Gen3 is cloud native and uses the elastic scale-out properties of cloud computing infrastructure to manage petabytes or tens of petabytes of genomic, imaging, and other large-scale biomedical data. 4. Narrow middle architecture. The design of Gen3 is based upon the end-to-end system design principle [ 10 ] that is the architectural basis of the internet. An end-to-end distributed system design for data platforms is based upon a small number of distributed software services that are used to support applications that produce data for the system at one end and consume data for the system at the other end [ 11 ] . With this approach, it is easier to add new sources of data (at one end) and new applications for working with data (at the other end) over time, without changing the core architecture of the system. 5. Life cycle support for data. Gen3 provides life cycle support for projects. All data objects managed by Gen3 data commons are assigned persistent digital IDs compliant with the GA4GH DRS data standard [ 12 ] . Gen3 persistent identifiers are opaque and do not specify the physical location of the data in a particular cloud, making it easy to move the data from one cloud-based location to another. Also, Gen3 structured data can be exported in a self-describing, encapsulated format [ 13 ] that can be versioned, assigned a persistent Gen3 identifier, and then imported at a later time into another data commons. In the remainder of the article, we describe the key features and overall architecture of Gen3 and how it is used to enable research and leverage the very large datasets that are now the norm in most branches of science. Figure 1: Overview of how a user interacts with the Gen3 Data Portal. A) Users begin on a landing page that summarizes the data content; B) The Discovery page provides discoverability of project level summary data; C) The Exploration Page allows users to explore data, build cohorts or create subsets of data, and find files of interest based on their selection criteria; D) Users can programmatically query data on the Query page or through an API query; E) The Dictionary Viewer allows users to explore the types of data present in the system and helps data submitters understand how to structure data for submission; and F) Workspaces leveraging both commercial (e.g. Stata) and non-commercial (e.g. R or Python) software can be used to analyze data. Note that not all of these services need be deployed for a given system. Results Gen3 Data Commons Gen3 is currently used by over a dozen different organizations around the world to develop and operate data commons. Organizations that operate Gen3 data commons include universities, academic medical centers, industry, and government agencies. As an example, the Center for Translational Data Science, the primary developer and maintainer of Gen3, operates ten data

Read the complete text