A Proposed End-To-End Principle for Data Commons
Robert L Grossman · 2025
Read A Proposed End-To-End Principle for Data Commons by Robert L Grossman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11857.txt.
Introduction
Robert Grossman proposes an end-to-end design principle for the shared data platforms research communities build. A data commons parks data beside cloud computing and the usual tools for managing, analyzing, and sharing it, forming one interoperable resource. Borrowing the end-to-end argument from systems design, he frames a narrow middle architecture. Keep the shared core thin and stable, push intelligence to the edges, and let standards hold the federation together while community tools evolve freely around it.
Catalog record
| CREATOR | Robert L Grossman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11857
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Proposed End-To-End Principle for Data Commons Robert L Grossman Abstract A data commons brings together (or co-locates) data with cloud computing infrastructure and commonly used software services, tools and applications for managing, analyzing and sharing data to create an interoperable resource for a research community. We introduce an architectural design principle for data commons called the narrow middle architecture that is broadly based upon the end-to-end argument in systems design. We also discuss important core services for data commons and the role of standards. Introduction. There are now several efforts that are building data commons for biomedical data, including the NCI Genomic Data Commons (GDC), the NIH Data Commons Pilot Consortium (DCPPC), and the NCI Cancer Research Commons (CRDC). I like to think of a data commons in the following way: a database organizes data for a project; a data warehouse organizes data for an organization; and a data commons organizes data for a field or discipline. More formally, a data commons brings together (or co-locates) data with cloud computing infrastructure and commonly used software services, tools and applications for managing, analyzing and sharing data to create an interoperable resource for a research community [ 1 ] . In a Medium post by Josh Denny, David Glazer, Benedict Paten, Anthony Philippakis and myself [ 3 ] , we proposed four governing principles for building data commons and related data platforms: (1) modular, composed of functional components with well-specified interfaces; (2) community-driven, created by many groups to foster a diversity of ideas; (3) open, developed under open-source licenses that enable extensibility and reuse, with users able to add custom proprietary modules as needed; and (4) standards-based, consistent with standards developed by coalitions such as the Global Alliance for Genomics and Health (GA4GH). In this article, I would like to discuss principles (1) and (4) in more detail. In particular, I would like to discuss: how many modular components with well-specified interfaces are needed for a data commons and when should standards for these components be developed? I want to emphasize that the opinions below are mine alone, and the community of data commons developers is still actively debating these and other principles. In particular, these opinions do not necessarily reflect the views of the projects that I am working on. Before we begin, let us review two principles that are important when building distributed applications that support a community of users. The end-to-end design principle. The end-to-end principle [ 4 ] is one of the design principles underlying the internet. It states that network features should be implemented as close to the end points of the network — the applications — as possible. This is commonly expressed by describing the system as a “dumb” network with “smart” applications at the endpoint. What this means in practice is that as new innovative applications are developed for creating new types of content (one “end”) and for giving viewers access to the content (the other “end”), the intermediate routers and switches do not need to be updated. A good example is provided by streaming media which is a new type of content that required new types of applications but did not require any changes to the core routers and switches that support the internet. Lawrence Lessig and Robert W. McChesney argue that the end-to-end principle is what has made the Internet such a success, since “All of the intelligence is held by producers and users, not the networks that connect them [ 2 ] .” The principle of avoiding premature standardization. I find it helpful to think of standardization along a temporal spectrum. One end of the spectrum is when well accepted interfaces and specifications are standardized. The other end of the spectrum is when an organization or group develops a new standard to encourage the emergence of a new community. The difference is how many implementations are available, how long have they been used, and how widely adopted they are. The critical question is when in the lifecycle of a technology is the appropriate time to standardize? If too early, innovation is put at risk or the standard is likely to be ignored; if too late, the overall benefit is less. We now discuss two possible architectures for a data commons: a standards-driven architecture and a narrow-middle architecture. Standards driven architecture. With a standard driven architecture, a list of standards is developed at the beginning of the project, and the goal is to develop software consistent with the standards. This works well with fields that are relatively mature. The challenge with data commons is that the field is moving ahead quickly and the technologies for importing, curating, and integrating data are still maturing, as are the processes for exploring, analyzing, and collaborating with data. Narrow middle architecture. With what you might call a narrow middle architecture, the fewest possible services are used for the core of the system (“the narrow middle”) and only these services are standardized. See Figure 1 . This is an example of an end-to-end design. With this approach, the importing, curation and integration services for getting data into the commons (one “end”) and the data exploration, analysis and collaboration services for getting knowledge out of the commons (the other “end”) are not standardized, but instead are the focus of different competing and innovative efforts until the community begins to recognize and adopt approaches that seem to be most effective. Figure 1: The narrow middle architecture Core services. A good question to ask is: what are the minimum services that would enable the development and operation of a data commons? I argue that only four are needed: 1. Permanent Digital IDs, which given a globally unique ID (GUID) return the one or more locations that contain the corresponding data object. We assume that the underlying data objects comprising datasets of interest are stored in public and private clouds that are accessible through S3-compatible APIs. 2. A metadata service, which given a GUID returns a core set of metadata. 3. Authentication and authorization services, which enable the authentication of users and also can determine whether a given authenticated user is authorized to see a particular dataset. In this category, I also include the associated security and compliance services. 4. Data model services for defining a data model, and for validating, submitting and querying data with respect to this data model. Given these four services, a data commons can be developed. For example, the GDC developed a graphQL-based API over these services based upon an extensible data model. A variety of third party tools have also been built over this API, creating the beginning of an ecosystem. Several other data commons have also been built using this approach, including the BloodPAC Data Commons. The GDC serves over 100,000 unique users a year, a data portal, a data exploration portal, a data submission portal, several million API calls a day and multiple petabytes of data over these core services. Note that the GDC does not currently use a separate metadata service, but returns metadata through its API, but could easily create a simple metadata service over its current API. When the GDC was developed during 2014–2016, Jupyter notebooks were not very common