The Biological Observation Matrix (BIOM) format or: how I learned to stop worrying and love the ome-ome.

McDonald D, Clemente JC, Kuczynski J, Rideout JR, Stombaugh J, Wendel D, Wilke A, Huse S, Hufnagle J, Meyer F, Knight R, Caporaso JG. · 2012

Read The Biological Observation Matrix (BIOM) format or: how I learned to stop worrying and love the ome-ome. by McDonald D, Clemente JC, Kuczynski J, Rideout JR, Stombaugh J, Wendel D, Wilke A, Huse S, Hufnagle J, Meyer F, Knight R, Caporaso JG. (2012) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc3626512.txt.

Introduction

Daniel McDonald and colleagues defined a standard file format for omics tables in GigaScience in 2012. Modern biology produces observation-by-sample tables on an industrial scale: which microbes appear in which gut sample, which genes in which tumor. Every tool kept its own format, blocking exchange and combined analysis. The Biological Observation Matrix, BIOM, pronounced biome, stores such tables in the widely used JSON text format, with room for notes about each row and column.

An open-source project supplies the software to read and write the format, starting with Python and inviting ports to other languages. Big microbial ecology platforms were early adopters, and the Earth Microbiome Project adopted BIOM as a standard, with the Genomic Standards Consortium listing it as a candidate. The authors aim squarely at the bioinformatics bottleneck: one shared envelope so tools built by strangers can pass data without translation.

Catalog record

CREATORMcDonald D, Clemente JC, Kuczynski J, Rideout JR, Stombaugh J, Wendel D, Wilke A, Huse S, Hufnagle J, Meyer F, Knight R, Caporaso JG.
YEAR2012
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC3626512

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Biological Observation Matrix (BIOM) format or: how I learned to stop worrying and love the ome-ome By Daniel McDonald, Jose C Clemente, Justin Kuczynski, Jai Ram Rideout, Jesse Stombaugh, Doug Wendel, Andreas Wilke, Susan Huse, John Hufnagle, Folker Meyer, Rob Knight, J Gregory Caporaso Published in: GigaScience DOI: 10.1186/2047-217X-1-7 ABSTRACT Abstract Background We present the Biological Observation Matrix (BIOM, pronounced “biome”) format: a JSON-based file format for representing arbitrary observation by sample contingency tables with associated sample and observation metadata. As the number of categories of comparative omics data types (collectively, the “ome-ome”) grows rapidly, a general format to represent and archive this data will facilitate the interoperability of existing bioinformatics tools and future meta-analyses. Findings The BIOM file format is supported by an independent open-source software project (the biom-format project), which initially contains Python objects that support the use and manipulation of BIOM data in Python programs, and is intended to be an open development effort where developers can submit implementations of these objects in other programming languages. Conclusions The BIOM file format and the biom-format project are steps toward reducing the “bioinformatics bottleneck” that is currently being experienced in diverse areas of biological sciences, and will help us move toward the next phase of comparative omics where basic science is translated into clinical and environmental applications. The BIOM file format is currently recognized as an Earth Microbiome Project Standard, and as a Candidate Standard by the Genomic Standards Consortium. Keywords: Microbial ecology, Comparative genomics, Metagenomics, QIIME, MG-RAST, VAMPS, BIOM BACKGROUND Advances in DNA sequencing have led to exponential increases in the quantity of data available for “comparative omics” analyses, including metagenomics (e.g., [1,2]), comparative genomics (e.g., [3]), metatranscriptomics (e.g., [4,5]), and marker-gene-based community surveys (e.g., [6,7]). With the introduction of a new generation of "benchtop sequencers" [8], accessible to small research, clinical, and educational laboratories, sequence-based comparative omic studies will continue to increase in scale. The rate-limiting step in many areas of comparative omics is no longer obtaining data, but analyzing that data (the “bioinformatics bottleneck”) [9,10]. One mechanism that will help reduce this “bioinformatics bottleneck” is standardization of common file formats to facilitate sharing and archiving of data [11]. As with the increasing prevalence of high-throughput technologies in the biological sciences, the categories of comparative omics data, which we collectively term the “ome-ome”, are rapidly increasing in number (Figure 1). Researchers are relying on more types of omics data to investigate biological systems, and the coming years will bring increased integration of different types of comparative omics data [2,12]. A common data format will facilitate the sharing and publication of comparative omics data and associated metadata and improve the interoperability of comparative omics software. Further, it will enable rapid advances in omics fields by allowing researchers to focus on data analysis instead of on formatting data for transfer between different software packages or reimplementing existing analysis workflows to support their specific data types. [Figure: Growth of the “ome-ome”, or the types of “omic” data, over time based on mentions in Medline abstracts. Chao1 analysis indicates that there may be over 3,000 “omes”: however, given the well-known limitations of such non-parametric extrapolation techniques, we can only wonder how many “omes” remain to be discovered as technological advances usher in a new era of “ome-omics”.] Despite the different types of data involved in the various comparative omics techniques (e.g., metabolomics, proteomics, or microarray-based transcriptome analyses), they all share an underlying, core data type: the “sample by observation contingency table”, or the matrix of abundances of observations on a per-sample basis. In marker gene surveys, this table contains counts of OTUs (Operational Taxonomic Units) or taxa on a per-sample basis; in metagenome analyses, counts of orthologous groups of genes, taxa, or enzymatic activities on a per-metagenome basis; in comparative genomics, counts of genes or orthologous groups on a per-genome basis; and in metabolomics, counts of metabolites on a per-sample basis. Many tools have been developed to analyze these contingency tables, but they are generally focused on a specific type of study (e.g., QIIME for marker gene analysis [13], MG-RAST for metagenome analysis [14], VAMPS for taxonomic analysis [15]). However, many techniques are applicable across data types, for example rarefaction analyses (i.e., collector curves). These are frequently applied in microbiome studies to compare how the rate of incorporation of additional sequence observations affects the rate at which new OTUs are observed. This allows us to determine whether an environment is approaching the point of being fully sampled (e.g., [13]). Rarefaction curves could similarly be applied in comparative genomics to study the rate of discovery of new gene families, as done in [16]; a researcher could compile a contingency table of genomes (samples) by genes (observations) and use a rarefaction curve to determine how quickly new gene families were accumulating as new genome sequences are added. A standard format for biological sample by observation contingency tables will support the use of bioinformatics pipelines for different data types than those they were initially designed for (e.g., QIIME could be applied to generate rarefaction curves for proteomic data, or MG-RAST could output metatranscriptome tables). Adoption of this standard will additionally facilitate the adoption of future analysis pipelines, as users can then directly apply those pipelines to their existing data. In many existing software packages (e.g., [13,14]), contingency tables are represented as tab-separated text, but minor syntactic differences prevent easy exchange of data between tools. For example, differing representation of samples and observations as either rows or columns, and the mechanism for incorporating sample or observation metadata (if possible at all), cause the formats used by different software packages to be incompatible. Additionally, in many of these applications a majority of the values (frequently greater than 90 %) in the contingency table are zero, which is taken to mean that the corresponding “observation” was not observed in the corresponding sample. The fraction of the table that has non-zero values is defined as the "density", and thus a matrix with a low number of non-zero values is said to have a low density. As data sets continue to increase in size, “dense” representations of these tables, where all values are represented (in contrast to “sparse” representations, where only non-zero values are represented), result in an increasingly inefficient use of disk space. For example, marker gene survey OTU tables with many samples (such as the one presented in Additional file 1: Table S1 containing 6,164 samples and 7,082 OTUs) can have as few as 1 % non-zero values. As the collection of samples becomes more diverse, these tables become even sparser and their size (both on disk and in memory) becomes a considerable barrier to performing meta-analyses. Sample and observation metadata are essential for the interpretation of omics data, and for facilitating future meta-analyses. Two projects have recently arisen to address the need for metadata standards: MIxS [17], which defines what metadata should be stored for diverse sequence types, and ISA-TAB [11], which defines a file format for storing that metadata. A standard file format for representing sample by observation contingency tables could compliment these existing standards by providing a means for associating MIxS-compliant metadata provided in ISA-TAB format with samples and observations. The Biological Observation

Read the complete text