Genomes on a Tree (GoaT): A versatile, scalable search engine for genomic and sequencing project metadata across the eukaryotic tree of life.

Challis R, Kumar S, Sotero-Caio C, Brown M, Blaxter M. · 2023

Read Genomes on a Tree (GoaT): A versatile, scalable search engine for genomic and sequencing project metadata across the eukaryotic tree of life. by Challis R, Kumar S, Sotero-Caio C, Brown M, Blaxter M. (2023) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc9971660.txt.

Introduction

Challis and co-authors introduce Genomes on a Tree, a search engine for genome project metadata. The Earth BioGenome Project aims to sequence all roughly 1.9 million described eukaryotic species. Many regional and taxon projects must coordinate to reach that goal. Genome sizes and karyotypes sit scattered across the literature.

GoaT stores such metadata in an Elasticsearch-powered datastore and index. It covers all eukaryotic species and fills gaps by phylogenetic comparison. It also holds target priority and sequencing status for linked projects. Users can query through an API, a web front end, or a command line. Visual summaries aid browsing on the web side. The tool helps planners avoid duplicate effort at global scale.

Catalog record

CREATORChallis R, Kumar S, Sotero-Caio C, Brown M, Blaxter M.
YEAR2023
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC9971660

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Genomes on a Tree (GoaT): A versatile, scalable search engine for genomic and sequencing project metadata across the eukaryotic tree of life By Richard Challis, Sujai Kumar, Cibele Sotero-Caio, Max Brown, Mark Blaxter Published in: Wellcome Open Research DOI: 10.12688/wellcomeopenres.18658.1 ABSTRACT As genomic data transform our understanding of biodiversity, the Earth BioGenome Project (EBP) has set a goal of generating reference quality genome assemblies for all ~1.9 million described eukaryotic taxa. Meeting this goal requires coordination among many individual regional and taxon-focussed projects working under the EBP umbrella. Large-scale sequencing projects require ready access to validated genome-relevant metadata, such as genome sizes and karyotypes, but these data are dispersed across the literature, and directly measured values are lacking for most taxa. To meet these needs, we have developed Genomes on a Tree (GoaT), an Elasticsearch-powered datastore and search index for genome-relevant metadata and sequencing project plans and statuses. GoaT indexes publicly available metadata for all eukaryotic species and interpolates missing values through phylogenetic comparison. GoaT also holds target priority and sequencing status information for many projects affiliated to the EBP to aid project coordination. Metadata and status attributes in GoaT can be queried through a mature API, a web front end, and a command line interface. The web front end additionally provides summary visualisations for data exploration and reporting (see https://goat.genomehubs.org). GoaT currently holds direct or estimated values for over 70 taxon attributes and over 30 assembly attributes across 1.5 million eukaryotic species. The depth and breadth of curated data, frequent updates, and a versatile query interface make GoaT a powerful data aggregator and portal to explore and report underlying data for the eukaryotic tree of life. We illustrate this utility through a series of use cases from planning through to completion of a genome-sequencing project. INTRODUCTION The Earth BioGenome Project (EBP) aims to sequence the complete genomes of every one of the ~1.9 million described eukaryotic species over the next 10 years 1 . This challenge is being met by an engaged community of individual EBP partner projects with lists of target species defined by geographic region, taxonomic clade, or based on criteria of economic importance, conservation status, or ecosystem service. The nature of species distributions coupled with the diverse definitions of these partner projects results in overlapping target species lists. Coordination of sequencing efforts across the different projects is essential to prevent duplication, to focus efforts on underrepresented branches of the tree of life, and to track the overall progress of the EBP. Public databases of the International Nucleotide Sequence Database Collaboration (INSDC: the National Centre for Biotechnology Information’s GenBank, the European Bioinformatics Institute’s European Nucleotide Archive ENA, and the DNA Database of Japan) 2 provide resources for the storage of, and access to, submitted reads and assembled genome sequences. While these are the databases-of-record for completed genome projects, it is not the role of the INSDC to coordinate planning of future genome sequencing projects. Until recently, the small scale of eukaryotic genome sequencing efforts meant that there was minimal duplication of effort. As EBP projects scale up to sequencing hundreds of thousands of species, the need for a global platform for registering interest and reporting progress in individual species becomes more pressing. The Genomes OnLine Database (GOLD) 3 registers the intent and status of ongoing and complete genome sequencing projects. GOLD was primarily developed by the US Department of Energy Joint Genomics Institute (JGI) to serve their in-house bacterial, algal, fungal and metagenome sequencing projects. It also periodically imports eukaryotic genome sequencing data from GenBank, but does not aim to be a wide-scope resource to collate genome metadata and live updates from projects from a diversity of sources. EBP partner projects require access to a diverse suite of genome-relevant metadata to shape and scope sequencing efforts. Estimates of genome size such as those collected by genomesize.com 4 and the Kew Plant C Values database 5 are valuable for target species selection and determining the amount of raw sequencing data needed to achieve the desired coverage. Karyotypic information and modes of sexual reproduction are also useful to genome curators when piecing together chromosomal assemblies and defining sex chromosomes 6 . Ideally, data for each attribute would be available for every taxon. However, genome-relevant metadata are sparse and unevenly distributed, with strong taxonomic biases. As a major aim of biodiversity genomics projects is to fill the gaps in the tree of life by sequencing underrepresented taxonomic groups, many of the initial targets of EBP-affiliated projects are among the least likely to have direct measurements of genome size and karyotype available. Coordination of EBP sequencing efforts thus requires a system that automates inference of missing values to provide searchable estimated values for all core attributes across every species. We have developed Genomes on a Tree (GoaT) to offer public access to plans and progress in genome sequencing at the Tree of Life programme at the Wellcome Sanger Institute, and to offer coordination across biodiversity genomics projects in the wider EBP network. GoaT is built on a flexible, high performance platform called GenomeHubs 7 that indexes metadata, infers missing values, and allows rapid querying across indexed attribute values for biological taxa at any taxonomic level. GoaT is a specialised instance of the GenomeHubs platform for genome project planning and status tracking. However, GoaT can also be used as an analytical engine to discover new patterns in genomic features across taxa as the quantity and quality of available metadata increases. Here we describe the core functioning of GoaT and how we import taxon-specific metadata into GoaT. We show how we use the underlying GenomeHubs platform to make taxonomically informed inferences and support querying across any combination of attributes. We summarise the data available in GoaT at the time of writing and present a set of worked use cases including examples of how GoaT is being used to plan sequencing projects, how other EBP sequencing projects can use GoaT, and how GoaT can be used as a taxonomically-informed search engine. We end with a discussion of how GoaT has rapidly grown to meet the needs of the eukaryotic genome sequencing community, and how we plan to keep extending GoaT to index additional metadata across hundreds of thousands of species. METHODS Core architectural design GoaT has been implemented as the first instance of our recently updated GenomeHubs codebase ( https://github.com/genomehubs/genomehubs). GenomeHubs provides a flexible data model, search implementation and user interface that has been customised to deliver GoaT through the Configuration-as-Code (CaC) paradigm, where the same code can be extended to different data sources, data types, front ends, and use cases based on human-readable configuration files in the code repository. In the following sections we outline the GoaT-specific customisations and describe the underlying GenomeHubs implementation that GoaT is built upon. Data sources and curation The metadata stored in GoaT are imported from diverse sources. The original data are retrieved in a variety of formats. GoaT currently imports genome and species metadata data dumps from source databases, retrieves information from public APIs, and reads tabular files from manuscripts. To report genome project activity, GoaT uses resources provided by each individual project within the EBP network, largely accessing online spreadsheets that report sequencing planning and progress.

Read the complete text