Cooltools: Enabling high-resolution Hi-C analysis in Python
Nezar Abdennur, Sameer Abraham, Geoffrey Fudenberg, Ilya M Flyamer, Aleksandra A Galitsyna, Anton Goloborodko, Maxim Imakaev, Betul A Oksuz, Sergey V Venev, Yao Xiao, Maxwell Wing Libbrecht
Read Cooltools: Enabling high-resolution Hi-C analysis in Python by Nezar Abdennur, Sameer Abraham, Geoffrey Fudenberg, Ilya M Flyamer, Aleksandra A Galitsyna, Anton Goloborodko, Maxim Imakaev, Betul A Oksuz, Sergey V Venev, Yao Xiao, Maxwell Wing Libbrecht in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc11098495.txt.
Introduction
Abdennur and colleagues present a Python toolkit for fine-scale maps of genome folding in this 2024 paper. Hi-C is a method that measures which DNA regions sit near each other inside the cell. Its maps keep growing in size, depth, and detail. Analysis tools had not kept pace, straining storage, memory, and researcher time.
Their suite, cooltools, builds on the cooler file format made for such maps. It offers both a command line tool for computing clusters and a Python interface for interactive work. The design stresses flexible, scalable, and reproducible analysis that users can adapt to new questions. The code is shared openly online. The authors position it for the largest current folding datasets, where older tools stall.
Catalog record
| CREATOR | Nezar Abdennur, Sameer Abraham, Geoffrey Fudenberg, Ilya M Flyamer, Aleksandra A Galitsyna, Anton Goloborodko, Maxim Imakaev, Betul A Oksuz, Sergey V Venev, Yao Xiao, Maxwell Wing Libbrecht |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cooltools: Enabling high-resolution Hi-C analysis in Python By Nezar Abdennur, Sameer Abraham, Geoffrey Fudenberg, Ilya M Flyamer, Aleksandra A Galitsyna, Anton Goloborodko, Maxim Imakaev, Betul A Oksuz, Sergey V Venev, Yao Xiao, Maxwell Wing Libbrecht Published in: PLOS Computational Biology DOI: 10.1371/journal.pcbi.1012067 ABSTRACT Abstract Chromosome conformation capture (3C) technologies reveal the incredible complexity of genome organization. Maps of increasing size, depth, and resolution are now used to probe genome architecture across cell states, types, and organisms. Larger datasets add challenges at each step of computational analysis, from storage and memory constraints to researchers’ time; however, analysis tools that meet these increased resource demands have not kept pace. Furthermore, existing tools offer limited support for customizing analysis for specific use cases or new biology. Here we introduce cooltools (https://github.com/open2c/cooltools), a suite of computational tools that enables flexible, scalable, and reproducible analysis of high-resolution contact frequency data. Cooltools leverages the widely-adopted cooler format which handles storage and access for high-resolution datasets. Cooltools provides a paired command line interface (CLI) and Python application programming interface (API), which respectively facilitate workflows on high-performance computing clusters and in interactive analysis environments. In short, cooltools enables the effective use of the latest and largest genome folding datasets. "This is a PLOS Computational Biology Software paper." INTRODUCTION Chromosome conformation capture technologies are powerful tools for experimentally characterizing higher-order genome organization [1]. Multiple labs and consortia, including the 4D Nucleome project (4DN) [2], the International Nucleome Consortium (INC), and ENCODE [3], are generating increasingly larger datasets to probe genome organization at ever higher resolutions. Larger datasets increase the challenges at each step of computational analysis, from storage to memory, to researchers’ time—at the highest resolutions, the data for even a single chromosome does not fit easily in memory. Computational analyses of genome folding typically employ multiple approaches to quantify the variety of patterns in contact maps. These include: (i) enrichment of contacts within chromosomes, referred to as chromosome territories, (ii) decreasing contact frequency versus distance within chromosomes, or P(s); (ii) plaid patterns of preferential contacts, referred to as compartmentalization, (iii) squares of enriched contact frequency, referred to as topologically associated domains (TADs), (iv) focal peaks of pairwise contact frequencies, referred to as loops or dots. To quantitatively describe these features, and assay how they change across conditions, maps are summarized in many ways. Summaries include 2D pairwise averages, 1D profiles, and lists of genomic positions. The variety of Hi-C features and summaries benefits from a set of modular tools, rather than end-to-end pipelines, for data analysis. The features in genome folding maps vary substantially through cell states, across organisms, and between protocols. As C-data is obtained from a growing variety of contexts, computational analyses must co-evolve and expand as well. Because of this, computational researchers require APIs that can not only perform standard analyses but also enable the design of new analyses. This includes flexibility over parameters and thresholds, but also encompasses more dramatic extensions. Python provides one of the most popular environments for flexible data analysis and software development. This is in part due to the rich ecosystem of numpy [4], scipy [5], scikit-learn [6], and pandas [7]. For Hi-C analysis in Python, the widely-adopted cooler [8] format and library present an ideal foundation. Cooler handles storage and access for high-resolution datasets via a sparse data model. To fully leverage the advantages of cooler, however, requires a corresponding analysis framework in Python. While there are existing computational suites for extracting features of genome folding via the CLI [9–13] or an R API [14], only FAN-C [13] and TADbit [11] have Python APIs (Table 1). However, FAN-C and TADbit neither take full advantage of the Python ecosystem nor provide consistent low-level access to the data, limiting their flexibility. We note the TADbit framework was developed before cooler was available, and hence without sparse storage considerations, and had an orthogonal primary focus on building 3D models. [Table: Overview of software suites for C-data analysis.] Here we introduce cooltools, a suite of computational methods that enable the analysis of high-resolution genome folding data accessible using cooler (Fig 1). Cooltools features a Python API paired with command line tools. The methods in cooltools are modular and provide access to both high-level and low-level APIs. The high-level API enables users to reproducibly perform routine analysis. The low-level API allows for troubleshooting challenges posed by new organisms or cell states and developing new advanced analyses. cooltools is part of a research community and broader ecosystem of open-source chromosome data analysis tools, Open2C (https://open2c.github.io/). cooltools provides detailed introductions to key concepts in Hi-C-data analysis with interactive notebook documentation. Collectively, cooltools enables computational analysis and methods development to keep pace with the rapidly-increasing quantity of experimental data and rapidly-evolving understanding of genome organization. [Figure: Overview of cooltools functionality.Open2C provides a modular ecosystem of software libraries for Hi-C analysis (highlighted with gray boxes). Pairtools [58] takes in paired-end sequence alignments and extracts contact pairs in the 4DN.pairs format. cooler [8] bins these contact pairs and stores the resulting sparse matrices in.cool and.mcool formats. The nextflow pipeline distiller [59] converts sequencing reads from FASTQ files directly into binned and normalized cooler files, integrating read alignment with pairtools and cooler. The library introduced in this paper, cooltools (in bold), provides methods to quantify and extract features from high-resolution contact maps stored by cooler.] Design and implementation The design of cooltools stems from its goal to enable flexible analyses of large quantities of high-resolution Hi-C data. - Provide a modular set of tools with a balance between comprehensiveness and flexibility. Modular tools can be used separately or combined in a custom order. The set of core tools extracts and quantifies the majority of common features in the Hi-C literature. - Separate processing from data analysis. Cooltools works with processed Hi-C data and is not concerned with mapping workflows. - Separate data analysis from plotting, with a focus on the former. For Hi-C data, plotting often has a greater range of user-specific needs than feature extraction and quantification. - Provide a paired Python API and CLI. CLI tools enable their use in computational pipelines and have a one-to-one correspondence with user-facing Python API functions. The user-facing API and CLI both have a standard format for arguments and take a cooler as their first input. For other types of genomic data besides Hi-C, the cooltools API accepts pandas dataframes, while the CLI reuses existing formats when possible: BED-like files for interval data and bedgraphs or bigwigs for genomic tracks. - Split the Python API into two layers: a high-level API for user-facing functions and a flexible low-level APIs. A modular, consistent, and well-documented internal Python API allows experienced users to customize existing analysis routines or create entirely new ones. - Enable analyses for custom genomes and non-model organisms. This includes user-defined subsets of regions from a reference genome, e.g. chromosomal arms. Leveraging its use of standard Python data structures, cooltools enables these analyses with bioframe [15]. A view specifies a unique 1D coordinate axis with an ordered set of non-overlapping genomic intervals, also referred to as regions. - Leverage indexed sparse storage. Cooltools is built directly on top of the cooler storage format and library, which allows it to operate on