Compendium Manager: a tool for coordination of workflow management instances for bulk data processing in Python
Richard J. Abdill, Ran Blekhman · 2025
Read Compendium Manager: a tool for coordination of workflow management instances for bulk data processing in Python by Richard J. Abdill, Ran Blekhman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11385.txt.
Introduction
Richard J Abdill and Ran Blekhman present a wrangler for bulk data pipelines, in a 2025 arXiv paper. Tools like Snakemake and Nextflow automate the steps of one gene study well. But merging many studies means launching and babysitting hundreds or thousands of runs at once, a chore those tools do not cover. Their Compendium Manager is a small command-line tool built for that scale.
It can set up, launch, and grade pipelines across a list of projects. It watches progress, loads results into a shared database, and logs fine processing stats for later checks. The aim is repeatable bulk work without a heavy platform. The authors pitch it to bioinformatics teams who pool public sequencing sets.
Catalog record
| CREATOR | Richard J. Abdill, Ran Blekhman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11385
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Compendium Manager: a tool for coordination of workflow management instances for bulk data processing in Python Richard J. Abdill, Ran Blekhman Abstract Compendium Manager is a command-line tool written in Python to automate the provisioning, launch, and evaluation of bioinformatics pipelines. Although workflow management tools such as Snakemake and Nextflow enable users to automate the processing of samples within a single sequencing project, integrating many datasets in bulk requires launching and monitoring hundreds or thousands of pipelines. We present the Compendium Manager, a lightweight command-line tool to enable launching and monitoring analysis pipelines at scale. The tool can gauge progress through a list of projects, load results into a shared database, and record detailed processing metrics for later evaluation and reproducibility. Keywords compendium, bioinformatics, microbiome, public data 1 Introduction The rise of high-throughput DNA sequencing confronted biologists with “big data” problems that they had previously been able to ignore. Biologists had performed software-based analysis tasks for decades (Ouzounis 2012; Ouzounis and Valencia 2003), including work with DNA sequences (Pearson and Lipman 1988; Altschul et al. 1990), but this new world, in which a single study’s DNA sequencing could generate terabytes of data, produced challenges impossible to address on consumer-grade workstations (Marx 2013). In the 2010s, as more biologists added computational skills and the number of sequencing projects expanded dramatically (Katz et al. 2022), bioinformatics software became more commonly built (Mangul et al. 2019), used, and cited (Wren 2016). Genomics processing frequently requires complicated, multi-step protocols in which the output of one (or more) tools is used as the input to subsequent tools, with many intermediate files being passed between processes. In smaller projects, it’s possible for a researcher to follow these steps (the “pipeline”) from a terminal window, monitoring progress and launching each step when ready. But it quickly becomes impractical to manually usher hundreds or thousands of samples through these pipelines in a consistent way. This concern was addressed with automation tools called workflow managers or bioinformatic pipeline frameworks (Leipzig 2017): Tools such as Snakemake (Köster and Rahmann 2012) operate by following information from a text file describing each step of the pipeline, such as the commands that must be run, resources required for the step, and how the environment should be configured. Users define their pipeline, which may have one or dozens of steps (Pohl et al. 2024), apply it to any collection of samples, and let the management software iterate through all steps for all samples. A common approach is to structure a pipeline to apply to a single project or sequencing run: tools such as DADA2 include experiment-specific error modeling (Callahan et al. 2016), for example. In ordinary circumstances, this is the usual way data would be processed anyway—however, researchers who seek to process many projects again encounter the problem of scale requiring an impractical level of human intervention. Meta-analysis efforts, which are becoming increasingly common as genomic data accumulates, may involve integrating hundreds or thousands of independent sequencing projects, each requiring its own pipeline instance with project-specific parameters and error handling. Although workflow managers like Snakemake excel at automating steps within a pipeline, they lack built-in capabilities for coordinating multiple independent pipeline instances across different projects, monitoring their collective progress, and consolidating results into a unified database. This gap often leads researchers to rely on ad-hoc scripts, manual monitoring, and fragmented approaches to project management. This creates bottlenecks that limit the scale of integrative analyses and introduces inconsistencies in how datasets are processed. To address this gap, we developed the Compendium Manager, a lightweight yet robust orchestration layer that provides automated coordination of multiple workflow instances, centralized error handling, and integrated result management for large-scale bioinformatics meta-analyses. Several tools are already available to address portions of this situation: The snk package can generate customizable command-line interfaces for conventional Snakemake pipelines (Wirth, Mutch, and Turnbull 2024), and Snakemaker uses large language models to parse command-line activity and generate pipeline code (Masera et al. 2025). It is possible to use Snakemake itself to process multiple projects independently, but when processing all projects could take weeks or months given resource constraints, having a single Snakemake process monitoring all of this work would be impractical and error-prone, particularly in a shared high-performance computing environment that likely imposes time restrictions. Existing compendia solve this orchestration issue in several ways: the recount3 project, a dataset of 730,000 human and mouse RNA-seq samples (Wilks et al. 2021), also uses Snakemake to process individual projects. These individual project pipelines are coordinated using their “Monorail” system, which distributes work across nodes either in the cloud (e.g. Amazon Web Services servers) or on HPC clusters before a “unify” step collects data together. The ARCHS4 project also used cloud resources; it used processing tasks automated via Python and SQLite (Lachmann et al. 2018), as the Compendium Manager does. However, its suite of scripts does not include a user interface for performing tasks such as processing projects or evaluating results (Clarke, Ma’ayan, and Lachmann 2020). The curatedMetagenomicData package, which contains data (and manually curated metadata) for more than 20,000 shotgun samples (Pasolli et al. 2017), has documentation describing a pipeline in which each step is launched manually by a curator (Home, n.d.). The GMrepo project also used DADA2 but did not specify any management-level automation (Dai et al. 2022). The Single-cell Pediatric Cancer Atlas automated project-level processing and merging using Nextflow (Di Tommaso et al. 2017), but does not describe compendium-level automation (Hawkins et al. 2024). 2 Compendium Manager overview Despite the utility of workflow management tools in a high-performance computing environment, there is no broadly accepted solution for “automating the automation”—that is, coordinating many Snakemake pipelines operating independently of each other. Projects seeking to integrate many projects together may still use Snakemake, but the coordination of those pipelines is done manually (Valderrama et al. 2025). The Compendium Manager software addresses these issues by providing a customizable platform that performs the following tasks: • Recording and organizing metadata for a corpus of samples split into projects • Provisioning project-specific pipeline environments • Tracking each project’s pipeline progress, errors encountered and the automated changes made to address them • Providing a command-line interface to monitor the status of computational jobs and compendium progress, plus methods for processing new projects • Storing project results in a database, providing a streamlined source of data for generating compendium files and integrated results The current codebase and example Snakemake workflow reflects its use in the Human Microbiome Compendium project for processing 16S rRNA gene amplicon sequencing datasets (Abdill et al. 2025), but each of the Compendium Manager’s tasks—starting new pipelines, monitoring progress, responding to errors, evaluating results, determining required re-processing steps, and so on—can be expanded to an arbitrary level of complexity for other types of data. In general, the Compendium Manager’s approach is straightforward: Given the results of a BioProject search, the software builds a database of projects and their associated samples. Then, either automatically or at the prompting of a user, the tool will generate the required files for a given project, launch a Snakemake pipeline to process its files. If the final step in the Snakemake process is configured to report progress back to the Compendium Manager software, the