Efficient Chromosome Parallelization for Precision Medicine Genomic Workflows

Daniel Mas Montserrat, Ray Verma, Míriam Barrabés, Francisco M. de la Vega, Carlos D. Bustamante, Alexander G. Ioannidis · 2025

Read Efficient Chromosome Parallelization for Precision Medicine Genomic Workflows by Daniel Mas Montserrat, Ray Verma, Míriam Barrabés, Francisco M. de la Vega, Carlos D. Bustamante, Alexander G. Ioannidis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15977.txt.

Introduction

Daniel Mas Montserrat, Ray Verma, and four co-authors make large genomic workflows for precision medicine run faster by managing memory per chromosome. One sample can span hundreds of gigabytes, and memory spikes, heavy disk use, and failures plague pipelines. Chromosomes differ greatly in memory demand, so fixed resource plans waste resources and stretch run times.

A symbolic regression model forecasts per-chromosome memory use, with a bias against over-allocation. A dynamic scheduler packs tasks like a knapsack problem using simpler forecasts. A static scheduler orders chromosome work to cut peak memory with no loss of speed. On tests in simulation and real pipelines, the methods cut memory overruns, balance load, and speed the full run.

Catalog record

CREATORDaniel Mas Montserrat, Ray Verma, Míriam Barrabés, Francisco M. de la Vega, Carlos D. Bustamante, Alexander G. Ioannidis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Chromosome Parallelization for Precision Medicine Genomic Workflows Daniel Mas Montserrat, Ray Verma, Míriam Barrabés, Francisco M. de la Vega, Carlos D. Bustamante, Alexander G. Ioannidis Abstract Large-scale genomic workflows used in precision medicine can process datasets spanning tens to hundreds of gigabytes per sample, leading to high memory spikes, intensive disk I/O, and task failures due to out-of-memory errors. Simple static resource allocation methods struggle to handle the variability in per-chromosome RAM demands, resulting in poor resource utilization and long runtimes. In this work, we propose multiple mechanisms for adaptive, RAM-efficient parallelization of chromosome-level bioinformatics workflows. First, we develop a symbolic regression model that estimates per-chromosome memory consumption for a given task and introduces an interpolating bias to conservatively minimize over-allocation. Second, we present a dynamic scheduler that adaptively predicts RAM usage with a polynomial regression model, treating task packing as a Knapsack problem to optimally batch jobs based on predicted memory requirements. Additionally, we present a static scheduler that optimizes chromosome processing order to minimize peak memory while preserving throughput. Our proposed methods, evaluated on simulations and real-world genomic pipelines, provide new mechanisms to reduce memory overruns and balance load across threads. We thereby achieve faster end-to-end execution, showcasing the potential to optimize large-scale genomic workflows. Introduction Precision medicine seeks to personalize prevention, diagnosis, and treatment to the molecular and demographic context of each individual. Enabled by rapid declines in sequencing cost and growing clinical adoption of whole–genome sequencing (WGS), recent developments increasingly rely on genomic features that improve risk stratification and therapeutic decision-making. Two key examples are polygenic risk scores (PRS), which aggregate measures that linearly combine the small effects of many common variants to estimate disease susceptibility, and high–resolution ancestry prediction via local ancestry inference (LAI), which identifies the ancestral origin of chromosomal segments in admixed genomes. PRS are clinically valuable for early risk assessment, screening prioritization, and preventive interventions, while LAI improves the accuracy and fairness of genetic analyses by accounting for population structure at fine genomic scales and informing downstream interpretation. Both PRS computation and LAI can require complex, multi–stage bioinformatics pipelines that operate on WGS–scale inputs, stressing I/O and memory through variant calling, phasing/imputation, reference–panel matching, and model application. Modern bioinformatics pipelines have grown immensely in both scale and complexity over the past decade. Today’s workflows often string together half a dozen or more distinct tools—ranging from low-level C/C++ programs for read alignment to Java-based variant callers to Python or R scripts for downstream analyses. Each of these components may have unique resource profiles and configuration parameters. At the same time, the underlying genomic datasets have increased in size: whole-genome VCFs or GVCFs generated from deep-coverage sequencing can easily exceed tens or even hundreds of gigabytes per sample. Handling such voluminous inputs without careful orchestration frequently leads to excessive disk I/O, unpredictable memory consumption, and long wall-clock times. Figure 1: Relationship between human chromosome number and its size. A natural strategy to manage both the data volume and memory demands is to split workloads by chromosome and parallelize them across compute resources. By treating each chromosome as an independent unit of work, one can avoid loading an entire genome’s worth of data into memory at once—mitigating out-of-memory errors when processing the largest files. Furthermore, parallel execution across chromosomes maximizes CPU utilization and reduces end-to-end latency. Typically, the computational complexity and, consequently, the memory consumption and runtime, for processing a chromosome exhibit an almost linear relationship with the chromosome’s length, itself correlated with its ordinal label (i.e., chromosome number). As shown in Figure 1 , chromosomes vary drastically in length (for example, human chromosome 1 is roughly 4\times larger than chromosome 22), and so naïvely distributing one chromosome per core can lead to imbalances wherein some jobs finish quickly while others become stragglers. This scaling behavior can inform the design of resource requirement prediction and management systems. In this work, we present three complementary systems for RAM-efficient chromosome-parallel genomics. (1) A static scheduler assumes a fixed concurrency budget K (maximum threads) and optimizes the chromosome processing order to minimize peak memory while preserving throughput. (2) A dynamic scheduler treats batching as a knapsack-style packing problem: each chromosome-level job is assigned an estimated RAM cost, and the scheduler launches the jobs that maximize memory utilization, while updating estimates online based on observed peak usage with a polynomial regression. (3) A RAM prediction module based on symbolic regression predicts per-task memory from input characteristics (e.g., file size, software/configuration flags); these predictions plug directly into the dynamic scheduler to improve packing decisions from the outset. Together, these components balance load, curb overruns, and reduce wall-time across heterogeneous compute environments of precision medicine genomic workflows. Related Works Workflow frameworks and resource-aware scheduling. Many workflow orchestrators have been applied for genomic and precision medicine pipelines. General-purpose workflow engines, including GATK’s Scatter–Gather, Nextflow, Snakemake, and Cromwell/WDL, among others, can support chromosome-level partitioning and per-task memory declarations ( 1 ; 22 ; 13 ; 12 ; 24 ) . Specialized systems such as ADAM & Disq for scalable formats, Toil for dynamic rescheduling, and NVIDIA Parabricks for GPU-accelerated pipelines, demonstrate adaptive resource management in practice ( 16 ; 7 ; 23 ; 18 ) . Machine learning for resource prediction. ML has long underpinned resource prediction: ( 17 ) introduced an SVM-based “Predicting Query Runtime” (PQR) for bioinformatics runtime estimation and ( 6 ) found SVMs outperform Logistic Regression (LR) and Neural Networks (NNs) for cloud CPU utilization prediction. Recent studies employ Functional Link Neural Networks for resource-usage prediction ( 14 ) and compare broad model families including LR, SVM, LSTM, and Bi-LSTM for CPU forecasting, with Bi-LSTM and LR among the strongest ( 21 ) . Memory and CPU prediction for workflows. A parallel line of work predicts task-level runtimes and memory to avert failures and over-provisioning. HPC studies mine scheduler logs with ML to estimate job memory and enable safer submissions ( 20 ) ; workflow-centric systems learn task-specific, dynamic memory sizing during execution ( 3 ) and perform online memory and time forecasting to steer provisioning ( 2 ) . Integrating prediction into allocation and scheduling. Beyond standalone prediction, integration into resource managers is crucial: history-based managers for genome analysis use multivariate LR to guide allocations ( 5 ) ; reinforcement learning (RL) methods dynamically select per-task CPU/RAM to reduce waste ( 4 ) ; and temporal-graph-network approaches enable sophisticated dynamic load balancing ( 19 ) . At cluster scale, RL DAG schedulers such as Decima learn workload-aware policies that surpass hand-tuned heuristics, suggesting closed-loop control when coupled with telemetry and runtime feedback ( 15 ) . Classical scheduling & packing foundations. Casting batching/packing as a knapsack/bin-packing problem aligns with classical results: 0–1 Knapsack is NP-complete ( 11 ) , and First-Fit Decreasing (FFD) heuristics yield strong approximations and practical performance ( 10 ) . This foundation supports our RAM-balanced chromosome batching and justifies greedy strategies with limited look-ahead under memory constraints. Static Scheduling for Parallel Processing We present a simple scheduling system where chromosomes are processed in parallel with up to K chromosomes simultaneously processed. In order to use resources efficiently and avoid out-of-memory errors, we optimize the order in which the chromosomes are processed based on

Read the complete text