NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data DOI: XXXXXXX.XXXXXXX
Fabian Woller, Lis Arend, Christian Fuchsberger, Markus List, David B. Blumenthal · 2025
Read NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data DOI: XXXXXXX.XXXXXXX by Fabian Woller, Lis Arend, Christian Fuchsberger, Markus List, David B. Blumenthal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00448.txt.
Introduction
Fabian Woller and co-authors present a Python package for fast statistics on messy, large data sets in a 2025 paper. Real data often mix types, such as numbers and labels, and often have gaps where values are missing. Standard Python tools slow down badly on such data, which blocks live exploration tools used in biomedical work.
Their package, NApy, pairs a compiled back end with parallel threads to run tests while handling gaps head-on. In checks, it beat rival tools and plain Python versions by orders of magnitude in both run time and memory use. That speed lets analysts test ideas on the fly in interactive apps. The code is shared in public for others to use and extend.
Catalog record
| CREATOR | Fabian Woller, Lis Arend, Christian Fuchsberger, Markus List, David B. Blumenthal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data DOI: XXXXXXX.XXXXXXX Fabian Woller, Lis Arend, Christian Fuchsberger, Markus List, David B. Blumenthal Abstract . Existing Python libraries and tools lack the ability to efficiently compute statistical test results for large datasets in the presence of missing values. This presents an issue as soon as constraints on runtime and memory availability become essential considerations for a particular usecase. Relevant research areas where such limitations arise include interactive tools and databases for exploratory analysis of biomedical data. To address this problem, we present the Python package NApy, which relies on a Numba and C++ backend with OpenMP parallelization to enable scalable statistical testing for mixed-type datasets in the presence of missing values. Both with respect to runtime and memory consumption, NApy outperforms competitor tools and baseline implementations with naïve Python-based parallelization by orders of magnitude, thereby enabling on-the-fly analyses in interactive applications. NApy is publicly available at https://github.com/DyHealthNet/NApy . Figure 1. Overview of NApy’s workflow (A) and benchmark analyses of runtime and memory consumption (B). NApy enables efficient computation of standard statistical tests in parallel with pairwise deletion of missing values. Benchmarking on both simulated and a population-based cohort study demonstrates that NApy outperforms state-of-the-art competitors by orders of magnitude. 1. Introduction Statistical analyses in biomedical research areas often deal with datasets consisting of several thousands of samples and variables. Especially population-based cohort studies such as the Cooperative Health Research in South Tyrol (CHRIS) study ( Pattaro et al. 2015 ) or the Study of Health in Pomeria (SHIP) ( Völzke et al. 2011 ) typically include several thousand study participants. These studies typically encompass diverse health and lifestyle records represented by quantitative (e.g. blood pressure, height), dichotomous (i.e binary, such as sex, presence of a disease), and categorical variables (e.g. body mass index categories, job). Additionally, due to factors like experimental design or the non-applicability of certain measurements to specific subgroups of participants, the population cohort data frequently exhibits a significant amount of missing data ( Little et al. 2012 ) . While, classically, biomedical data analyses on population cohort data mostly investigate statistical associations between specific variables (e.g., correlations between measurements as body mass index and blood pressure in population cohorts), the rapidly growing popularity of systems approaches in biomedicine makes it increasingly relevant to be able to efficiently compute pairwise statistical associations for all available pairs of variables in a dataset. For instance, one could imagine an (online) data explorer where users can interactively mine population cohort data for networks of variables whose pairwise association strengths differ between two user-selected conditions or contexts. Similar network-based data explorers already exist for pre-computed expert-curated networks, e.g., CoVex ( Sadegh et al. 2020 ) for exploration of the SARS-CoV-2 virus-host-drug interactome or NeDRex ( Sadegh et al. 2021 ) and Drugst.One ( Maier et al. 2024 ) for general-purpose exploration of knowledge graphs defined over biomedical entities such as drugs, diseases, and genes. To leverage the data analysis paradigm exemplified in these approaches for the analysis of context-dependent association networks that must be computed on the fly from population cohort data, a statistics tool is required that combines the following two properties: • The tool should be runtime- and memory-efficient enough to enable real-time computation of association networks for large-scale datasets (i.e., computation of all pairwise statistical associations between variables). • The tool should be capable of handling heterogeneous data with high rates of missing values while minimizing information loss (i.e., pairwise data removal instead of complete-case analysis). As an easily accessible and yet powerful programming language with plenty of available modular extensions, Python has become one of the most popular languages in statistical programming and data mining. Several state-of-the-art Python libraries for statistical analysis exist. While the core Python libraries NumPy ( Harris et al. 2020 ) and pandas ( Wes McKinney 2010 ) primarily support basic correlation computations, SciPy ( Virtanen et al. 2020 ) and Pingouin ( Vallat 2018 ) provide a broader spectrum of statistical functions and distributions. However, all of these libraries exhibit specific limitations with respect to the previously described use case. Firstly, pairwise missing data removal or pairwise feature calculations are restricted to a limited subset of statistical tests. Secondly, their application to large-scale datasets within an interactive data explorer is impractical due to the prolonged runtime and the lack of inherent parallelization capabilities. This limitation arises primarily because more efficient popular Python statistics libraries, such as SciPy and pandas, are mainly implemented in Cython, which inherently lacks support for intrinsic parallelization. For this reason, we developed our Python tool NApy, which addresses all of the above-mentioned issues. We demonstrate the performance of NApy on both simulated data (with varying numbers of features, samples, threads and missing value ratios) and a real-world population cohort. Our results show that NApy uses memory much more efficiently than most available Python competitors and outperforms state-of-the-art competitors in terms of runtime and memory by orders of magnitude. 2. Related Work Table 1. Overview of state-of-the-art Python libraries for statistical tests computation including NApy. NumPy pandas SciPy Pingouin NApy Input data NumPy matrix pandas DataFrame NumPy matrix pandas DataFrame NumPy matrices Statistical tests only r^{2} only r^{2} & \rho > 20 > 20 7 Pairwise input feature calculation ✓ ✓ only \rho only r^{2} & \rho ✓ Pairwise missing value removal \times ✓ only \rho only r^{2} & \rho ✓ P-value calculation \times \times ✓ ✓ ✓ Effect size calculation \times \times limited extensive ✓ Multiple testing correction \times \times 2 5 3 Output format NumPy matrix ( r^{2} ) pandas DataFrame ( r^{2} & \rho ) SciPy object (test statistics & P -values) pandas DataFrame (test statistics, un- & adjusted P -values, effect size, additional results) dictionary of NumPy arrays (test statistics, un- & adjusted P -values, effect sizes) Parallelization \times \times \times \times ✓ A range of computational tools in Python, including NumPy ( Harris et al. 2020 ) , SciPy ( Virtanen et al. 2020 ) , Pingouin ( Vallat 2018 ) , and pandas ( Wes McKinney 2010 ; pandas development team 2024 ) , are available for conducting statistical tests, yet each comes with specific limitations concerning our use case ( Table 1 ). NumPy and pandas, two popular Python libraries for data handling and modeling, are largely limited to correlation analyses and thus only support analyses of continuous data. NumPy is restricted to the calculation of Pearson correlation coefficients and is lacking pairwise deletion of missing data. Conversely, pandas expands correlation options, providing Pearson, Spearman, and Kendall coefficients across features in the data with pairwise deletion of missing values, but does not provide P -values. SciPy and Pingouin, on the other hand, offer a broader selection of statistical tests. Expanding upon NumPy’s core features, SciPy offers additional array computation tools, incorporating efficient, low-level language implementations in Fortran, C, and C++ to enhance processing speed. It provides a range of statistical tests for categorical and continuous data, including two-sided P -value reporting. Notably, SciPy enables pairwise feature calculations and missing value deletion specifically for Spearman’s rank correlation, while SciPy’s other statistical tests lack these capabilities. SciPy further includes multiple