\pkg bigMICE: Multiple Imputation of Big Data
Hugo Morvan Linköping University, Jonas Agholme Linköping University, Björn Eliasson Göteborg University, Katarina Olofsson Region Västra Götaland, Ludger Grote Göteborg University, Fredrik Iredahl † Linköping University, Oleg Sysoev †* Linköping University · 2026
Read \pkg bigMICE: Multiple Imputation of Big Data by Hugo Morvan Linköping University, Jonas Agholme Linköping University, Björn Eliasson Göteborg University, Katarina Olofsson Region Västra Götaland, Ludger Grote Göteborg University, Fredrik Iredahl † Linköping University, Oleg Sysoev †* Linköping University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21613.txt.
Introduction
Hugo Morvan, Jonas Agholme, and co-authors present a software package for filling gaps in very large datasets. Missing data are common in big medical registries, where they can bias results if handled badly. Multiple imputation fills each gap several times to reflect the doubt around it.
Their package, bigMICE, scales that process to data too large for standard tools. It was built with Swedish health quality registries in mind. Registries guide real care decisions, so honest handling of missing values matters. The tool lets analysts keep the rigor of imputation without shrinking their data to fit old software.
Catalog record
| CREATOR | Hugo Morvan Linköping University, Jonas Agholme Linköping University, Björn Eliasson Göteborg University, Katarina Olofsson Region Västra Götaland, Ludger Grote Göteborg University, Fredrik Iredahl † Linköping University, Oleg Sysoev †* Linköping University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
\pkg bigMICE: Multiple Imputation of Big Data Hugo Morvan Linköping University, Jonas Agholme Linköping University, Björn Eliasson Göteborg University, Katarina Olofsson Region Västra Götaland, Ludger Grote Göteborg University, Fredrik Iredahl † Linköping University, Oleg Sysoev †* Linköping University Abstract \Plainauthor Hugo Morvan, Jonas Agholme, Björn Eliasson, Katarina Olofsson, Ludger Grote, Fredrik Iredahl, Oleg Sysoev \Plaintitle bigMICE: Multiple Imputation of Big Data \Shorttitle \pkg bigMICE: Multiple Imputation of Big Data \Abstract Missing data is a prevalent issue in many applications, including large medical registries such as the Swedish Healthcare Quality Registries, potentially leading to biased or inefficient analyses if not handled properly. Multiple Imputation by Chained Equations (MICE) is a popular and versatile method for handling multivariate missing data but traditional implementations face significant challenges when applied to big data sets due to computational time and memory limitations. To address this, the \pkg bigMICE package was developed, adapting the MICE framework to big data using Apache \pkg Spark MLLib and \pkg Spark ML. Our implementation allows for controlling the maximum memory usage during the execution, enabling processing of very large data sets on a hardware with a limited memory, such as ordinary laptops. The developed package was tested on a large Swedish medical registry to measure memory usage, runtime and dependence of the imputation quality on sample size and on missingness proportion in the data. In conclusion, our method is generally more memory efficient and faster on large data sets compared to a commonly used MICE implementation. We also demonstrate that working with very large datasets can result in high quality imputations even when a variable has a large proportion of missing data. This paper also provides guidelines and recommendations on how to install and use our open source package. \Keywords missing data, multiple imputation, MICE, big data, Apache \pkg Spark, \proglang R \Plainkeywords missing data, multiple imputation, MICE, big data, Apache Spark, R 2 2 footnotetext: : shared last authorship 1 1 footnotetext: : corresponding author, email: [email protected] 1 Introduction Missing data is a prevalent issue in many applications that can significantly impede data analysis across various scientific and engineering domains Yang et al. 2020 . In research, the absence of values for certain attributes within a data set is a common occurrence Wang et al. 2020 ; Austin et al. 2020 ; Arefin and Masum 2024 , arising from a multitude of reasons, including clerical errors during data entry, system malfunctions, participant non-disclosure, participants dropouts from studies, data corruption, or variations in data collection procedures Austin et al. 2020 ; Cesare and Were 2022 ; Arefin and Masum 2024 . The presence of missing data poses substantial challenges, as many algorithms require a complete matrix of indicators without gaps Wang et al. 2020 , and statistical results derived from incomplete data sets, particularly those with nonrandom missing values, can be biased Hasan et al. 2021 . 1.1 Problem Statement Many big data sets in society, including medical and administrative registries, are invaluable resources for research and decision-making. However, these data sets often contain missing values. Traditional approaches to handling missing data have proven inadequate for big data contexts. Typical approaches for handling missing data include case deletion (also known as complete case analysis or list-wise deletion) and simple imputation methods such as replacing missing values with the mean, median or mode Pampaka et al. 2016 ; Grund et al. 2016 ; Austin et al. 2020 . However, these single imputation methods do not account for the uncertainty associated with missing data and can attenuate variances and covariances, potentially leading to artificially narrow confidence intervals and inflated Type I error rates Austin et al. 2020 . Multiple Imputation by Chained Equations (MICE) is a popular and flexible method for imputing missing values van Buuren 2007 . It works by generating multiple plausible values for each missing data point, creating multiple complete data sets for analysis. This approach ensures that the uncertainty of the predictions is properly taken into account, leading to more valid and more reliable downstream results. Despite the widespread availability of MICE implementations in various programming languages and statistical software packages, a critical limitation emerges when applying these methods to large-scale data sets. The application of MICE to big data sets can lead to huge computational times due to the sheer size of these data sets (sometimes hundreds of gigabytes of data) and the iterative nature of MICE. However, the main problem is that computations may fail because the amount of memory required exceeds the Random Access Memory (RAM) space of a given computational resource. As modern society is constantly collecting new data, data registries are getting bigger every year, and it becomes of a critical importance to be able to create accurate imputations of these large data. 1.2 Existing Software Limitations There are many MICE implementations in various programming languages and statistical software packages. The R statistical computing environment offers one of the most comprehensive collection of MICE implementations, including the mice package van Buuren and Groothuis-Oudshoorn 2011 which commonly serves as a reference implementation for the MICE algorithm. Other notable \proglang R implementations include parlMICE for parallel computation, miceRanger using random forests, Amelia using bootstrapping and the EM algorithm Pampaka et al. 2016 ; Khan and Hoque 2020 ; Little and Rubin 2019 ; Lo et al. 2019 , and specialized packages like mitml for multilevel data Grund et al. 2016 . Beyond R, implementations exist in Python ( scikit-learn ’s IterativeImputer, miceforest ), SAS ( PROC MI ), Stata ( ice command), and various other platforms He et al. 2021 ; Kombo et al. 2017 ; Pampaka et al. 2016 ; van Buuren and Groothuis-Oudshoorn 2011 . Recognizing the computational demands of MICE, efforts have been made to improve computational time through parallelization. While parallelization of the imputation process itself is not possible due to its iterative nature, it is possible to parallelize the different imputations as they are independent, which is the approach taken in the \pkg parlMICE package. However, this approach still does not provide a solution to the fundamental memory challenges when big data are used. A MICE implementation capable of handling large data sets (in particular, large medical records) in the presence of limited computational resources has not yet been developed. 1.3 The \pkg bigMICE solution To address the gap in existing methodologies related to computing multiple imputations on large data sets, we have developed our package \pkg bigMICE. This package provides a framework for distributed missing data imputation, leveraging Apache \pkg Spark’s capabilities to fit computations to a given RAM space by dynamically exchanging information between RAM and the hard drive Apache Software Foundation 2025 . This makes it possible to handle large data sets with millions of observations on an ordinary computer having typical technical characteristics, for example, 16Gb of RAM or even less. To handle large datasets, the package \pkg bigMICE integrates the MICE framework with the Apache \pkg Spark and \pkg MLLib functionalities. Existing methodologies lack the capability to perform multiple imputation on large data sets natively within \pkg Spark’s distributed computing environment. The package incorporates several key design principles: distributed data handling using \pkg Spark dataframes, scalable regression models leveraging \pkg Spark MLLib’s optimized implementations, memory-efficient