Scalable Machine Learning Analysis of Parker Solar Probe Solar Wind Data

Daniela Martin, Connor O’Brien, Valmir P. Moraes Filho, Jinsu Hong, Jasmine R. Kobayashi, Evangelia Samara, Joseph Gallego · 2025

Read Scalable Machine Learning Analysis of Parker Solar Probe Solar Wind Data by Daniela Martin, Connor O’Brien, Valmir P. Moraes Filho, Jinsu Hong, Jasmine R. Kobayashi, Evangelia Samara, Joseph Gallego (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21066.txt.

Introduction

Daniela Martin leads six co-authors on a scalable machine learning framework for Parker Solar Probe solar wind data. Exceed one hundred fifty gigabytes for twenty eighteen to twenty twenty four and defy conventional analysis. Distributed processing with Dask handles the scale.

A quantum inspired kernel density matrix method estimates single and paired distributions for wind speed, proton density, and proton thermal speed, plus anomaly thresholds per parameter. Trends in the inner heliosphere emerge, including rising values closer to the Sun. The framework shows how modern data engineering can keep pace with flagship mission archives.

Catalog record

CREATORDaniela Martin, Connor O’Brien, Valmir P. Moraes Filho, Jinsu Hong, Jasmine R. Kobayashi, Evangelia Samara, Joseph Gallego
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21066

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable Machine Learning Analysis of Parker Solar Probe Solar Wind Data Daniela Martin, Connor O’Brien, Valmir P. Moraes Filho, Jinsu Hong, Jasmine R. Kobayashi, Evangelia Samara, Joseph Gallego Abstract We present a scalable machine learning framework for analyzing Parker Solar Probe (PSP) solar wind data using distributed processing and the quantum-inspired Kernel Density Matrices (KDM) method. The PSP dataset (2018–2024) exceeds 150 GB, challenging conventional analysis approaches. Our framework leverages Dask for large-scale statistical computations and KDM to estimate univariate and bivariate distributions of key solar wind parameters, including solar wind speed, proton density, and proton thermal speed, as well as anomaly thresholds for each parameter. We reveal characteristic trends in the inner heliosphere, including increasing solar wind speed with distance from the Sun, decreasing proton density, and the inverse relationship between speed and density. Solar wind structures play a critical role in enhancing and mediating extreme space weather phenomena and can trigger geomagnetic storms; our analyses provide quantitative insights into these processes. This approach offers a tractable, interpretable, and distributed methodology for exploring complex physical datasets and facilitates reproducible analysis of large-scale in situ measurements. Processed data products and analysis tools are made publicly available to advance future studies of solar wind dynamics and space weather forecasting. The code and configuration files used in this study are publicly available to support reproducibility. 1 Introduction The Parker Solar Probe (PSP), launched in 2018, aims to unveil the mechanisms driving the heating and acceleration of the solar wind raouafi2023parker . This work focuses on analyzing PSP measurements of solar wind plasma properties obtained by the SWEAP instrument fox_solar_2016 . The dataset spanning 2018 - 2024 exceeds 150 GB, presenting challenges for scalable and interpretable analysis of high-volume time series data. Prior studies seeking to determine the overall properties of the solar wind from data taken in-situ have been based on smaller datasets with poorer temporal resolution in more limited spatial regimes ma_statistical_2020 ; wilson_iii_statistical_2018 , and focus on basic statistical techniques such as binning and averaging. Traditional approaches, including feature extraction, probabilistic models, and deep learning methods, face limitations when applied to solar wind measurements at this scale henderson2021empirical ; jadon2021challenges . For instance, kernel density estimation (KDE) parzen1962estimation becomes computationally prohibitive, while normalizing flows rezende2015variational ; normalizationflows_gallego_2023 and autogressive models are often difficult to train. Variational autoencoders only approximate the underlying distribution, and implicit generative models such as Generative Adversarial networks generativeadversarialnetworks_goodfellow_2014 or diffusion models diffusion_croitoru_2023 do not provide an explicit density function. To address these challenges, we develop a distributed framework using Dask dask and apply the quantum-inspired Kernel Density Matrices (KDM) method gonzalez2023kernel for scalable density estimation and anomaly detection. The computation of the statistical quantities are parallelized, while not a major contribution for Dask itself, represents a practical application for the heliophysics community. Our contributions are threefold: (1) statistical characterization of solar wind properties using distributed processing; (2) application of KDM to uncover multi-parameter distributions and relationships in large PSP solar wind datasets; and (3) open-source tools for analyzing large in situ datasets. 2 Background gonzalez2023kernel define KDM over a set \mathbb{X} as a triplet \rho=(\mathbf{C},\mathbf{p},\theta) where \rho represents a density matrix defined by f_{\rho}(\mathbf{x})=\sum_{\mathbf{x}^{(i)}\in\mathbf{C}}p_{i}k_{\theta}^{2}(\mathbf{x},\mathbf{x}^{(i)}), Here, \mathbf{C} is the set of components, \mathbf{x} are the original points, and \mathbf{x}^{(i)} is the i -th component of \mathbf{C} within \mathbb{X} . Each component has an associated mixture weight p_{i}\in\mathbb{R} , representing its probability. The kernel function k_{\theta}:\mathbb{X}\times\mathbb{X}\rightarrow\mathbb{R} is defined such that k(\mathbf{x},\mathbf{x})=1 . Using a Gaussian kernel, each component \mathbf{x}^{(i)} is assumed to follow a Gaussian distribution with a normalization constant. Components define a categorical distribution, where each category corresponds to a normal distribution with mean equal to the component value and standard deviation \sigma/2 , i.e. \mathcal{N}(x^{(i)},\sigma/2) . The parameters (\mathbf{C},\mathbf{p},\theta) are learned by maximizing the log-likelihood of the observed data: \max_{\mathbf{C},\mathbf{p},\theta}\sum_{i=1}^{\ell}\log\hat{f}_{\rho}(x_{i}), where \hat{f}_{\rho} is the projection function associated with the KDM density matrix \rho of the component \mathbf{x}_{i} . The model is trained using automatic differentiation. This approach allows us to compute both univariate and joint distributions, enabling subsequent statistical analyses and anomaly detection. The model can also be used to sample new points from the learned distribution. 3 Experimental Setup Dataset. The Parker Solar Probe (PSP) is a mission launched on Aug. 12, 2018, by NASA, which has completed 24 orbits as of Jun. 19, 2025. Its primary goal is to better understand the structure and dynamics of the inner corona of the Sun, where the solar wind is heated and accelerated. Determining the processes that cause the acceleration of energetic particles is of great importance to the heliophysics community raouafi2023parker . PSP carries four instruments: FIELDS (measuring electric and magnetic fields), SWEAP (sampling charged particles such as electrons, protons, and alpha particles), WISPR (imaging the solar corona), and IS \odot IS (sampling high-energy particles) fox_solar_2016 . In this work, we focus on SWEAP data to analyze the plasma properties of the solar wind kasper_solar_2016 ; whittlesey_solar_2020 ; case_solar_2020 . The SWEAP data must be preprocessed before it can be used for this study. The SWEAP data are first downloaded as CDF files and converted to Zarr format zarr-github2025 to enable distributed computation. Fill values due to data outages are flagged and removed, and PSP’s radial distance to the Sun is computed and stored alongside the plasma data. The plasma parameters of interest are derived from the raw instrument data either by fitting several Gaussians to the particle distributions measured by the instruments or by taking moments of the particle distributions (see kasper_solar_2016 ; whittlesey_solar_2020 ; case_solar_2020 for details). Here we use data derived using the fit technique due to their improved temporal coverage. Processing Architecture. Our processing architecture comprises two parts: (1) distributed processing using Dask and (2) KDM for estimating univariate and joint distributions of PSP solar wind parameters. The PSP data are processed using Dask Array across multiple cores and nodes. The analysis was performed in two stages: first, Dask was used to compute statistical metrics from the PSP solar wind dataset; second, KDM was applied to generate univariate and bivariate distributions of parameters such as solar wind speed, proton density, and proton thermal speed. These parameters were selected because they represent the primary physical quantities governing solar wind behavior and provide complementary information on both its bulk motion and thermal state, making them ideal for studying correlations and variability in the inner heliosphere. All analyses were conducted as a function of heliocentric distance, ranging from 0 to 1 astronomical units (AU) in increments of 0.1 AU. Codebase. The full codebase supporting this work is available at https://github.com/spaceml-org/PSP-KDM . Hardware. All experiments were conducted on Google Cloud Platform using a c2-standard-8 VM instance (8 vCPUs, 32 GB RAM) with 1 TB SSD storage and a Nvidia L4 (24 GB V-GPU). Figure 1: Boxplots of three PSP solar wind parameters from 2018 to 2024: (left) solar wind speed [km/s], (middle) proton density [cm -3 ], and (right) proton thermal speed [eV]. 4 Results and Discussion Several statistics computed with Dask reveal notable features in the PSP solar wind data. Figure 1 shows boxplots for solar wind speed, proton density, and proton thermal speed for the entire dataset (2018 to 2024). Extreme solar wind speed

Read the complete text