KGMM: A K-means Clustering Approach to Gaussian Mixture Modeling for Score Function Estimation Journal: Physica D
Ludovico T Giorgini, Tobias Bischoff, Andre N Souza · 2025
Read KGMM: A K-means Clustering Approach to Gaussian Mixture Modeling for Score Function Estimation Journal: Physica D by Ludovico T Giorgini, Tobias Bischoff, Andre N Souza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18054.txt.
Introduction
Giorgini, Bischoff, and Souza estimate the score function, the slope of a system's steady-state density, with a hybrid trick. Their method, KGMM, first clusters data with a bisecting form of K-means, builds a Gaussian mixture on the clusters, and then lets a neural network smooth the result. The score is the key ingredient for building reduced stochastic models that mimic a system's long-run behavior.
Across test systems, from simple potentials to chaotic Lorenz models and a fluid equation, KGMM rebuilds the steady state faithfully. It stays accurate where plain mixture models fail through noise blow-up, and for small systems it beats the standard denoising score matching approach at a fraction of the cost.
Catalog record
| CREATOR | Ludovico T Giorgini, Tobias Bischoff, Andre N Souza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18054
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KGMM: A K-means Clustering Approach to Gaussian Mixture Modeling for Score Function Estimation Journal: Physica D Ludovico T Giorgini, Tobias Bischoff, Andre N Souza Abstract We propose a hybrid method for accurately estimating the score function—the gradient of the logarithm of a system’s steady-state probability density function—using Gaussian Mixture Model (GMM) in conjunction with a bisecting K-means clustering step. Our approach, which we call KGMM, offers a systematic way to combine statistical density estimation with a neural-network-based interpolation of the score, leveraging the strengths of both. We demonstrate its ability to accurately reconstruct the long-time statistical properties of several paradigmatic systems, including lower-dimensional potential systems and chaotic Lorenz-type models. Numerical experiments show that KGMM yields robust estimates of the score function, even for small values of the covariance amplitude in the GMM, where the standard GMM methods tend to fail because of noise amplification. These accurate estimates allow us to build effective stochastic reduced-order models that reproduce the invariant measures of the target dynamics. PACS 05.45.-a , 02.50.Ey , 05.10.Gg 1 Introduction The score function, defined as the gradient of the logarithm of a system’s steady-state probability density function, is a fundamental quantity in statistical physics, dynamical systems, and machine learning. It underpins key theoretical frameworks such as the Generalized Fluctuation-Dissipation Theorem (GFDT) [ 1 , 2 , 3 , 4 , 5 ] , which links spontaneous fluctuations to system responses, and plays a crucial role in generative modeling [ 6 ] , parameter estimation [ 7 ] , and causal inference [ 8 ] . Crucially, knowledge of the score function provides insights into the dynamical features of a system without requiring explicit knowledge of its governing equations. Instead, it can be inferred from statistical properties, which are often more accessible in experimental and numerical settings [ 9 , 10 , 11 , 12 , 13 ] . Accurate and efficient estimation of the score function remains a formidable challenge, particularly in high-dimensional systems. Gaussian Mixture Models (GMMs) are widely used to approximate complex probability distributions due to their flexibility and well-established probabilistic framework [ 14 ] . In a GMM, the probability density function is modeled as a weighted sum of Gaussian components, where the mean vectors of the Gaussians are chosen to span the state space explored by the underlying dynamical system. A critical aspect of using GMMs is the selection of the covariance matrix amplitude for each Gaussian component. Larger covariance amplitudes result in a smoother estimated invariant density because the Gaussian kernel effectively averages out local fluctuations. However, this smoothing comes at a cost: the estimated density is perturbed relative to the true invariant density, as the convolution with the Gaussian kernel tends to blur finer details of the distribution. Conversely, smaller covariance amplitudes produce an invariant density estimate that more closely resembles the true distribution. Yet, the reduction in smoothing increases the noise level in the estimate, which is particularly problematic when differentiating the density to compute the score function. Here, even slight noise amplification can lead to significant inaccuracies in the gradient estimates. Although increasing the number of Gaussian mixture components can help mitigate these issues by providing a more detailed approximation, this solution introduces additional computational burdens and an elevated risk of overfitting [ 15 ] . Recent advancements in score-based generative modeling [ 16 , 17 , 6 , 18 , 19 , 20 ] offer an alternative strategy by directly training a neural network to approximate the score function via a dataset-wide loss minimization procedure. This method relies on the implicit regularization afforded by the neural network training procedure to define a “smoothed" version of the Gaussian mixture score function. However, this approach is computationally expensive, as the loss function depends on the entire dataset, and there is no guarantee that the learned score function converges to the true underlying gradient field. In this work, we propose a hybrid approach that leverages both GMM-based statistical estimation and neural network interpolation. Our method first computes the score function at representative points in the state space by combining a bisecting K-means clustering algorithm with GMM. As we will show, this strategy enables the efficient evaluation of a discretized version of the score function by leveraging information from the whole dataset. We then train a neural network to interpolate between these points, ensuring a scalable and efficient reconstruction of the score function while maintaining the statistical robustness of GMM-based estimation. This method combines the advantages of probabilistic density modeling with the flexibility of machine learning, leading to a computationally efficient and precise framework for score function estimation in large datasets. The article is structured as follows. Section 2 presents the KGMM method, detailing its derivation and advantages over standard GMMs. Section 3 validates KGMM through numerical experiments on potential and chaotic systems, comparing estimated score functions with analytical solutions when available. Section 4 concludes with key findings and future directions. 2 Method 2.1 Motivation The dynamics of physical systems often exhibit a hierarchical structure in their spatiotemporal evolution, wherein predictable, low-dimensional processes emerge on longer timescales and larger spatial scales, while chaotic, high-dimensional fluctuations dominate at shorter timescales and finer spatial resolutions. In many complex systems, the details of small-scale, fast processes become increasingly irrelevant under coarse-graining transformations and can be effectively replaced by stochastic forcing terms that preserve essential statistical and dynamical properties. This paradigm not only provides a faithful representation of the underlying physics but also enables a significant reduction in the dimensionality of high-dimensional systems, facilitating both analytical tractability and numerical efficiency. A paradigmatic example of this approach is found in climate physics, where large-scale, slow dynamics, such as ocean circulation and seasonal variations, coexist with small-scale, rapid processes, including turbulent eddies and convective storms. Reduced-order stochastic models provide an effective means of capturing the statistical and dynamical structure of such multiscale interactions, successfully replicating phenomena like the El Niño-Southern Oscillation (ENSO), monsoonal cycles, and long-range teleconnections, as well as the coupling of climate variables observed in paleoclimate data [ 21 , 22 , 23 , 24 , 25 ] . Based on observations of a physical system characterized by a steady-state distribution \rho_{S}(\textbf{x}) and time correlation function \bm{C}(t) , the following Langevin equation is constructed to inherently reproduce these properties: \dot{\bm{x}}(t)=\bm{\Sigma}\bm{\Sigma}^{T}\nabla\ln\rho_{S}(\bm{x})+\sqrt{2}\bm{\Sigma}\bm{\xi}(t), (1) where \bm{\xi}(t) is a vector of independents Gaussian white noise process, and the covariance matrix \bm{\Sigma} is chosen to match the time-correlations of the observed data. This formulation ensures that \rho_{S} remains invariant under the corresponding Fokker-Planck operator, \mathcal{L}_{FP}\rho_{S}=0,\quad\text{with}\quad\mathcal{L}_{FP}f=-\nabla\cdot\left(\bm{\Sigma}\bm{\Sigma}^{T}\nabla\ln\rho_{S}f\right)+\nabla\cdot\left(\bm{\Sigma}\bm{\Sigma}^{T}\nabla f\right), (2) which governs the evolution of probability density in the reduced-order model. The key observation here is that the deterministic drift term in the Langevin equation ( 1 ) is determined by the score function, \nabla\ln\rho_{S}(\bm{x}) , which encapsulates the structure of the underlying dynamical system. Knowledge of this drift term provides insight into the statistical and dynamical properties of the observed system, including the ability to quantify how the system responds to external perturbations [ 26 ] . In the next section, we will show how, by leveraging statistical estimation techniques alongside machine learning approaches, it becomes possible to reconstruct this fundamental quantity with high fidelity, offering new avenues for the systematic derivation of stochastic