The Nearest-Neighbor Derivative Process: Modeling Spatial Rates of Change in Massive Datasets
Jiawen Chen, Aritra Halder, Yun Li, Sudipto Banerjee, Didong Li · 2025
Read The Nearest-Neighbor Derivative Process: Modeling Spatial Rates of Change in Massive Datasets by Jiawen Chen, Aritra Halder, Yun Li, Sudipto Banerjee, Didong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02752.txt.
Introduction
Jiawen Chen, Aritra Halder, Yun Li, Sudipto Banerjee, and Didong Li present a 2025 spatial statistics paper on estimating not just a spatial field but its rates of change across massive datasets. Derivatives expose subtle pattern shifts in environmental and biological data, yet standard Gaussian processes scale cubically and choke on large samples.
Cheap plug-in fixes, fitting a scalable model then differencing, depend on tuning choices and can produce invalid derivative inference. The Nearest-Neighbor Derivative Process instead models field and derivatives jointly inside one scalable Gaussian process. Its cost grows linearly with sample size, making huge datasets feasible. The authors supply supporting theory and test the method in simulations and real data analyses, reporting accurate derivative estimates where older routes struggle.
Catalog record
| CREATOR | Jiawen Chen, Aritra Halder, Yun Li, Sudipto Banerjee, Didong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 142 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02752
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Nearest-Neighbor Derivative Process: Modeling Spatial Rates of Change in Massive Datasets Jiawen Chen, Aritra Halder, Yun Li, Sudipto Banerjee, Didong Li Abstract Gaussian processes (GPs) are instrumental in modeling spatial processes, offering precise interpolation and prediction capabilities across fields such as environmental science and biology. Recently, there has been growing interest in extending GPs to infer spatial derivatives, which are vital for analyzing spatial dynamics and detecting subtle changes in data patterns. Despite their utility, traditional GPs suffer from computational inefficiencies, due to the cubic scaling with the number of spatial locations. Fortunately, the computational challenge has spurred extensive research on scalable GP methods. However, these scalable approaches do not directly accommodate the inference of derivative processes. A straightforward approach is to use scalable GP models followed by finite-difference methods, known as the plug-in estimator. This approach, while intuitive, suffers from sensitivity to parameter choices, and the approximate gradient may not be a valid GP, leading to compromised inference. To bridge this gap, we introduce the Nearest-Neighbor Derivative Process (NNDP), an innovative framework that models the spatial processes and their derivatives within a single scalable GP model. NNDP significantly reduces the computational time complexity from O ( n 3 ) O(n^{3}) to O ( n ) O(n) , making it feasible for large datasets. We provide various theoretical supports for NNDP and demonstrate its effectiveness through extensive simulations and real data analysis. Keywords: Bayesian modeling; Mean-square derivatives; Nearest-Neighbor Gaussian processes; Spatial gradients. 1 Introduction Spatial processes play a critical role in environmental science, ecology, geology, and biology ( Banerjee et al. 2014 ) and are prominent in the modeling and analysis of scientific data referenced with respect to geographic space. Gaussian processes (GPs) are widely used in spatial modeling due to their flexibility in capturing complex spatial dependencies through covariance functions. They provide a probabilistic framework for spatial interpolation, commonly known as “kriging”, prediction, and uncertainty quantification, making them indispensable tools in spatial statistics ( Williams and Rasmussen 2006 ) . Estimation of the underlying spatial process is often followed by analysis of local features of the spatial surface. This exercise is often referred to as “wombling”, named after a seminal paper by Womble 1951 . Spatial gradients (first-order derivatives) and curvature (second-order derivatives) provide information on the rate and direction of changes in the spatial domain ( Banerjee et al. 2003 ; Halder et al. 2024 ) . For example, gradients are crucial to identify boundaries of ecological differences ( Banerjee and Gelfand 2006 , see) or changes in hedonic house prices across administrative boundaries ( Majumdar et al. 2006 ) . Estimating derivatives from spatial data enables the analysis of local features such as peaks, troughs, and ridges, providing deeper insight into spatial phenomena, for example discovering the lurking effects of a natural boundary (see, e.g., Halder et al. 2024 , models heavy-metal concentrations in a river-valley) . Recent investigations into modeling how biological characteristics change in tissues within the human brain ( Chen et al. 2024 , see, e.g.,) provide insight into the neural microenvironment. Although several of the aforementioned articles have addressed probabilistic inference of spatial gradients, such inference has not scaled to analyze massive volumes of spatial data arising from rapidly emerging spatial data science technologies such as remote sensing. Some of the most effective spatial processes for scaling massive datasets are derived from parent GPs using directed acyclic graphs to induce sparsity. Although such constructions produce computationally efficient GPs such as the Nearest-Neighbor Gaussian Process (NNGP) introduced by Datta et al. 2016 and generalizations thereof, they lose the smoothness properties of the parent GP and generally do not admit spatial gradients. Inferring derivatives requires specifying the smoothness of the spatial process to ensure that spatial gradients are well-defined. Although such smoothness can be modeled using a flexible spatial covariance kernel such as the Matérn Stein 2012 to ensure that gradient and curvature processes can be derived, such modeling is hindered if the GP needs to be scaled to analyze massive volumes of spatial data. Our current contribution here is to develop derivative processes from an NNGP to enable spatial modelers to infer spatial gradients in surfaces estimated from massive volumes of spatial data. Typically, joint inference is performed on the parent process and its induced derivatives ( Banerjee et al. 2003 ) . Further advances by Halder et al. 2024 extend inference to spatial curvature processes. The GP framework offers attractive exact posterior predictive inference for such processes. It only requires a valid cross-covariance matrix, which depends on the smoothness specification of the parent process. However, incorporating derivative processes into GPs exacerbates the computational burden. The combined process comprised of the parent and induced derivative processes introduces a complex dependence structure that does not facilitate efficient computation. This limitation becomes severe in applications involving massive spatial datasets, such as those arising from high-resolution environmental monitoring, remote sensing, and spatial genomics. The need for scalable methods that can efficiently estimate spatial derivatives without sacrificing accuracy is therefore pressing. To address these challenges, Chen et al. 2024 proposed StarTrail, a two-step scalable gradient process, where the spatial process is modeled as an NNGP followed by the plug-in estimate of spatial gradients through finite-difference approximations ( Liu and Li 2022 ) . While this method offers computational scalability, it inherits limitations inherent in finite difference techniques. Specifically, finite difference approximations can be sensitive to the choice of step size and may introduce significant approximation errors, particularly when estimating higher-order derivatives or dealing with irregularly spaced data. Moreover, the finite difference approach does not provide valid probabilistic inference for the gradients; it treats the gradient estimates as deterministic quantities derived from the approximated function rather than modeling the joint distribution of the process and its derivatives. This article introduces the Nearest Neighbor Derivative Process (NNDP) to jointly model the spatial process and its derivatives within a unified probabilistic framework. By leveraging the properties of GPs, where the derivatives of a GP are themselves GPs, we construct the derivatives of NNGP kernels and prove that NNDP is a rigorous GP. Furthermore, we establish the smoothness of the approximated process and derive the joint distribution for NNGP and NNDP while ensuring mathematical rigor and robustness. NNDP provides several advantages over existing methods. It eliminates the need for step size selection inherent in finite difference methods, provides accurate derivative estimates, and enables parameter inference and uncertainty quantification. Its scalability is maintained by exploiting the conditional independence structure of the NNGP, making it computationally efficient for large datasets. Our method is supported by extensive simulation studies and two real data applications in high-resolution ST data and climate science. The remainder of the paper is organized as follows. Section 2 reviews preliminaries. Section 3 introduces NNDP. Section 4 presents simulations while Section 5 applies the NNDP to spatial transcriptomics and air temperature data. Section 6 concludes with a discussion. Proofs and additional experiments are supplied in an Appendix. 2 Preliminary 2.1 Gaussian Process (GP) A GP is a stochastic process such that any finite collection of its realizations follows a multivariate normal distribution ( Stein 2012 ) . Formally, if \omega is a GP defined over the domain \mathcal{D} with