SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields

David Keetae Park, Xihaier Luo, Guang Zhao, Seungjun Lee, Miruna Oprescu, Shinjae Yoo · 2025

Read SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields by David Keetae Park, Xihaier Luo, Guang Zhao, Seungjun Lee, Miruna Oprescu, Shinjae Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12262.txt.

Introduction

David Keetae Park and five co-authors introduce SCENT, a framework for learning from data that vary over space and time. Such learning is hard: links across space and time intertwine, data are high in dimensions, and science data often arrive unevenly, with gaps from sensor failures. SCENT unifies filling gaps, rebuilding signals, and forecasting in one design. A transformer-based chain of encoder, processor, and decoder forms its backbone.

Learnable queries improve how it generalizes, and a query-wise attention step captures links at multiple scales. A sparse attention step keeps the system able to grow in data size and model size, with flexible output at any resolution. Simulations and real-world tests show leading results across tasks, with strong scaling.

Catalog record

CREATORDavid Keetae Park, Xihaier Luo, Guang Zhao, Seungjun Lee, Miruna Oprescu, Shinjae Yoo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12262

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields David Keetae Park, Xihaier Luo, Guang Zhao, Seungjun Lee, Miruna Oprescu, Shinjae Yoo Abstract Spatiotemporal learning is challenging due to the intricate interplay between spatial and temporal dependencies, the high dimensionality of the data, and scalability constraints. These challenges are further amplified in scientific domains, where data is often irregularly distributed (e.g., missing values from sensor failures) and high-volume (e.g., high-fidelity simulations), posing additional computational and modeling difficulties. In this paper, we present SCENT, a novel framework for scalable and continuity-informed spatiotemporal representation learning. SCENT unifies interpolation, reconstruction, and forecasting within a single architecture. Built on a transformer-based encoder-processor-decoder backbone, SCENT introduces learnable queries to enhance generalization and a query-wise cross-attention mechanism to effectively capture multi-scale dependencies. To ensure scalability in both data size and model complexity, we incorporate a sparse attention mechanism, enabling flexible output representations and efficient evaluation at arbitrary resolutions. We validate SCENT through extensive simulations and real-world experiments, demonstrating state-of-the-art performance across multiple challenging tasks while achieving superior scalability. † † affiliationnotice: 1 Introduction Spatiotemporal learning focuses on modeling and interpreting data that exhibit variations in both space and time. This approach is crucial for analyzing intricate real-world phenomena where spatial structures are inextricably linked with temporal dynamics, including applications such as climate modeling ( Reichstein et al., 2019 ) , traffic forecasting ( Li et al., 2018 ) , medical imaging ( Litjens et al., 2017 ) , and video analysis ( Wang et al., 2020 ) . Achieving accurate spatiotemporal learning, however, presents significant challenges due to the presence of complex spatial-temporal dependencies, including spatial heterogeneity and temporal non-stationarity, compounded by the high dimensionality of the data. These challenges are further amplified in scientific and engineering domains, where datasets are frequently characterized by irregular distributions (e.g., arising from sensor malfunctions) and large volumes (e.g., generated by high-fidelity simulations), introducing further complexities in both computation and modeling. Figure 1: Challenging data scenarios motivating this study. (a) Learning continuous ground truth signal (GT) given noisy, malfunctioning, sparse, or moving sensors is a daunting challenge. However, these challenges are common in scientific data, including AirDelhi ( Chauhan et al., 2024 ) data which measure the particulate matter levels (PM2.5) from moving vehicles. To address the aforementioned challenges, significant research efforts have been dedicated to developing solutions from both traditional signal processing and, more recently, machine learning perspectives (a detailed literature review is provided in the Appendix A ). Among the emerging machine learning approaches, Implicit Neural Representations (INRs) have garnered increasing attention due to their inherent flexibility ( Sitzmann et al., 2020b ; Mildenhall et al., 2020 ) . INRs parameterize data as a continuous function, mapping coordinates to signal values (e.g., (x,y)\rightarrow(r,g,b) for images), using a neural network. Despite their potential, the adoption of INRs in scientific domains has been limited. This can primarily be attributed to two key challenges: scalability and generalizability. To bridge this methodological gap, we propose SCENT ( S calable C ondition e d N eural Field for Spatio T emporal Learning), a novel framework designed to address the limitations of existing INR approaches. SCENT leverages a Transformer-based encoder-processor-decoder architecture to efficiently process large-volume spatiotemporal data, demonstrating strong scalability with respect to both dataset size and model parameter count. Furthermore, SCENT incorporates trainable query mechanisms to enhance generalizability, circumventing the computational overhead associated with existing strategies such as latent optimization ( Dupont et al., 2022 ) or meta-learning ( Chen & Wang, 2022 ) . Overall, the major contributions of our work include: • We offer a unified framework for spatiotemporal learning, capable of performing joint interpolation, reconstruction, and forecasting. • We empirically demonstrate that SCENT is scalable with larger models and datasets. • We conduct extensive experiments to evaluate the efficacy of the proposed SCENT framework, demonstrating its superiority over state-of-the-art (SOTA) methods in a range of spatiotemporal learning tasks. Figure 2: SCENT overview. (a) Detailed architecture of SCENT is illustrated. Our unique contributions are drawn with red boxes. Specifically, we introduce time coordinates to both encoder and decoder for learning continuous time representations. Also, we introduce Context Embedding Network and Calibration Network for improved spatial encoding and decoding, respectively. n_{\text{enc}} , n_{\text{proc}} , and n_{\text{dec}} denote the number of layers in encoder, processor, and decoder respectively. (b) At the inference stage, a single SCENT model is capable to jointly perform reconstruction, spatiotemporal interpolation, and forecasting. 2 Problem Setting We consider a spatiotemporal discrete observation denoted as u^{t}_{\mathbf{x}} , where \mathbf{x}=(x,y,z)\in\mathbb{R}^{3} represents the spatial coordinates, and t\in\mathbb{R} denotes the sampled time. Given a set of N_{\text{i}} input observations measured at time t_{\text{i}} , we define: U^{t_{\text{i}}}=\left\{u^{t_{\text{i}}}_{\mathbf{x}_{1}},u^{t_{\text{i}}}_{\mathbf{x}_{2}},\dots,u^{t_{\text{i}}}_{\mathbf{x}_{N_{\text{i}}}}\right\}. Our objective is to develop a model F capable of learning the underlying continuous spatiotemporal function from these discrete N_{\text{i}} observations, enabling it to predict M target outputs at a different time t_{\text{o}} : \hat{U}^{t_{\text{o}}}=F\left(U^{t_{\text{i}}}\right), where the target outputs are given by: U^{t_{\text{o}}}=\left\{u^{t_{\text{o}}}_{\mathbf{x}_{1}^{\prime}},u^{t_{\text{o}}}_{\mathbf{x}_{2}^{\prime}},\dots,u^{t_{\text{o}}}_{\mathbf{x}_{N_{\text{o}}}^{\prime}}\right\}. The specific task performed by ( F ) is determined by the relationship between the input and output spatiotemporal coordinates: • Reconstruction. If \left(\{\mathbf{x}_{j}\}_{j=1}^{N_{\text{i}}},t_{\text{i}}\right)=\left(\{\mathbf{x}^{\prime}_{k}\}_{k=1}^{N_{\text{o}}},t_{\text{o}}\right) , the model is tasked with reconstructing the same observations it was provided. • Spatiotemporal Interpolation. If \left(\{\mathbf{x}_{j}\}_{j=1}^{N_{\text{i}}},t_{\text{i}}\right)\neq\left(\{\mathbf{x}^{\prime}_{k}\}_{k=1}^{N_{\text{o}}},t_{\text{o}}\right) , the model estimates the function at novel spatiotemporal locations. • Forecasting. A special case of interpolation where t_{\text{o}}>t_{\text{i}} , requiring the model to predict future values. Our goal is to develop a model ( F ) capable of jointly performing reconstruction, interpolation, and forecasting of complex scientific data given arbitrary spatiotemporal coordinates (x,y,z,t) . Figure 3: Warp-unrolling forecasting (WUF). (a) Conventional forecasting includes single-step unrolling which accumulates error. A dimming blue color is used to represent the increasing error. (b) WUF helps mitigate error accumulations caused by extensive unrolling steps. 3 SCENT 3.1 Encoder-Processor-Decoder Framework The encoder-processor-decoder architecture is proposed as a general framework that scales linearly with input and output sizes, overcoming the quadratic scaling limitations of transformers with input sequence length ( Jaegle et al., 2022 ) . Prior works have leveraged this architecture to learn conditioned neural fields (CNFs) for solving partial differential equations ( Lee & Oh, 2024 ; Serrano et al., 2024 ; Serrano et al., 2023 ; Li et al., 2023 ) , where the architecture is particularly beneficial, as these models flatten the input and treat each sample point as an individual token. Additionally, this architecture facilitates learning continuity-informed representations using inducing point learning within cross-attention layers ( Jaegle et al., 2022 ; Lee & Oh, 2024 ) . Building on these advancements, we evaluate the models’ potentials in complex real-world scientific problems, often characterized by intricate noise patterns and irregular sensor distributions. In the remaining subsections, we describe notable developments on top of the existing architecture, as displayed by red-colored objects in Fig. 2 (a). 3.2 Time-Targeted Spatial Encoder The encoder processes an input data U^{t_{\text{i}}}=\left\{u^{t_{\text{i}}}_{\mathbf{x_{1}}},u^{t_{\text{i}}}_{\mathbf{x_{2}}},\dots,u^{t_{\text{i}}}_{\mathbf{x}_{N_{\text{i}}}}\right\} , representing N_{\text{i}} samples from the space coordinate \{\mathbf{x}_{j}\}_{j=1}^{N_{\text{i}}} at a given time t_{\text{i}} . Here, \{\mathbf{x}_{j}\}_{j=1}^{N_{\text{i}}} can be structured on a grid or an irregular mesh. Using a cross-attention mechanism, the encoder transforms U^{t_{\text{i}}} into a fixed-size set of tokens Z^{t_{\text{i}}}_{M}=\{z_{1},z_{2},\ldots,z_{M}\} , where M is the number of latent trainable query tokens. The encoder E_{\theta} is then expressed as: \displaystyle E_{\theta}:\left(U^{t_{\text{i}}},\{\mathbf{x}_{j}\}_{j=1}^{N_{\text{i}}},t_{\text{i}},t_{\text{o}}\right)\rightarrow

Read the complete text