Effective Dataset Distillation for Spatio-Temporal Forecasting with Bi-dimensional Compression
Taehyung Kwon 1, Yeonje Choi 1, Yeongho Kim, Kijung Shin · 2026
Read Effective Dataset Distillation for Spatio-Temporal Forecasting with Bi-dimensional Compression by Taehyung Kwon 1, Yeonje Choi 1, Yeongho Kim, Kijung Shin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10410.txt.
Introduction
Taehyung Kwon and co-authors present STemDist, the first dataset distillation method built for spatio-temporal forecasting, as in traffic and weather data. Such data are huge in both space and time, so training deep models gets costly. Distillation replaces the full data with a compact synthetic set, yet past methods squeezed only one dimension. STemDist compresses both in balance, works at cluster level to cut cost, then adds subset refinement. On five real datasets, training on its output ran faster, used far less memory, and gave lower error.
Catalog record
| CREATOR | Taehyung Kwon 1, Yeonje Choi 1, Yeongho Kim, Kijung Shin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83824 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10410
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective Dataset Distillation for Spatio-Temporal Forecasting with Bi-dimensional Compression Taehyung Kwon 1, Yeonje Choi 1, Yeongho Kim, Kijung Shin Abstract Spatio-temporal time series are widely used in real-world applications, including traffic prediction and weather forecasting. They are sequences of observations over extensive periods and multiple locations, naturally represented as multi-dimensional data. Forecasting is a central task in spatio-temporal analysis, and numerous deep learning methods have been developed to address it. However, as dataset sizes and model complexities continue to grow in practice, training deep learning models has become increasingly time- and resource-intensive. A promising solution to this challenge is dataset distillation, which synthesizes compact datasets that can effectively replace the original data for model training. Although successful in various domains, including time series analysis, existing dataset-distillation methods compress only one dimension, making them less suitable for spatio-temporal datasets, where both spatial and temporal dimensions jointly contribute to the large data volume. To address this limitation, we propose STemDist, the first dataset distillation method specialized for spatio-temporal time series forecasting. A key idea of our solution is to compress both temporal and spatial dimensions in a balanced manner, reducing training time and memory. We further reduce the distillation cost by performing distillation at the cluster level rather than the individual location level, and we complement this coarse-grained approach with a subset-based granular distillation technique that enhances forecasting performance. On five real-world datasets, we show empirically that, compared to both general and time-series dataset distillation methods, datasets distilled by our STemDist method enable model training (1) faster (up to 6 × \times ) (2) more memory-efficient (up to 8 × \times ), and (3) more effective (with up to 12% lower prediction error). \textsuperscript{1} \textsuperscript{1} footnotetext: Equal contribution. I Introduction Fig. 1: Synthetic data compressed in both temporal and spatial dimensions in a balanced manner yields significantly lower training time and GPU memory usage compared to datasets of the same size compressed only along the temporal dimension. See Preliminary Experiment 1 for details. Spatio-temporal time series datasets are universal, playing a key role in real-world applications, including traffic prediction [ 22 , 14 , 11 , 30 , 32 ] and weather forecasting [ 1 , 39 ] . These datasets consist of observations from multiple locations over time, naturally represented as multi-dimensional arrays where different dimensions capture spatial and temporal information along with one or more measured features (e.g., wind, temperature, and pressure in weather datasets). Since such time series are typically collected from numerous locations over extended periods [ 22 , 14 , 41 ] , they are often massive in scale. A central task using spatio-temporal time series is forecasting [ 39 , 11 , 53 ] , which aims to predict future time‑series values across locations based on past observations. To this end, diverse deep learning methods [ 60 , 58 , 4 , 57 , 46 ] have been developed. However, as dataset sizes and model complexities continue to scale in real-world scenarios, training these models leads to significant storage demands and computation costs. Dataset distillation has emerged as a promising approach to mitigate the heavy computational and storage costs of training deep learning models. The key idea is to create a small and informative synthetic dataset that can replace original data in the training process while preserving comparable model performance. Several well-known approaches to it include gradient matching [ 64 ] , trajectory matching [ 6 ] , and meta-learning [ 49 ] . Gradient matching minimizes the discrepancy between gradients computed on original and synthetic data, while trajectory matching aligns the training dynamics of models trained on both datasets. Meta-learning focuses on improving the transferability of models trained on synthetic data to original datasets. Although dataset distillation has been mainly studied for image and graph data [ 28 , 27 , 19 , 35 ] , recent work has explored its applications to time series [ 37 , 13 , 40 ] . However, since existing dataset distillation methods typically compress only a single dimension, they are less suitable for spatio-temporal datasets, where both spatial and temporal dimensions jointly contribute to the large data volume. In particular, time-series distillation techniques compress only the temporal dimension, while leaving the spatial dimension (i.e., the number of locations) unchanged, which can lead to substantial training time and GPU memory usage, as evidenced by the preliminary experiment below. Preliminary Experiment 1 (Necessity of Joint Reduction) . In Figure 1 , we compare two random sampling strategies with the same overall reduction ratio : (1) reducing only the temporal dimension by 0.5%, and (2) jointly reducing both dimensions by 5% (spatial) and 10% (temporal). After applying the strategies to five real-world spatio-temporal time-series datasets (see Section V-A ), we measure the training time per outer iteration and GPU memory usage of MTGNN [ 56 ] , a representative deep learning model for spatio-temporal forecasting. Figure 1 shows that, even with the same overall reduction ratio, jointly reducing the spatial and temporal dimensions significantly reduces training time and GPU memory usage, compared to temporal-only reduction. Refer to Section V-A for detailed experimental settings, and see also Theorems 1 and 2 in Section IV-F for related analyses. In this work, we propose STemDist ( S patio- Tem peral Dataset Dist illation), the first dataset distillation method specialized for spatio-temporal time series. A crucial step needed to compress the spatial dimension is to design a surrogate model that can be trained on a smaller number of locations (i.e., spatially compressed synthetic data) yet remains applicable for inference on a larger number of locations, which is not supported by most existing spatio-temporal forecasting models. To this end, we integrate a location encoder module that produces embeddings independent of the number of locations into Spatio-Temporal Graph Neural Networks (STGNNs), which are widely-used deep-learning models for spatio-temporal forecasting [ 66 , 38 , 55 , 54 , 8 , 25 , 10 ] . Using the surrogate model with the location encoder module, we build STemDist upon gradient matching (described above) with two enhancements. First, for efficiency, we reduce distillation time by performing distillation at the cluster level. Specifically, locations in the original dataset are grouped into clusters, and the averaged data within each cluster, rather than data from individual locations, are distilled. Second, to offset potential quality loss from this coarse-grained approach, we employ subset-based granular distillation. This technique distills data using varying subsets of clusters, ensuring that all parts (i.e., subsets) of the data are adequately reflected and thereby enhancing the quality of the output synthetic dataset. Using five real-world spatio-temporal datasets and nine baseline approaches, we conduct a comprehensive evaluation of STemDist, which reveals the following advantages over the baselines under the same compression ratio: • Faster: Model training is up to 6\times faster on synthetic data distilled by STemDist than on data distilled by the baselines. • Memory-efficient: Model training requires up to 8\times less GPU memory on synthetic data distilled by STemDist than on data distilled by the baselines. • Effective: Models trained on synthetic data distilled by STemDist achieve up to 12% lower prediction error compared to those trained on data distilled by the baselines. The rest of the paper