UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines
Chen Tang, Xinzhu Ma, Encheng Su, Xiufeng Song, Xiaohong Liu, Wei-Hong Li, Lei Bai, Wanli Ouyang, Xiangyu Yue · 2025
Read UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines by Chen Tang, Xinzhu Ma, Encheng Su, Xiufeng Song, Xiaohong Liu, Wei-Hong Li, Lei Bai, Wanli Ouyang, Xiangyu Yue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20748.txt.
Introduction
Chen Tang leads a team of nine authors in a 2025 paper that introduces UniSTD, a unified model for spatio-temporal learning. Most past models were built for one task at a time, which held back their use across fields with distinct needs.
UniSTD is a Transformer model trained in two stages. First it pretrains on plain 2D images and image-text data to build a broad base. Then joint training on spatio-temporal data tunes it for such tasks. A rank-adaptive mixture-of-experts step helps it adapt, and a new temporal part aids how it reads change over time.
Catalog record
| CREATOR | Chen Tang, Xinzhu Ma, Encheng Su, Xiufeng Song, Xiaohong Liu, Wei-Hong Li, Lei Bai, Wanli Ouyang, Xiangyu Yue |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines Chen Tang, Xinzhu Ma, Encheng Su, Xiufeng Song, Xiaohong Liu, Wei-Hong Li, Lei Bai, Wanli Ouyang, Xiangyu Yue Abstract Traditional spatiotemporal models generally rely on task-specific architectures, which limit their generalizability and scalability across diverse tasks due to domain-specific design requirements. In this paper, we introduce UniSTD , a unified Transformer-based framework for spatiotemporal modeling, which is inspired by advances in recent foundation models with the two-stage pretraining-then-adaption paradigm. Specifically, our work demonstrates that task-agnostic pretraining on 2D vision and vision-text datasets can build a generalizable model foundation for spatiotemporal learning, followed by specialized joint training on spatiotemporal datasets to enhance task-specific adaptability. To improve the learning capabilities across domains, our framework employs a rank-adaptive mixture-of-expert adaptation by using fractional interpolation to relax the discrete variables so that can be optimized in the continuous space. Additionally, we introduce a temporal module to incorporate temporal dynamics explicitly. We evaluate our approach on a large-scale dataset covering 10 tasks across 4 disciplines, demonstrating that a unified spatiotemporal model can achieve scalable, cross-task learning and support up to 10 tasks simultaneously within one model while reducing training costs in multi-domain applications. Code will be available at https://github.com/1hunters/UniSTD . 1 Introduction In recent years, substantial progress has been made across a wide array of computer vision tasks, including image classification [ 59 , 17 , 7 ] , object detection [ 81 , 47 , 18 ] , and more [ 30 , 34 , 19 , 52 ] . Among these, spatiotemporal learning—which focuses on predicting future events based on historical data—has emerged as crucial for numerous fields and practical applications, such as human motion prediction, traffic management, robotic planning, weather forecasting, etc . By capturing both spatial correlations and temporal dynamics, spatiotemporal learning offers a robust framework for modeling the complexities of the physical world, enabling more accurate and responsive predictions in diverse, real-world contexts. Figure 1 : Top: Existing works need specialized models for both tasks within the same disciplines ( e.g . , weather forecasting, traffic control). Bottom: Unified Spatio-Temporal Learning. UniSTD unifies 4 disciplines with 10 tasks under one model and is trained on a massive collection of datasets. While achieving meaningful progress, most of the existing methods [ 51 , 68 , 69 , 70 , 71 , 15 , 72 , 75 , 64 , 65 , 2 , 28 , 57 , 10 ] still resort to designing specialized architectures to learn the spatiotemporal representations for each task. Recurrent-based approaches have traditionally relied on Long Short-Term Memory (LSTM) networks [ 20 ] for sequential modeling. ConvLSTM [ 51 ] pioneered the application of convolutional LSTM networks for spatiotemporal predictive learning, it extends fully connected LSTM to incorporate convolutional layers. PredRNN [ 68 ] introduced Spatiotemporal LSTM (ST-LSTM) units, enabling a unified memory pool to capture both spatial appearances and temporal dynamics. Furthermore, E3D-LSTM [ 70 ] integrated 3D convolutional networks within LSTM units, facilitating robust representations that capture both short-term frame dependencies and long-term structural relationships. In contrast, recurrent-free approaches have emerged to circumvent iterative temporal predictions. Instead of generating temporal sequences incrementally, these methods simultaneously produce spatial and temporal predictions. SimVP [ 11 ] and SimVPv2 [ 56 ] employed a CNN backbone as a feature translator within an Encoder-Decoder framework. Earthformer [ 10 ] decomposes data into cuboidal regions to parallelize spatiotemporal forecasting for earth system applications. TAU [ 57 ] further explores generalized architectures by introducing the Temporal Attention Unit. While effective in eliminating recursion, existing methods primarily focus on re-engineering architectures to tackle specific tasks. In other words, these models are each designed for a single task, heavily relying on task-specific domain knowledge [ 10 ] . Consequently, they suffer from unstable performance when they are directly applied to other tasks [ 58 , 73 ] , limiting their generality and inevitably leading to expensive computational and memory cost. This leads us to pose a critical, yet challenging question: Can we comprehensively address diverse spatiotemporal domains with a single, universal framework? A model with such generality would offer several distinct advantages: (i) it allows the usage of standard network architecture ( e.g . , Transformer) and hence enables the integration of extensive pretrained knowledge from widely available open-source weights within the research community, (ii) joint training across multiple spatiotemporal tasks could promote cross-task learning benefits, and (iii) It enhances scalability and operational efficiency for large-scale deployments, where simultaneous processing of tasks across various domains is required. The comparison between existing works and the proposed method is shown in Fig. 1 . To unify the diverse data patterns of spatiotemporal tasks within a single model, inspired by recent advances in large language models (LLMs) [ 44 , 45 , 3 ] and vision-language models (VLMs) [ 16 , 40 , 35 ] —where a well-pretrained Transformer can serve as a universal encoder for further multi-modal multi-task fine-tuning—we frame spatiotemporal learning as a two-stage optimization problem. Since transformers has been standard blocks for various modalities, e.g . , vision and text, we employ a standard transformer model to build a generalizable architecture, and further demonstrate that this choice allows us to leverage the extensive knowledge embedded in large-scale, task-agnostic pretraining on datasets such as 2D vision data and image-text pairs ( e.g . , OpenCLIP-ViT [ 24 ] and ImageNet-ViT [ 7 ] ). In the second stage, which is the main focus of this paper, we perform joint training on the single model using task-specific spatiotemporal datasets for embedding domain-specific knowledge into the model, to fit the spatiotemporal tasks and improve the adaptability of the model. Despite the promising prospect, performing joint training on a single model is extremely challenging due to the intricate properties of diverse disciplines ( e.g . , weather forecasting vs. traffic control), easily triggering conflicts between disciplines and resulting in sub-optimal convergence. To address this with minimal training cost, we propose a rank-adaptive Mixture-of-Experts (MoE) mechanism that dynamically optimizes low-rank adapter ranks based on task properties and interdependencies while selectively activating adapters according to input characteristics. To mitigate the complexity of rank optimization, we reformulate the problem using an auxiliary matrix-based approach, which reduces complexity and enables fine-grained rank adjustments. Our method achieves full differentiability by incorporating a continuous relaxation of discrete rank values, facilitating efficient optimization with minimal computational overhead. Additionally, to imbue models originally trained on 2D data with temporal awareness without introducing substantial computational overhead, we design a lightweight temporal module that incorporates zero-initialized projection MLP layers. This design eliminates the need to fine-tune the transformer’s computationally intensive FFN layers, thereby enhancing temporal modeling capabilities while maintaining efficiency. Based on OpenSTL [ 58 ] and PredBench [ 73 ] , we create a large-scale dataset encompassing 4 representative disciplines with a total of 10 tasks to support the joint training. Extensive experiments demonstrate the effectiveness of the proposed method, UniSTD achieves lossless performance when scaling up the number of tasks. Specifically, the existing methods have encountered significant performance drops at only 3 tasks jointly trained together, while at that time, UniSTD has up to 18.8 PSNR advanced compared to existing methods. The overall framework of UniSTD is