Multi-modal Time Series Analysis: A Tutorial and Survey
Yushan Jiang 1* , Kanghui Ning 1* , Zijie Pan 1* , Xuyang Shen 1 , Jingchao Ni 2 , Wenchao Yu 4 , Anderson Schneider 3 , Haifeng Chen 4 † {\dagger} , Yuriy Nevmyvaka 3 † {\dagger} , Dongjin Song 1 † {\dagger} · 2025
Read Multi-modal Time Series Analysis: A Tutorial and Survey by Yushan Jiang 1* , Kanghui Ning 1* , Zijie Pan 1* , Xuyang Shen 1 , Jingchao Ni 2 , Wenchao Yu 4 , Anderson Schneider 3 , Haifeng Chen 4 † {\dagger} , Yuriy Nevmyvaka 3 † {\dagger} , Dongjin Song 1 † {\dagger} (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13709.txt.
Introduction
Yushan Jiang and co-authors survey a growing field in a 2025 tutorial on time series that come in many forms at once. Real data often pairs a changing signal with text, images, or tables, and each form carries part of the story. Mixing them is hard: the forms differ, drift out of step, and carry noise.
The survey maps the datasets in use and sorts methods by how the forms interact. Mixing can happen at input, midway, or at output, through blending, lining up, or passing knowledge across. The authors also cover uses for both plain and place-based series, in general and in special fields. A public resource list tracks the fast-moving literature. The paper closes with open problems for newcomers and experts alike.
Catalog record
| CREATOR | Yushan Jiang 1* , Kanghui Ning 1* , Zijie Pan 1* , Xuyang Shen 1 , Jingchao Ni 2 , Wenchao Yu 4 , Anderson Schneider 3 , Haifeng Chen 4 † {\dagger} , Yuriy Nevmyvaka 3 † {\dagger} , Dongjin Song 1 † {\dagger} |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13709
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-modal Time Series Analysis: A Tutorial and Survey Yushan Jiang 1* , Kanghui Ning 1* , Zijie Pan 1* , Xuyang Shen 1 , Jingchao Ni 2 , Wenchao Yu 4 , Anderson Schneider 3 , Haifeng Chen 4 † {\dagger} , Yuriy Nevmyvaka 3 † {\dagger} , Dongjin Song 1 † {\dagger} Abstract . Multi-modal time series analysis has recently emerged as a prominent research area in data mining, driven by the increasing availability of diverse data modalities, such as text, images, and structured tabular data from real-world sources. However, effective analysis of multi-modal time series is hindered by data heterogeneity, modality gap, misalignment, and inherent noise. Recent advancements in multi-modal time series methods have exploited the multi-modal context via cross-modal interactions based on deep learning methods, significantly enhancing various downstream tasks. In this tutorial and survey, we present a systematic and up-to-date overview of multi-modal time series datasets and methods. We first state the existing challenges of multi-modal time series analysis and our motivations, with a brief introduction of preliminaries. Then, we summarize the general pipeline and categorize existing methods through a unified cross-modal interaction framework encompassing fusion, alignment, and transference at different levels ( i.e. , input, intermediate, output), where key concepts and ideas are highlighted. We also discuss the real-world applications of multi-modal analysis for both standard and spatial time series, tailored to general and specific domains. Finally, we discuss future research directions to help practitioners explore and exploit multi-modal time series. The up-to-date resources are provided in the GitHub repository † † * equal contribution. † Yuriy Nevmyvaka, Haifeng Chen and Dongjin Song are the corresponding authors. 1 1 1 https://github.com/UConn-DSIS/Multi-modal-Time-Series-Analysis . 1. Introduction Figure 1. The framework of our tutorial and survey. Time series analysis is a fundamental task in data mining, driven by the proliferation of sequential data exhibiting rich temporal dynamics across diverse real-world systems. With the advent of deep learning, various methods have been proposed to effectively model complex temporal relationships within time series ( Qin et al. 2017 ; Wu et al. 2020 ; Wu et al. 2023 ; Nie et al. 2023 ; Chen et al. 2023a ; Pan et al. 2024 ; Yi et al. 2024 ) , facilitating downstream tasks in diverse domains, including healthcare ( Jin et al. 2018 ; Zhang et al. 2020 ; Feuerriegel et al. 2024 ) , finance ( Zhang et al. 2017 ; Rezaei et al. 2021 ) , transportation ( Guo et al. 2019 ; Zhang et al. 2021 ; Ji et al. 2023 ) and environmental sciences ( Chen et al. 2022 ; Chen et al. 2023b ) . In practice, time series are often associated with external contexts beyond their temporal dynamics ( Williams et al. 2024 ; Chattopadhyay et al. 2025 ) . Such contexts are multi-modal, encompassing a variety of representations, such as texts ( Lee et al. 2024 ; Wang et al. 2024 ) , images ( Ekambaram et al. 2020 ; Skenderi et al. 2024 ) , tables ( Chattopadhyay et al. 2025 ) , and graphs ( Sawhney et al. 2020 ) , which carry rich semantic information for time series analysis. As such, incorporating the multi-modal contexts allows models to have a comprehensive view of underlying systems, capture subtle dependencies, and explain complex temporal behaviors more accurately. Effective analysis of multi-modal time series, however, is hindered by several key challenges in terms of data heterogeneity , modality gap and contextual relevance . First, different modalities exhibit distinct statistical properties, structures, and dimensionalities, leading to discrepancies in feature distributions and semantic meanings. For instance, while time series data is sequentially ordered with temporal dependencies, textual and image data contains rich contextual semantics and correlations. Aligning these heterogeneous data into a unified representation space is non-trivial. Second, the textual, tabular, or visual contexts may appear at different timesteps or granularities. Such temporal misalignment may impede meaningful cross-modal interactions. Third, real-world data is inevitably noisy with irrelevant information that may mislead correlation learning, resulting in suboptimal performance. For example, in finance, news articles related to stock market prediction often contain much redundant or speculative narratives that does not reflect actual market conditions. Therefore, the focus of multi-modal time series analysis is to effectively capture complementary and relevant information from multi-modal context and leverage it for predictive or analytical tasks. More recently, an increasing number of multi-modal methods have shown promise in exploiting contextual information from diverse data sources, which boosts performance in wide tasks ranging from forecasting ( Lee et al. 2024 ; Liu et al. 2025 ) , classification ( Lee et al. 2025 ; Li et al. 2023b ) , anomaly detection ( Xing and He 2023 ) to retrieval ( Bamford et al. 2023 ) and causal discovery ( Shen et al. 2024 ; Zheng et al. 2024 ) . Despite the promising results of multi-modal time series methods, they are tailored for their own tasks with domain-specific applications. The existing literature lacks a comprehensive and systematic review that provides a unified perspective on the underlying principles and pipelines for multi-modal time series learning. In this survey, we provide a systematic and up-to-date overview of existing methods for multi-modal time series analysis. As shown in Figure 1 , we discuss the challenges, motivations, and preliminaries of multi-modal time series. Then we introduce the general pipeline for multi-modal time series analysis and propose three types of interactions for cross-modal modeling between time series and other modalities — fusion , alignment , and transference — at the input, intermediate and output level, respectively. We also discuss the applications of multi-modal time series across multiple domains. Furthermore, we provide Table 2 to comprehensively summarize representative methods, encapsulating the modalities, fine-grained cross-modal interactions, real-world domains and tasks. Finally, we highlight potential future research opportunities to further advance time series analysis with multi-modal data. In summary, the major contributions of our survey are: • We systematically catalog over 40 multi-modal time series methods with the corresponding open-source datasets. • We uniquely categorize the existing methods into a unified cross-modal interaction framework, highlighting fusion, alignment, and transference at the input/intermediate/output levels. • We discuss real-world applications of multi-modal time series and identify promising future directions, encouraging researchers and practitioners to explore and exploit multi-modal time series. 2. Background and Our Scope 2.1. Multi-modal Machine Learning Recent advancements in multi-modal machine learning have significantly enhanced models’ ability to process and integrate data from diverse modalities, such as language, acoustic, vision, and tabular data ( Hager et al. 2023 ; Ruan et al. 2023 ; Zhang et al. 2024a ) . With the development of deep learning architectures and sophisticated interaction designs, models are able to learn, infer, and reason by integrating multiple communicative modalities. Current research in multi-modal machine learning spans multiple key areas, including (1) representing multi-modal data to encode joint and individual characteristics, (2) identifying interconnections between modality elements, (3) transferring knowledge across modalities, and (4) theoretically and empirically analyzing the underlying learning process in a quantitative manner. We refer the audiences to the recent surveys ( Baltrušaitis et al. 2018 ; Liang et al. 2024 ) for a more detailed overview of general multi-modal machine learning