Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives

Sixun Dong Wei Fan Teresa Wu Yanjie Fu · 2025

Read Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives by Sixun Dong Wei Fan Teresa Wu Yanjie Fu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-24124.txt.

Introduction

Sixun Dong and co-authors teach time series models to use pictures and text in this 2025 paper. Forecasting from raw numbers alone can miss high-level patterns. People, by contrast, read charts by eye and grasp shapes at a glance.

Their method turns each numeric series into both a visual view and a text view. Contrastive learning then aligns the two views in a shared space. A selection module picks the most useful variables for multi-series forecasts. Tests on fifteen short-term and six long-term benchmarks beat strong single-mode rivals. The gains suggest aligned views give a richer signal than numbers alone.

Catalog record

CREATORSixun Dong Wei Fan Teresa Wu Yanjie Fu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-24124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives Sixun Dong Wei Fan Teresa Wu Yanjie Fu Abstract Time series forecasting traditionally relies on unimodal numerical inputs, which often struggle to capture high-level semantic patterns due to their dense and unstructured nature. While recent approaches have explored representing time series as text using large language models (LLMs), these methods remain limited by the discrete nature of token sequences and lack the perceptual intuition humans typically apply, such as interpreting visual patterns. In this paper, we propose a multimodal contrastive learning framework that transforms raw time series into structured visual and textual perspectives. Rather than using natural language or real-world images, we construct both modalities directly from numerical sequences. We then align these views in a shared semantic space via contrastive learning, enabling the model to capture richer and more complementary representations. Furthermore, we introduce a variate selection module that leverages the aligned representations to identify the most informative variables for multivariate forecasting. Extensive experiments on fifteen short-term and six long-term forecasting benchmarks demonstrate that our approach consistently outperforms strong unimodal and cross-modal baselines, highlighting the effectiveness of multimodal alignment in enhancing time series forecasting. Code is available at: https://github.com/Ironieser/TimesCLIP . 1 Introduction Time series data, consisting of records captured over time, are prevalent in our lives and various application domains [ 55 ] . As one of the essential tasks in time series data, time series forecasting, involves predicting future data points by analyzing historical data. Given its widespread demand in real-world scenarios, time series forecasting has attracted considerable research interest from the artificial intelligence and deep learning community, particularly in fields such as meteorology [ 132 , 117 , 133 ] , energy [ 112 , 55 , 85 ] , healthcare [ 44 , 11 ] , and financial investment [ 17 , 74 ] . Method Input LM Train Emb. LM Use Vision MM Feat. Time-LLM Prompted LLaMA ✗ LM-based TSF. ✗ ✗ Chronos (T5) Tokenized T5 ✓ LM-based TSF. ✗ ✗ PatchTST Numerical ✗ ✓ ✗ ✗ ✗ iTransformer Numerical ✗ ✓ ✗ ✗ ✗ TimesCLIP Numerical CLIP-Text ✓ Extract Feat. ✓ ✓ (c) Comparison of input modalities and modeling strategies. Figure 1 : Comparison of frameworks and modeling strategies. Overview of modeling paradigms for Time Series Forecasting(TSF). The left figure contrasts unimodal and multimodal frameworks, while the right table summarizes representative methods across core modeling dimensions. TimesCLIP uniquely constructs and aligns visual and textual features directly from numerical time series, enabling multimodal learning without requiring external modalities. Classic deep learning-based approaches to time series forecasting usually rely on designing unique architectures as different end-to-end forecasting models. Researchers have explored various architectures, including Transformer-based [ 77 , 70 ] , CNN-based [ 97 ] , MLP-based [ 79 , 131 ] , and RNN-based models [ 59 , 71 ] , to accomplish forecasting. However, these methods mostly treats time series data purely as numerical sequences; in other words, they can be depicted as unimodal frameworks, as shown in Figure 1 (a). Unfortunately, unimodal frameworks are limited in modeling complex patterns, contextual semantics, and long-term dependencies, and, thus, often result into fragile models. With the success of Large Language Models (LLMs), researchers in time series have brought up the new perspective of cross modality . There are studies that treat time series as a “foreign language” that can be decoded using the language representation strategies of large language models through fine-tuning [ 90 , 62 , 71 ] or reprogramming dedicated time series encoders [ 41 ] . Despite these advances, representing time series as a language is still less intuitive, as humans typically interpret time series data visually rather than textually [ 92 ] . For example, doctors usually adopt the visual information in ECG to diagnose heart conditions [ 83 ] ; financial professionals would conduct analysis based on the stock market charts instead of pure numbers [ 9 ] . Recent studies have suggested that directly applying LLM techniques to time series forecasting may not always yield effective results [ 92 , 43 ] . Recent cross-modality studies [ 56 , 86 , 7 , 57 , 47 ] have converted time series data into images for diverse tasks, but are not performant in time series forecasting [ 86 , 56 ] . More recent studies have utilized Fourier-transformed spectrograms [ 7 , 111 ] , which, however, don’t provide visually interpretative time series representations. Besides, efforts have been made to extract multimodal knowledge for time series related domain-specific applications. The studies in [ 66 , 12 ] attempted to classify different sensors; however, the sensors capture the same type of data-human motion, which does not conform to the true multimodality. In healthcare, the integration of medical time series and disease description texts to construct multimodal frameworks has been explored [ 83 ] . However, these applied studies benefit domain-specific data with annotated multimodal information, but are limited in generalized time series analysis and forecasting. In response to the above challenges, we introduce TimesCLIP , a novel Multimodal Contrastive Learning approach for time series modeling and forecasting. Inspired by the effective modeling of CLIP [ 81 ] that bridges the gap between text and visual understanding, TimesCLIP introduces a novel idea to jointly learn both numeric time series and visual representations. Specifically, it involves converting time series data into images and employing a multimodal contrastive learning framework. Figure 1 (b) shows that, in the vision branch, original numerical variates are transformed into distinct figures using different colors. In the language branch, similar to LLM-based approaches for time series, we treat numerical time series data as a "foreign" language. However, instead of employing a complex projection module to align time series with "real" language, we employ a simpler learning tokenizer to align time series with the feature space of pretrained language model. We further introduce a modified contrastive loss specifically designed for multimodal time series, aiming to align the vision representation with the language representation within a multimodal space. Subsequently, we introduce an innovative variate selection module. We designate the aligned classification feature as “Query” and use a cross-attention layer to identify the most correlative features for downstream tasks in the multivariate feature sequence. This module enables the model to effectively utilize multimodal features and classification information to identify the most significant one among multiple variables for downstream tasks. In brief, we summarize our contribution in threefold: • We present a novel concept for converting numerical multivariate time series into vision and language representations. We propose an innovative multimodal contrastive learning framework to align time series data with a vision-language multimodal space. • We introduce a variate selection module designed to leverage aligned multimodal features, enabling the identification of the most relevant variable among the multivariate time series for time series forecasting. • Extensive experiments demonstrate that our proposed model, TimesCLIP, achieves strong performance on both short-term and long-term forecasting tasks. 2 Related Work Multimodal Learning for Time Series. Contrastive learning has significantly advanced the processing and understanding of multimodal data, especially CLIP [ 81 ] , which illustrates the generalization ability of multimodal contrastive learning. [ 119 ]

Read the complete text