Joint Embeddings Go Temporal
Sofiane Ennadir, Siavash Golkar, Leopoldo Sarra · 2025
Read Joint Embeddings Go Temporal by Sofiane Ennadir, Siavash Golkar, Leopoldo Sarra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25449.txt.
Introduction
Sofiane Ennadir, Siavash Golkar, and Leopoldo Sarra carry a self-supervised learning design into time series data. Joint-embedding predictive architectures learn by predicting hidden representations of data rather than reconstructing the data itself, which makes them less vulnerable to noise and confounding details. Until now the approach had proven itself mainly on text and images.
Their version, TS-JEPA, is built for sequences measured over time. They validate it on both labeling and forecasting tasks across standard datasets. The method matched or beat current state-of-the-art baselines, and, notably, it performed well across both task families rather than specializing in one. The authors present the work as groundwork for general-purpose foundation models for time series built on this idea.
Catalog record
| CREATOR | Sofiane Ennadir, Siavash Golkar, Leopoldo Sarra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25449
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Joint Embeddings Go Temporal Sofiane Ennadir, Siavash Golkar, Leopoldo Sarra Abstract Self-supervised learning has seen great success recently in unsupervised representation learning, enabling breakthroughs in natural language and image processing. However, these methods often rely on autoregressive and masked modeling, which aim to reproduce masked information in the input, which can be vulnerable to the presence of noise or confounding variables. To address this problem, Joint-Embedding Predictive Architectures (JEPA) has been introduced with the aim to perform self-supervised learning in the latent space. To leverage these advancements in the domain of time series, we introduce Time Series JEPA (TS-JEPA), an architecture specifically adapted for time series representation learning. We validate TS-JEPA on both classification and forecasting, showing that it can match or surpass current state-of-the-art baselines on different standard datasets. Notably, our approach demonstrates a strong performance balance across diverse tasks, indicating its potential as a robust foundation for learning general representations. Thus, this work lays the groundwork for developing future time series foundation models based on Joint Embedding. 1 Introduction The paradigm of self-supervised learning (SSL) has emerged as a crucial technique for the development of Foundation Models bommasani2021opportunities . These models are built on large unlabeled datasets and then fine-tuned for specific tasks with smaller labeled datasets. This approach has been very successful for language and vision tasks, and has been recently applied also to time series das2023decoder ; goswami2024moment . SSL methodologies can be broadly categorized into two families. The first is based on contrastive approaches zhang2022survey , such as SimCLR chen2020simple developed for computer vision, which learns representations by juxtaposing positive and negative pairs of samples. Building upon these principles, researchers have also introduced novel adaptations yue2022ts2vec ; zhang2022survey ; zhang2022self that account for the specific properties of time series. The second family encompasses predictive or generative-based methods. These techniques, such as the Masked Auto-Encoder (MAE) he2022masked , task the model to reconstruct or predict missing segments of the input. This approach can be particularly effective for time series, given their inherent temporal ordering and often cyclical structure, such as in seasonal retail sales ensafi2022time . A variation of MAE is the Autoregressive Encoder, where the model’s objective becomes the prediction of the next time steps. This adaptation aligns particularly well with the forward-looking nature of many time series applications, enhancing the capacity to capture temporal dependencies specifically in the forecasting task. However, since masked models are designed to reconstruct missing parts in the input space, they can be susceptible to the presence of noise and other non-predictable confounding factors. Indeed, a good reconstruction requires to model the entire input, and such elements may prevent the extraction of meaningful and predictive features, because the model will focus on this noise rather than the underlying patterns within the data lecun2022path . Recently, the Joint-Embedding Predictive Architecture (JEPA) lecun2022path has been introduced as a way to only maintain the relevant information in the representation, leading to competitive results in both the image assran2023self and video domains bardes2024vjepa . The JEPA approach first encodes the input into a latent space and then performs masked reconstruction in this space. Latent space reconstruction makes this technique robust to the presence of confounding variables and noise in the input. Inspired by the success of JEPA in addressing confounding factors in images and videos, we believe that this approach can similarly benefit time series data, which are often inherently noisy. Here, we propose TS-JEPA, an adaptation of the JEPA architecture specifically tailored to the unique characteristics of temporal sequences. We validate our model through experiments on various standard datasets for time series classification and short and long-term forecasting. TS-JEPA exhibits promising performance compared to other baselines with the induced representation performing competitively in the forecasting task while outperforming significantly on classification. While a few papers, have already appeared mentioning JEPA in the context of time series, they were either a specific application to encoded frames girgis2024time , or in combination with other techniques for in-context prediction verdenius2024lat . Ours is the first systematic study of the JEPA architecture to time series, investigating whether this method can compete against the more common autoregressive approach. Our results positions TS-JEPA as a promising building block to be used for time series foundation models. 2 Joint Embedding Predictive Architecture for Time Series We consider the task of representation learning for time series. We focus on the univariate case, but our architecture is easily adaptable to multivariate time series. Formally, let \mathcal{X}=\{x_{1},x_{2},...,x_{N}\} represent a set of N time series, where each x_{j} is of length T . We aim to find a function f:\mathcal{X}\rightarrow\mathcal{H} mapping the input time series to a representation h\in\mathcal{H}\subset\mathbb{R}^{m} , where m is the hidden representation’s dimension. Once the self-supervised representation learning process has been completed, the learned representations can be used for downstream tasks such as classification or forecasting. The fundamental idea of JEPA is to use non-masked parts of the input to predict other masked parts, but in an abstract latent space rather than in the original input space. This approach allows the model to capture underlying patterns and relationships in the data that may not be immediately apparent in the raw input. Our architecture is composed of four primary components (Fig. 1 ): Figure 1: Illustration of TS-JEPA: it consists of ( 1 ) a tokenizer, ( 2 ) an encoder that processes the non-masked patches, ( 3 ) a Predictor that generates the target predictions from the encoder’s output and ( 4 ) the EMA-Encoder, which encodes the target masked patches. (i) Tokenizer: serves as the initial processing stage. It takes as input a time series x\in\mathcal{X} and transforms it into a sequence of non-overlapping patches {p_{i}\in\mathcal{P}} . To embed these patches, we use a one-dimensional convolutional neural network (1D-CNN). The aim is to capture local patterns and features along the temporal dimension within each patch. In addition, to preserve the temporal information, we incorporate a positional encoding mechanism. While alternative positional encoding schemes could easily be incorporated in our architecture, we have adopted an absolute sin-cos positional embedding vaswani2017attention for simplicity. The sequence of patches are split into a set of masked \mathcal{P}_{\mathcal{M}} and non-masked patches \mathcal{P}_{\mathcal{N}} following a uniform masking strategy. (ii) Encoder E_{\theta} : is the cornerstone of our architecture, transforming the patches into a useful representation that can be used afterwards for downstream tasks. We use a standard transformer architecture incorporating self-attention mechanisms. The encoder takes as input the non-masked patches \mathcal{P}_{\mathcal{N}}=\{p_{i}\in\mathcal{P}\mid i\in\mathcal{N}\} , where \mathcal{N} represents the indices of non-masked elements. The inputs are transformed into the latents z_{\mathcal{N}}=E_{\theta}(\mathcal{P}_{\mathcal{N}}) . (iii) Predictor P_{\beta} : takes the output of the encoder z_{\mathcal{N}} as input. The goal is to learn a mapping from the encoded observed (i.e. non-masked) tokens to the encoded unobserved (masked) ones. Similar to the Encoder, we use a transformer-based architecture with self-attention mechanisms. Specifically, the output of the predictor can be formulated as z^{\prime}_{\mathcal{M}}=P_{\beta}(E_{\theta}(\mathcal{P}_{\mathcal{N}})) . (iv) EMA-Encoder E_{\bar{\theta}} : The Exponential Moving Average encoder is an instance of the encoder with different weights. Instead of them being directly optimized through backpropagation like in the main encoder, the weights of the EMA-encoder are updated as an exponential moving average of the main encoder’s weights.