Quantifying the Pre-training Dividend: Generative versus Latent Self-Supervised Learning for Time Series Foundation Models
Noam Major, Kathy Razmadze, Yoli Shavit · 2026
Read Quantifying the Pre-training Dividend: Generative versus Latent Self-Supervised Learning for Time Series Foundation Models by Noam Major, Kathy Razmadze, Yoli Shavit (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19462.txt.
Introduction
Major, Razmadze and Shavit benchmark seven self-supervised methods from five paradigms for time-series tasks under matched parameter and data budgets. Transfer success hinges on the downstream task: pretraining helps anomaly detection substantially and aids classification under fine-tuning, but gives little or no edge over untrained controls in forecasting. Linear probing also predicts fine-tuning poorly, synthetic pretraining often rivals real-world data, and deeper encoders worsen forecasting. The authors distill these findings into practical guidelines for evaluating and building time-series self-supervised systems.
Catalog record
| CREATOR | Noam Major, Kathy Razmadze, Yoli Shavit |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96117 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19462
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying the Pre-training Dividend: Generative versus Latent Self-Supervised Learning for Time Series Foundation Models Noam Major, Kathy Razmadze, Yoli Shavit Abstract The success of self-supervised learning (SSL) in vision and NLP has motivated its rapid adoption for time series. However, research has focused primarily on Generative paradigms and forecasting tasks, leaving the broader utility of learned representations unquantified. We establish a controlled framework to evaluate the "pre-training dividend": the value added by SSL across diverse temporal tasks. We systematically compare Generative paradigms against Latent Alignment architectures, introducing adaptations of LeJEPA and DINO for time series. These adaptations utilize Discrete Wavelet Transform (DWT) augmentations to enforce invariance to local fluctuations. Our analysis reveals that the pre-training dividend is highly asymmetric: SSL yields gains of up to 375% for anomaly detection and classification, yet remains marginal for forecasting. We demonstrate that representational utility is non-universal, governed by a precision-invariance trade-off where the specific signal resolution required by the task must align with the objective. Finally, we show that representation quality is largely independent of data origin and saturates at moderate architectural depths, suggesting a path to scaling via massive synthetic generation. Our code is available at: https://github.com/noammajor/Models . 1 Introduction The success of self-supervised learning (SSL) in natural language processing 17 and computer vision 4 ; 10 has motivated its widespread adoption for time-series analysis. However, temporal signals lack the discrete vocabularies or spatial hierarchies found in other domains; they are continuous, non-stationary, and often dominated by stochastic noise. Despite these distinctions, time series SSL has been largely evaluated through incremental gains in downstream forecasting 1 ; 5 , leaving the broader utility of learned representations poorly understood. This motivates a critical re-examination of SSL paradigms for time series analysis. Generative methods, such as Masked Auto-Encoding (MAE) 13 , Diffusion Models 21 and Next Token Prediction (NTP) 5 ; 1 , restore signals in the target domain but risk overfitting to local fluctuations. Conversely, Latent Alignment methods, such as invariance-based distillation (DINO) 4 and Joint-Embedding Predictive Architectures (JEPA) 10 ; 2 , learn structural relationships in representation space but may collapse informative variability. To date, the lack of a principled comparison between these paradigms means the added value of SSL remains largely unquantified across diverse tasks. In this work, we establish a controlled framework to address three central research questions: RQ1. What is the added value of SSL across diverse time series task families? RQ2. How does the choice of SSL objective dictate downstream performance? RQ3. To what extent is representation quality driven by data origin and architectural scale? To resolve these questions, we fix the backbone architecture and data pipeline to isolate the SSL objective as the primary variable. We evaluate Generative paradigms (NTP, MAE, Diffusion) alongside Latent Alignment methods (JEPA, LeJEPA, DINO). For methods which leverage augmented views (LeJEPA and DINO), we recognize that standard vision-based augmentations (e.g., cropping) often fail to respect temporal dependencies and propose enforcing frequency-domain invariance via Discrete Wavelet Transform (DWT). In our Wavelet-based DINO and Le-JEPA adaptations, the model aligns a Teacher view, generated via soft-thresholding of wavelet coefficients, with a "perturbed" Student view subjected to high-frequency noise and detail zeroing. By aligning these views in latent space, the model learns to remain invariant to local fluctuations while preserving the frequency components that define the system’s state. Our analysis reveals three fundamental properties of SSL for time series. First, the pre-training dividend is highly asymmetric. SSL establishes a robust foundation for anomaly detection and classification, but provides only marginal gains in forecasting. Second, representational utility is non-universal. Transferability depends on whether an objective preserves the signal resolution required by the downstream task. Latent Alignment paradigms prioritize global structural characteristics, which proves effective for morphological anomaly detection. However, a divergence emerges in classification: LeJEPA excels in linear probing while DINO proves superior under full fine-tuning. We hypothesize that the same mechanisms enabling LeJEPA’s performance on semantic tasks also lead to its severe performance degradation in forecasting. Conversely, Generative paradigms prioritize the high-resolution fidelity necessary for granular anomaly detection, yet lack the semantic abstraction required for complex classification. Finally, we observe that data scale overshadows data origin, as pre-training on massive synthetic datasets achieves parity with real-world data, and that performance saturates at moderate architectural depths. Collectively, these findings suggest that current SSL methods are bottlenecked by the their objective design and data scale, rather than model capacity, and that hybrid, paradigms and massive-scale synthetic data generation are promising directions. In summary, our contributions are as follows: • We introduce a unified and controlled benchmarking framework to quantify the added value of SSL across diverse time series task families, providing empirical answers to how different paradigms shape temporal representations. • We propose a novel adaptation of Le-JEPA and DINO for time series SSL by introducing augmentations which are more suitable for temporal signals. • We establish empirical scaling laws for temporal SSL, demonstrating that increasing architectural depth yields diminishing returns beyond a specific threshold under current regimes and showing that large-scale synthetic pre-training achieves parity with real-world data. 2 Related Work 2.1 Paradigms of Self-Supervised Learning in Time Series We categorize the temporal SSL landscape by the pretext objective. Generative Paradigms. These methods utilize signal-space alignment as a learning signal. Masked Auto-encoding (MAE), such as PatchTST 13 and SimMTM 6 , reconstructs masked segments, while models like Chronos 1 and TimesFM 5 utilize Next-Token Prediction (NTP) and NTP-like objectives (forecasting) for continuous or quantized signals. Recently, diffusion-based frameworks like TimeDART 21 have introduced denoising objectives. While effective for forecasting, these paradigms risk overfitting to stochastic signal noise. Latent Alignment Paradigms. Shifting the objective to the latent space, predictive approaches like JEPA 10 and invariance-based methods like DINO 4 align representations directly. This is critical for continuous time series, where reconstructive targets are highly susceptible to local fluctuations. Early explorations in this space include CPC 15 , TS-JEPA 8 , and TimeSiam 7 . We decouple the objective from proprietary architectures and focus on JEPA, LeJEPA 2 and DINO, while introducing the necessary adaptations for time series analysis implemented through wavelets-based augmentations. We compare these augmentations to global/local view-based transformations, introduced for vision 4 and time series classification 12 in Appendix A.7 . Data Curation and Scaling. The field has moved from single-domain datasets toward multi-domain corpora like the Monash Archive 9 . Concurrently, synthetic data (e.g., Gaussian processes in Chronos 1 or TimePFN 16 ) has emerged to enable zero-shot generalization. Our study investigates the interaction between these data sources and specific SSL objectives. 2.2 The Attribution Gap and Evaluation Biases Existing surveys provide high-level taxonomies of contrastive and generative families 20 ; 22 ; 11 , but lack a framework for isolating the specific impact of the pre-training objective. Furthermore, they do not cover emerging latent-alignment methods like DINO, JEPA and LeJEPA. Moreover, most temporal SSL research introduces new objectives alongside specialized architectures, creating an attribution gap where gains cannot be cleanly disentangled from architectural innovations. We address this by isolating the objective from data and architecture, covering both Generative and Latent paradigms while purposefully excluding contrastive methods (e.g., 19 ) to eliminate confounding factors like negative sampling bias. In addition, SSL evaluation is dominated by