Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density
Jingru Fei, Kun Yi, Alex Xing Wang, Qingsong Wen, Xiangxiang Zhu, Wei Fan · 2026
Read Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density by Jingru Fei, Kun Yi, Alex Xing Wang, Qingsong Wen, Xiangxiang Zhu, Wei Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17340.txt.
Introduction
Fei and colleagues present Olivia, a time-series base model that lines up data sets before training so their mismatched rhythms do not clash. A Harmonizer module reshapes spectral structure across sets, matching their normalized power spectra, while HarmonicAttention runs self-attention through a small set of resonators. Tests on two big benchmarks plus six extra sets show top scores for zero-shot, few-shot and full-data forecasting.
Catalog record
| CREATOR | Jingru Fei, Kun Yi, Alex Xing Wang, Qingsong Wen, Xiangxiang Zhu, Wei Fan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120700 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17340
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density Jingru Fei, Kun Yi, Alex Xing Wang, Qingsong Wen, Xiangxiang Zhu, Wei Fan Abstract Time series foundation models rely on large-scale pretraining over diverse datasets across domains, yet their heterogeneity in temporal patterns could hinder the effectiveness of training and learning transferable time series representations. Inspired a fundamental concept, normalized power spectral density (PSD) in signal processing, we assume harmonizing datasets via PSDs in the spectral domain could reduce mismatches and enhance pretraining. We then go beyond the direct intractable minimization optimization and innovatively reformulate it as a principled harmonization approach. Specifically, we propose Harmonizer , a module that reshapes spectral structures and implicitly harmonizing PSDs across datasets, which theoretically corresponds to a shared reparameterization of second-order temporal correlations. Our theoretical analysis further reveals token interactions with Harmonizer can be efficiently mediated by a compact set of resonators, motivating a HarmonicAttention design that performs self-attention in a low-dimensional interaction space. Then, we propose Olivia , a novel time series foundation model built upon these harmonization mechanisms. Extensive experiments on two large-scale benchmarks (TSLib and GIFT-Eval) and extra 6 datasets from GluonTS, demonstrate Olivia consistently achieves state-of-the-art performance under zero-shot, few-shot, and full-shot forecasting scenarios. Our code is available at Olivia . 1 Introduction Figure 1 : (a): Original and Harmonizer-processed time series from different domains. Raw time series from multiple domains (left) show distinct temporal patterns. After processing (right), patterns show more consistent across domains. (b): Jensen–Shannon (JS) divergence between dataset-level normalized power spectral density distributions across different domain pairs. JS divergence is bounded in [0,\ln 2] (approximately [0,0.693] ), with larger values indicating greater spectral discrepancy. More details can be found in the Appendix C . Time series has been a fundamental modality in many real-world applications, including energy management, traffic prediction, climate modeling, finance, and healthcare ( 46 ; 61 ; 47 ) . Modern end-to-end deep learning models for time series forecasting, which could be built upon recurrent networks, convolution structures, Transformer architectures, etc ( 38 ; 34 ; 16 ) , have achieved strong performance on individual datasets, but they are typically trained in task- or dataset-specific settings, limiting their ability to generalize across domains. Inspired by large-scale pretraining in natural language processing ( 54 ) and computer vision ( 4 ) , recent work has been exploring time series foundation models ( 37 ; 3 ) , which aim to learn an unified, general-purpose model from diverse time series data and transfer to downstream forecasting tasks effectively. Generally, time series foundation models are pretrained on large collections of datasets spanning multiple application domains ( 55 ) . The diversity from different data is a significant characteristic in large-scale pretraining, which aims for exposing the model to a wide range of temporal behaviors ( 58 ) . As illustrated in the left part of Figure 1 (a), time series originating from different domains exhibit markedly different temporal patterns, reflecting variations in periodic structures and long-term temporal dependencies that arise from domain-specific data generation processes, such as distinct physical dynamics, operational cycles, environmental influences, and measurement settings ( 31 ) . While such diversity is a prerequisite for learning broadly applicable temporal knowledge, it also introduces substantial challenges to pretraining. From an optimization perspective, jointly training on datasets with disparate temporal characteristics often leads to slower convergence and suboptimal optimization behavior. From a representation learning perspective, the model might need to simultaneously accommodate incompatible temporal structures, which could make it difficult to form unified and transferable representations that could generalize well across datasets. Then, these challenges naturally motivate us to ask an important research question: how can time series foundation models learn more effectively on diverse pretraining datasets? To answer this question, the core idea is to make time series foundation models characterize and address such diversity across data and domains in a principled manner. Inspired by a fundamental concept in the domain of signal processing ( 44 ; 21 ) , we have found the normalized power spectral density (PSD) could serve as an informative dataset-level descriptor by capturing the frequency-wise distribution of temporal variations that could reflect the underlying second-order temporal structure and provide a stable and discriminative summary across datasets ( 8 ; 17 ) , which directly suggests a potential principled remedy: rather than treating dataset diversity as an unstructured domain gap, we assume that harmonizing datasets via PSDs in the spectral domain, could reduce mismatches between their PSDs and thus improve the efficacy of large-scale time series pretraining. Alone this line, a following straightforward solution is to measure PSDs and minimize their discrepancies among datasets. As an instance, the distributional discrepancies between PSDs of datasets can be quantified using standard measures such as the Jensen–Shannon (JS) divergence ( 32 ; 14 ) . Experimentally, as shown in Figure 1 (b), different domain pairs exhibit substantial JS divergence in their PSDs quantitatively indicating pronounced spectral discrepancies, and minimizing such divergences seems to be a reasonable objective for promoting PSD consistency. However, directly pursuing this objective is non-trivial in large-scale time series pretraining. Since this objective is defined over aggregated, dataset-level spectral statistics, whereas time series foundation models are usually trained via instance (sequence)-level optimization, this mismatch makes direct optimization hard to implement. To address above challenges, we go beyond direct divergence minimization for PSDs and innovatively reformulate it as a structural and principled harmonization approach. We demonstrate that cross-domain spectral discrepancies can be effectively addressed by projecting time series onto a shared representation space, based on which, we propose an according module termed Harmonizer . Specifically, Harmonizer introduces learnable orthogonal temporal transformations that reorganize temporal dynamics, reshaping spectral structures and implicitly harmonizing PSDs across datasets. From a theoretical perspective, it corresponds to a shared reparameterization of second-order temporal correlations, which induces a common structural form across datasets. Moreover, Harmonizer further reshapes temporal relationships in attention mechanisms in Transformer-based time series foundation models ( 22 ) . Our theoretical analysis reveals that token interactions can be efficiently mediated by a compact set of resonators, motivating a HarmonicAttention design that performs self-attention ( 51 ) in a low-dimensional interaction space instead of dense all-to-all interactions. Derived from harmonized temporal representations, these resonators act as a condensed intermediary for modeling temporal relationships. Building upon these insights, we introduce a novel time series foundation model, Olivia , for forecasting. Olivia starts with incorporating Harmonizer , which comprises two bidirectional submodules to perform temporal reorganization before encoding and representation recovery after decoding. For the core architecture, Olivia follows a Transformer-style encoder–decoder design, but departs from standard formulations by employing HarmonicAttention as its primary interaction mechanism which results in a HarmonicFormer architecture that models temporal relationships through low-dimensional harmonic interactions rather than dense all-to-all attention. In summary, our contributions are: • We introduce a principled harmonization approach for addressing diversity across pretraining time series datasets by implicitly harmonizing PSDs across data through the reorganization of second-order temporal correlations. • We propose HarmonicAttention , a novel attention mechanism enabling low-dimensional token interactions mediated by a compact set of resonators, and further introduce HarmonicFormer for efficient foundational modeling. • We propose Olivia , a novel time series foundation model built upon