EMTSF: Extraordinary Mixture of SOTA Models for Time Series Forecasting
Musleh Alharthi, Kaleel Mahmood, Sarosh Patel, Ausif Mahmood · 2025
Read EMTSF: Extraordinary Mixture of SOTA Models for Time Series Forecasting by Musleh Alharthi, Kaleel Mahmood, Sarosh Patel, Ausif Mahmood (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23396.txt.
Introduction
Musleh Alharthi, Kaleel Mahmood, Sarosh Patel, and Ausif Mahmood present EMTSF, a mixture of state-of-the-art models for time series forecasting. Transformers moved from language into forecasting with strong results, yet a simple linear model once beat them, sparking debate. Later models such as PatchTST and TimeLLM restored the transformer lead, and follow-up papers challenged each claim in turn.
Against that contested backdrop, the authors combine several top models into one extraordinary mixture. Ensembling diverse forecasters can average out individual weaknesses. Their system targets robust accuracy across the many kinds of series forecasters face in practice.
Catalog record
| CREATOR | Musleh Alharthi, Kaleel Mahmood, Sarosh Patel, Ausif Mahmood |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23396
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EMTSF: Extraordinary Mixture of SOTA Models for Time Series Forecasting Musleh Alharthi, Kaleel Mahmood, Sarosh Patel, Ausif Mahmood Abstract The immense success of the Transformer architecture in Natural Language Processing has led to its adoption in Time Series Forecasting (TSF), where superior performance has been shown. However, a recent important paper questioned their effectiveness by demonstrating that a simple single layer linear model outperforms Transformer-based models. This was soon shown to be not as valid, by a better transformer-based model termed PatchTST. More recently, TimeLLM demonstrated even better results by repurposing a Large Language Model (LLM) for the TSF domain. Again, a follow up paper challenged this by demonstrating that removing the LLM component or replacing it with a basic attention layer in fact yields better performance. One of the challenges in forecasting is the fact that TSF data favors the more recent past, and is sometimes subject to unpredictable events. Based upon these recent insights in TSF, we propose a strong Mixture of Experts (MoE) framework. Our method combines the state-of-the-art (SOTA) models including xLSTM, enhanced Linear, PatchTST, and minGRU, among others. This set of complimentary and diverse models for TSF are integrated in a Transformer based MoE gating network. Our proposed model outperforms all existing TSF models on standard benchmarks, surpassing even the latest approaches based on MoE frameworks. Our code is available at: https://github.com/muslehal/EMTSF 1 Introduction Time Series Forecasting (TSF) models predict future values based on patterns learned from historical data. TSF is used in many different fields such as weather, health care, traffic, finance, electricity consumption, and market demand, among others. TSF has been a challenging area due to the time dependent nature of data which is often susceptible to factors like seasonality, trend changes, uncommon events, non-stationary nature, and noise issues. While TSF has been an active area of research for decades, it has drawn renewed attention due to the advances in the field of AI. Recent research works have explored AI architectures ranging from simple linear neural networks to enhanced transformer-based architectures, as well as state-space based models. As for the best AI-based architecture for TSF, there have been some interesting antithetical developments in the recently published literature. Before we delve into the details of this perplexity, we briefly describe some of the important research in the field of TSF in a chronological manner. Some of the early successes in the TSF field have utilized classical statistics and mathematics-based approaches involving moving average filters, exponential smoothing, and AutoRegressive Integrated Moving Average (ARIMA) processing. By taking into seasonality patterns, techniques such as SARIMA [ Box et al.(2015)Box, Jenkins, Reinsel, and Ljung ] and TBATs [ De Livera et al.(2011)De Livera, Hyndman, and Snyder ] provide enhanced prediction as compared to ARIMA. In the last decade, classical machine learning models such as Linear Regression [ Ristanoski et al.(2013)Ristanoski, Liu, and Bailey ] , XGBoost [ Chen and Guestrin(2016) ] , Random Forests and ensemble methods [ Masini et al.(2023)Masini, Medeiros, and Mendes ] have been explored for the TSF field. These techniques require the data to be transformed into a supervised learning problem by using a sliding window approach. Such models improve performance as they cancel out uncorrelated errors in the averaging process. For larger training data, machine learning approaches usually outperform the classical mathematical techniques of SARIMA and TBATs [ De Livera et al.(2011)De Livera, Hyndman, and Snyder ] . The highly successful Convolutional Neural Networks (CNNs) in computer vision were also attempted for TSF. One such example is the Long and Short-term Time-series network (LSTNet) proposed in [ Lai et al.(2018)Lai, Chang, Yang, and Liu ] which used both the CNN and RNN to extract both short and long-term dependency patterns for improving the TSF. Most of the recent research in the TSF domain uses the Transformer architecture that was originally proposed for Natural Language Processing (NLP) [ Vaswani(2017) ] . The Transformer uses the attention mechanism to determine the pair-wise similarity in the input sequence to predict the output. It has revolutionized the AI field, leading to development of LLMs such as ChatGPT [ Achiam et al.(2023)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, et al. ] , Llama [ Touvron et al.(2023)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar, et al. , Dubey et al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, et al. ] , Gemini [ Team et al.(2023)Team, Anil, Borgeaud, Wu, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth, et al. ] , DeepSeek [ Guo et al.(2025)Guo, Yang, Zhang, Song, Zhang, Xu, Zhu, Ma, Wang, Bi, et al. ] , among others. Thus, it is natural that these developments be applied to create more effective TSF models. 2 Preliminaries In a TSF problem, we are given historical time series data x , where \textbf{x}\in\mathbb{R}^{L\times m} with m being the number of multivariate time series features and L the number of look-back time steps. The goal is then to train a model that learns to predict the output y , where \textbf{y}\in\mathbb{R}^{T\times m} is the prediction for the future T time steps. When using a Transformer-based model for TSF, the input data is often divided into a sequence of n patches, i.e., \mathbf{x_{1},x_{2},….,x_{n}} , where \mathbf{x_{i}}\in\mathbb{R}^{p\times m} with p=\lfloor\frac{L}{n}\rfloor . These patches are converted to embedding vectors of size d\times 1 by a linear transformation according to a fixed dimension d for the transformer model. Position vectors of size d\times 1 are added to each embedding vector to maintain relative temporal information. In the Transformer architecture, three parallel learnt representations of the above transformed data are computed producing \mathbf{Q} , \mathbf{K} and \mathbf{V} matrices, with each \in\mathbb{R}^{n\times d} . The Transformer then computes a simple similarity in the form of an inner product on the learnt position encoded embeddings of the sequence of n input patches. The pairwise similarity between patches computed as, \mathbf{A}=\text{softmax}(\mathbf{QK}^{T}) is referred to as the “attention”. If there are n patches being input, referred to as the context, then \mathbf{A}\in\mathbb{R}^{n\times n} . Each layer in the Transformer divides the attention calculation into parallel heads by dividing the data along the embedding dimension. The output from a Transformer layer has the same dimensionality as the input, and is obtained by a matrix computation of (\mathbf{A}\times\mathbf{V})\in\mathbb{R}^{n\times d} where \mathbf{V}\in\mathbb{R}^{n\times d} contains rows of learnt position encoded representations of the input patches. For a Transformer-based TSF implementation, the output is often produced in one step rather than an autoregressive generation of one patch size at a time. 3 Related Work Some of the important works that use the Transformer architecture for TSF include: Informer [ Zhou et al.(2021)Zhou, Zhang, Peng, Zhang, Li, Xiong, and Zhang ] , Autoformer [ Wu et al.(2021)Wu, Xu, Wang, and Long ] , Fedformer [ Zhou et al.(2022)Zhou, Ma, Wen, Wang, Sun, and Jin ] , Pyraformer [ Liu et al.(2022a)Liu, Yu, Liao, Li, Lin, Liu, and Dustdar ] , PatchTST [ Nie et al.(2023)Nie, Nguyen, Sinthong, and Kalagnanam ] and iTransformer [ Liu et al.()Liu, Hu, Zhang, Wu, Wang, Ma, and Long ] . To better adapt the Transformer for TSF, some pre or post processing features have been added to it, e.g., an auto-correlation mechanism is used