MoHETS : Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts

Evandro S. Ortigossa, Guy Lutsker, Eran Segal · 2026

Read MoHETS : Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts by Evandro S. Ortigossa, Guy Lutsker, Eran Segal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21866.txt.

Introduction

Evandro Ortigossa, Guy Lutsker, and Eran Segal work on forecasting long stretches of multivariate time series. Real series mix global trends, local cycles, and regimes that shift without warning, which makes long-horizon prediction hard. Mixture-of-experts models help by letting specialist sub-networks handle different patterns.

Most such models, though, build every expert the same way, from identical simple blocks. Those uniform experts capture diverse time patterns poorly. The team's 2026 preprint, MoHETS, uses a mixture of heterogeneous experts, specialists with different designs suited to different dynamics. Better long-range forecasts feed planning in energy, logistics, and finance, where decisions must be made weeks ahead on the strength of today's data.

Catalog record

CREATOREvandro S. Ortigossa, Guy Lutsker, Eran Segal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21866

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoHETS : Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts Evandro S. Ortigossa, Guy Lutsker, Eran Segal Abstract Real-world multivariate time series can exhibit intricate multi-scale structures, including global trends, local periodicities, and non-stationary regimes, which makes long-horizon forecasting challenging. Although sparse Mixture-of-Experts (MoE) approaches improve scalability and specialization, they typically rely on homogeneous MLP experts that poorly capture the diverse temporal dynamics of time series data. We address these limitations with MoHETS, an encoder-only Transformer that integrates sparse Mixture-of-Heterogeneous-Experts (MoHE) layers. MoHE routes temporal patches to a small subset of expert networks, combining a shared depthwise-convolution expert for sequence-level continuity with routed Fourier-based experts for patch-level periodic structures. MoHETS further improves robustness to non-stationary dynamics by incorporating exogenous information via cross-attention over covariate patch embeddings. Finally, we replace parameter-heavy linear projection heads with a lightweight convolutional patch decoder, improving parameter efficiency, reducing training instability, and allowing a single model to generalize across arbitrary forecast horizons. We validate across seven multivariate benchmarks and multiple horizons, with MoHETS consistently achieving state-of-the-art performance, reducing the average MSE by 12 % 12\% compared to strong recent baselines, demonstrating effective heterogeneous specialization for long-term forecasting. 1 Introduction Time series forecasting is a critical task for decision-making in a wide variety of domains such as energy management ( 29 ) , financial planning ( 43 ) , healthcare ( 41 ) , and climate analysis ( 69 ) . However, accurately predicting future values from historical observations is challenging, as real‐world time series data often present complex temporal dependencies, seasonality, trends, non-stationarity, and exogenous influences that lead to varied distributions even within short context windows ( 58 ; 33 ) . Traditional statistical methods, including ARIMA, exponential smoothing, and vector autoregression ( 45 ) , often struggle to capture nonlinear patterns, multiple seasonalities, or high-dimensional multivariate interactions. Furthermore, these challenges intensify in long-term time series forecasting, where predicting intricate cross-variate dependencies over extended contexts demands models that scale efficiently while maintaining predictive accuracy ( 34 ) . Deep learning approaches have demonstrated remarkable performance in time series forecasting by enabling robust modeling of nonlinear and multivariate patterns. Initially designed for natural language processing (NLP), the Transformer architecture ( 56 ) has been successfully extended to computer vision (CV) ( 14 ) , audio ( 20 ) , and time series ( 71 ; 60 ) . Transformer models introduce attention mechanisms that adaptively weight historical information to capture long-range dependencies ( 71 ) . However, a fundamental misalignment persists in the context of time series: standard Transformers, designed for the discrete semantics of NLP, apply homogeneous processing, typically using dense MLPs, to all tokens. Time series data, conversely, are composed of distinct structural components, persistent periodicities, and transient trends that require fundamentally different inductive biases. Applying a uniform architecture to disentangle these heterogeneous patterns often results in inefficient parameter usage and suboptimal fittings ( 13 ) . Recent improvements have pushed Transformers to mitigate these issues. Patching techniques reduce complexity ( 44 ) , integration of exogenous variables allows learning contextual correlations ( 37 ; 58 ) , and sparse designs enable more efficient scaling ( 53 ; 33 ) . Although recent sparse approaches have introduced Mixture-of-Experts (MoE) to reduce computational overhead, they essentially inherit the homogeneous expert design of large language models (LLMs), where every expert is an identical MLP ( 53 ; 33 ) . This design ignores the multi-scale nature of time series, where capturing high-frequency local variations requires different operators than modeling long-term global dependencies. To address these challenges, we designed MoHETS , a novel encoder-only Transformer model that integrates sparse Mixture-of-Heterogeneous-Experts ( \operatorname{MoHE} ) with patch-based embedding and covariate integration, improving its robustness to diverse temporal dependencies and patterns. We demonstrate that MoHETS consistently outperforms well-known state-of-the-art models in forecasting benchmark experiments. In summary, the main contributions of this research are as follows: • We introduce MoHETS , an encoder-only Transformer with a Mixture-of-Heterogeneous-Experts ( \operatorname{MoHE} ) strategy that applies architecturally distinct experts to effectively model time patterns at different levels, ensuring the model architecture aligns with the intrinsic decomposition of time-series data. • We incorporate a multimodal cross-attention module that integrates external information from exogenous covariates. With this design, MoHETS enhances time series representations by capturing interactions between endogenous features and exogenous information. • We propose the \operatorname{MoHE} layer, which combines depthwise convolutions and Fourier-based experts to capture global trends and local periodicities at the patch level, respectively, enhancing specialization while maintaining the scaling benefits of standard \operatorname{MoEs} . 2 Related Work 2.1 Deep Learning for Time Series Forecasting Deep learning models have significantly advanced time series forecasting, transitioning from MLP-based networks ( 57 ) , recurrent neural networks (RNNs) ( 49 ) , and convolutional neural networks (CNNs) ( 50 ) to Transformer-based architectures ( 60 ) . The attention mechanism of Transformers is able to adaptively weight historical information, making it a natural choice for handling long-term dependencies. Early Transformer models, such as Informer ( 71 ) , introduced the ProbSparse self-attention to address the quadratic complexity of standard Transformers, while Autoformer ( 64 ) leveraged auto-correlation to discover period-based dependencies at the subseries level. Recently, PatchTST ( 44 ) further improved efficiency by applying channel-independent processing and segmenting time series into subseries-level patches, reducing computational overhead while preserving local semantics and allowing the model to attend longer context windows. iTransformer ( 34 ) inverts the input dimensions to apply attention across variates rather than time, prioritizing multivariate correlations over temporal dependencies. Foundation models, such as TimeGPT ( 17 ) and TimesFM ( 12 ) , explore pre-training paradigms to improve adaptability. However, these architectures are predominantly dense and homogeneous. By processing diverse temporal dynamics, such as high-frequency noise and low-frequency trends, through identical operators (e.g., standard MLPs), they suffer from parameter redundancy and struggle to decouple entangled temporal patterns. 2.2 Forecasting with Covariates Real-world time series are often partially observed, and endogenous variables (the primary series to forecast) are frequently influenced by exogenous covariates, which capture external contexts that can affect temporal dynamics and consequently predictions ( 58 ) . Covariates encompass external contexts such as calendar events, weather metrics, or economic indicators that drive the non-stationary dynamics of the target series. Transformer-based models have increasingly introduced covariates to improve contextual understanding. The Temporal Fusion Transformer (TFT) ( 31 ) employed variable selection networks and entity embeddings to dynamically weigh covariates, while Timer-XL ( 35 ) supports covariate-informed forecasting in a decoder-only patched architecture. Incorporating such data requires careful handling to address issues such as missing values or temporal misalignment. In this context, TimeXer ( 58 ) refined covariate incorporation by implementing a patch-wise self-attention module for endogenous series and a variate-wise cross-attention module for exogenous inputs, thus mitigating issues related to partial observability and temporal misalignment. Exogenous covariates can provide valuable external information to enhance robustness to non-stationarity and improve forecasting accuracy. However, current methods often append covariates as auxiliary tokens or simplified concatenations ( 31 ; 11 ) , failing to explicitly model the cross-modal interaction between static external contexts and dynamic time patches. 2.3 Sparse Mixture-of-Experts (MoE) Deep learning models are dense, imposing high memory and computational costs during training and inference ( 53

Read the complete text