Learning Novel Transformer Architecture for Time-series Forecasting

Juyuan Zhang, Wei Zhu, Jiechao Gao · 2025

Read Learning Novel Transformer Architecture for Time-series Forecasting by Juyuan Zhang, Wei Zhu, Jiechao Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13721.txt.

Introduction

Juyuan Zhang, Wei Zhu, and Jiechao Gao let a search process design a fresh Transformer shape for time series forecasts, in work from twenty twenty-five. Transformers forecast well, but their stock shape may not suit time data best. Past work rarely searched widely for better shapes for this task. Their frame, AutoFormer-TS, searches attention, activation, and encoding choices together. A new differentiable search method sharpens how the best parts are picked.

The search improves on older methods at spotting top operations. Found shapes beat state-of-the-art baselines across many forecast benchmarks. Accuracy gains hold while training cost stays within reason. The result questions the habit of copying text shapes for time tasks. Tailored shapes, found by search, can do better on real series. Forecasters gain a recipe for task-fit model design.

Catalog record

CREATORJuyuan Zhang, Wei Zhu, Jiechao Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13721

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Novel Transformer Architecture for Time-series Forecasting Juyuan Zhang, Wei Zhu, Jiechao Gao Abstract . Despite the success of Transformer-based models in the time-series prediction (TSP) tasks, the existing Transformer architecture still face limitations and the literature lacks comprehensive explorations into alternative architectures. To address these challenges, we propose AutoFormer-TS, a novel framework that leverages a comprehensive search space for Transformer architectures tailored to TSP tasks. Our framework introduces a differentiable neural architecture search (DNAS) method, AB-DARTS, which improves upon existing DNAS approaches by enhancing the identification of optimal operations within the architecture. AutoFormer-TS systematically explores alternative attention mechanisms, activation functions, and encoding operations, moving beyond the traditional Transformer design. Extensive experiments demonstrate that AutoFormer-TS consistently outperforms state-of-the-art baselines across various TSP benchmarks, achieving superior forecasting accuracy while maintaining reasonable training efficiency. 1 1 1 Code will be publicly available upon acceptance. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Machine learning algorithms CCS: Information systems Data mining CCS: Information systems Data analytics 1. Introduction Time series forecasting (TSP) represents a crucial modeling endeavor ( Jin et al., 2023b ) , spanning a wide array of practical applications such as climate modeling, inventory management, and energy demand prediction. Many efforts have been devoted in proposing different types of models to enhance the performance of TSP. In the 1970s, the dominant approaches are the statistical models like ARMA, ARIMA ( Box et al., 2015 ) , GARCH ( Bauwens et al., 2006 ) , and structural models ( Bollen, 1989 ) . With the raise of machine learning (ML) in the 1990s, many ML approaches have been applied to TSP, such as SVM ( Meyer and Wien, 2001 ) , decision tree, and ensemble models like Gradient boosting regression tree (GBRT) ( Drucker et al., 1994 ) . In the deep learning (DL) era, various deep learning-based time series models, including recurrent neural networks, convolutional neural networks, linear models and Transformer ( Wen et al., 2022 ) , are proposed in the literature. Among these models, Transformer ( Vaswani, 2017 ) now plays an important role in the recent works like iTransformer ( Liu et al., 2023 ) and PatchTST ( Nie et al., 2022 ) , achieving better performances across different benchmark tasks. With the widespread applications of Transformer in different domains and modalities, the research field has witnessed many sophisticated Transformer variants tailored for TSP ( Zhou et al., 2021 ; Kitaev et al., 2020 ; Wu et al., 2021 ; Zhou et al., 2022 ) . However, the existing methods have the following limitations. First, a branch of Transformer-based methods focused on manually altering the attention mechanisms ( Zhou et al., 2021 ) . However, these works focused on designing more efficient attention functions, reducing the overall complexity. However, as ( Nie et al., 2022 ) demonstrates, these models fail to perform well on the TSP tasks, and have been outperformed by simple architectures like multi-layer perceptrons (MLPs). Second, recent models like iTransformer ( Liu et al., 2023 ) and PatchTST ( Nie et al., 2022 ) have shown that with the proper tokenization methods, the Transformer model can achieve the state-of-the-art (SOTA) performance with the original architecture design. However, alternative architectures have not been fully investigated. To address the above issues, we propose the AutoFormer-TS framework (see Figure 1 ). First, we look into the architecture of the vanilla Transformer, and design a comprehensive search space for TSP Transformer models. Our search space contains alternative attention mechanisms, activation functions, and encoding operations that substitute the residual connections. As for the search algorithm, we employ the differentiable neural architecture search (DNAS) ( Liu et al., 2018a ) framework. We have looked into the workflow of DNAS and find that the current DNAS methods may have shortcomings in identifying which operations to keep. Thus, we propose a novel DNAS method, AB-DARTS. Extensive experimentation has proved that our AutoFormer-TS framework can successfully identify architectures that surpasses recent state-of-the-art baseline methods on the TSP task at hand. The contributions of our work are summarized as follows: • Our AutoFormer-TS framework constructs a comprehensive and compact search space, which identifies the alternative operations or functions that could benefit the Transformer architecture. • Our AutoFormer-TS framework proposes a novel DNAS method, AB-DARTS, which modifies the mechanism of identifying the most contributing operation on an edge of the hyper-network. • AutoFormer-TS consistently exceeds the state-of-the-art performance in TS forecasting tasks, while requiring reasonable training time cost. Figure 1. The architecture for our AutoFormer-TS framework . 2. Related works 2.1. Time series modeling As a classical research problem with widespread applications, models constructed from the statistical approaches for time series modeling have been used from the 1970s. The representative models are autoregressive integrated moving average (ARIMA) ( Box et al., 2015 ) , exponential smoothing ( Gardner Jr, 1985 ) , and structural models ( Bollen, 1989 ) . The most significant characteristic for these methods is that they require significant domain expertise to build. With the development of machine learning (ML) ( Biship, 2007 ) , many ML techniques are introduced to time series modeling to reduce manual efforts. Gradient boosting regression tree (GBRT) ( Drucker et al., 1994 ; Prokhorenkova et al., 2018 ) gains popularity by learning the temporal dynamics of time series in a data-driven manner. However, these methods still require manual feature engineering and model designs. With the powerful representation learning capability of deep learning (DL) from large-scale data, various deep learning-based time series models are proposed in the literature ( Lim and Zohren, 2021 ) , achieving better forecasting accuracy than traditional techniques in many cases. Before the era of Transformer ( Vaswani, 2017 ) , the two popular DL architectures are: (a) Recurrent neural networks (RNNs) based methods ( Hochreiter and Schmidhuber, 1997 ) , which summarize the past information compactly in internal memory states and recursively update themselves for forecasting. (b) Convolutional neural networks (CNNs) based methods ( Li et al., 2021 ) , wherein convolutional filters are used to capture local temporal features. More recently, multi-layer perceptron (MLP) based methods, like ( Tang et al., 2025 ) and ( Zeng et al., 2023 ) , have raised attention in the research field, since these models are simple and light-weight. 2.2. Transformer architectures in time series The progressive advancements in natural language processing and computer vision have led to the development of sophisticated Transformer ( Vaswani et al., 2017 ) variants tailored for a wide array of time series forecasting applications ( Zhou et al., 2021 ; Wu et al., 2021 ) . Central to these innovations is the methodology by which Transformers handle time series data. For instance, iTransformer ( Liu et al., 2023 ) treats each univariate time series as a distinct token, forming multivariate time series into sequences of such tokens. More recently, PatchTST ( Nie et al., 2022 ) adopts an assumption of channel independence, transforming a univariate time series into multiple patches, which are subsequently treated as tokens and processed through a Transformer encoder. Another important research direction is to design alternative Transformer architectures.

Read the complete text