A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting
Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, Charlotte Debus · 2024
Read A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting by Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, Charlotte Debus (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12883.txt.
Introduction
Nicholas Kiefer and five co-authors benchmark pruning methods for transformer forecasters in this 2024 paper. Transformers lead time series forecasting but carry heavy parameter counts. That bulk blocks use on low power devices. Pruning cuts weights away; the team tested both loose, unstructured cuts and block shaped, structured cuts across several leading models. They tracked forecast accuracy, model size, operation counts, and run speed.
Some models tolerated deep cuts and even beat their full versions after pruning. Fine tuning after the cut was required to recover quality. Structured pruning disappointed on speed: even with hardware and software support, it saved little time. The authors say the field should not assume a smaller model is a faster one. Their benchmark gives deployers real numbers instead of hopes. For forecasting on tight hardware, unstructured pruning plus fine tuning is the route their data support.
Catalog record
| CREATOR | Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, Charlotte Debus |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, Charlotte Debus Abstract . The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings. CCS: Computing methodologies Neural networks CCS: Computing methodologies Supervised learning by regression CCS: Mathematics of computing Multivariate statistics 1. Introduction Time series analysis and forecasting is a central task in many commercial and scientific applications, such as meteorology, particle physics, life sciences, and energy research. Examples include the forecasting of electricity generation and consumption ( Torres et al., 2017 ; Weyrauch et al., 2024 ) , weather phenomena ( Nguyen et al., 2023 ) or technical components in particle detectors ( Debus et al., 2024 ) . While deep learning techniques have an established position next to classical statistical methods for quite some time, the introduction of the Transformer architecture ( Vaswani et al., 2023 ) for natural language processing (NLP) and its adaptation to time series data have truly revolutionized the field ( Zhou et al., 2021 ; Zhou et al., 2022 ; Wu et al., 2022 ; Liu et al., 2024a ) . Transformers are, however, known to be computationally demanding, both in memory and training time, due to their large parameter count, the quadratic complexity with respect to the sequence length, and their heavy use of dense layers with large hidden dimensions ( Fournier et al., 2023 ) . Indeed, experiments on scaling laws have demonstrated that linear increases in predictive capabilities of Transformer models require exponential increases in trainable weights ( Kaplan et al., 2020a ) , which results in a growing need for computational resources. Similarly, Transformer-based models for time series forecasting problems are growing with respect to their parameter counts, as illustrated in Table 1 . Additionally, these models quickly suffer from an overfitting problem, due to an imbalance between parameter count and training dataset size ( Zeng et al., 2023 ; Jin et al., 2024 ) . Table 1 . Recent proposed Transformer-based models for time series forecasting task, their parameter count and publication year. Model Characteristic Year Params (M) Transformer Language translation 2017 10 Informer Probabilistic attention 2020 11 Autoformer Sequence decomposition 2020 10 FEDformer Fourier transform 2022 16 Time-LLM LLM capabilities 2024 3405 This high demand in memory and computation time of Transformers stands in sharp contrast to their prospective deployment on low-resource devices, as required in real-world applications ( Chen et al., 2024a ) . Meanwhile pruning, i.e., removal of neural network weights estimated to be unnecessary for the predictive performance, has become an established method to reduce computational demand, in particular for model deployment ( Mao et al., 2021 ) . Pruning is based on the observation that the majority of weights in a neural network are close to zero after training, and thus, do not contribute to the output ( LeCun et al., 1989 ) . Consequently, these weights can be removed from the neural network weight matrices, thereby reducing the number of numerical operations performed in every pass of the model. The corresponding networks are termed sparse . Since the inception of sparse neural networks, research interest in network pruning has spiked, and numerous methods and algorithms have been proposed in the last five years ( Hoefler et al., 2021 ) . While the natural occurrence of sparsity and the phenomenon of an a-priori better sparse networks, so-called lottery tickets ( Frankle and Carbin, 2019 ) , has only been observed in selected models, it is generally assumed that these findings translate across network architectures and learning tasks ( Liu et al., 2024b ) . Pruning of Transformer-based models for time series forecasting is a promising approach to reduce computational resource demand in real-world application, but has not been studied to date to the best of the authors’ knowledge. In this work, we aim to close this gap by providing a corresponding benchmark study. We empirically study the behavior of previously proposed time series models under the effect of pruning with respect to dataset and model size. In summary, our contributions are: (1) We train and prune five state-of-the art Transformer-based models for time series forecasting on a number of datasets, using both unstructured and structured pruning. We investigate different data set sizes, and apply models both in their originally published version and with adapted smaller parameter count, to rule out the effect of overfitting. (2) We then evaluate model performance with respect to the loss on the test dataset, the number of non-zero parameters, the parameter density, and the number of floating point operations (FLOPs). We further investigate, how fine-tuning after pruning can aid in recovering predictive performance lost during the pruning phase. Our findings provide insight into the benefits and drawbacks of pruning Transformer-based models for time series forecasting in terms of predictive performance and resource requirements. 2. Related Work Transformer-based Time Series Forecasting Multivariate time series (MTS) forecasting is the task of forecasting the next T_{pred} time steps of a given times series x with length T_{inp} , where each time step has a real-valued feature vectors with dimension D . These D univariate series are assumed to be dependent and relevant to the task. Models for MTS forecasting have to tackle multiple problems: dependencies and patterns can emerge inside a single dimensional long time series, and so the models have to be adequate to handle long sequences without forgetting; events may be correlated across time and across features. Deep learning-based MTS forecasting has received increasing attention since the introduction of the Transformer ( Vaswani et al., 2023 ) . Originally introduced for NLP, the ability to process and forecast sequential data makes it a natural fit for temporally resolved data. However, while the capability of the self-attention-mechanism to relate individual sequence elements to one another provides impressive predictive skills, the \mathcal{O}(L_{inp}L_{out}) complexity in sequence length L poses computational challenges, especially for longer time series. Hence, researchers have dedicated special attention to reducing this computational complexity with respect to memory and computational load ( Fournier et al., 2023 ) . One of the first Transformer-based models specifically designed for time series forecasting was the Informer model ( Zhou et al., 2021 ) . It introduced the ProbSparse attention mechanism, which attends to the most important of a random subset of past time