VarDrop : Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting

Junhyeok Kang, Yooju Shin, Jae-Gil Lee · 2025

Read VarDrop : Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting by Junhyeok Kang, Yooju Shin, Jae-Gil Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14183.txt.

Introduction

Junhyeok Kang, Yooju Shin, and Jae-Gil Lee published a computing paper in 2025 on training periodic time series forecasters faster by dropping redundant channels. Methods that tokenize each channel separately pay a cost that grows with the square of the channel count. Their strategy, VarDrop, leaves out repeated channel tokens during training while keeping the signal. Channels are grouped by their dominant rhythms through a hashing trick.

Only representative tokens from each rhythm group enter the attention step, chosen by stratified sampling. Costs fall sharply while accuracy holds. On public benchmarks, the method beat other efficiency-focused baselines. The authors pitch it as a simple upgrade for large multivariate forecasting jobs.

Catalog record

CREATORJunhyeok Kang, Yooju Shin, Jae-Gil Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14183

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VarDrop : Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting Junhyeok Kang, Yooju Shin, Jae-Gil Lee Abstract Variate tokenization, which independently embeds each variate as separate tokens, has achieved remarkable improvements in multivariate time series forecasting. However, employing self-attention with variate tokens incurs a quadratic computational cost with respect to the number of variates, thus limiting its training efficiency for large-scale applications. To address this issue, we propose VarDrop , a simple yet efficient strategy that reduces the token usage by omitting redundant variate tokens during training. VarDrop adaptively excludes redundant tokens within a given batch , thereby reducing the number of tokens used for dot-product attention while preserving essential information. Specifically, we introduce k k -dominant frequency hashing ( k k -DFH), which utilizes the ranked dominant frequencies in the frequency domain as a hash value to efficiently group variate tokens exhibiting similar periodic behaviors. Then, only representative tokens in each group are sampled through stratified sampling. By performing sparse attention with these selected tokens, the computational cost of scaled dot-product attention is significantly alleviated. Experiments conducted on public benchmark datasets demonstrate that VarDrop outperforms existing efficient baselines. 1 LG AI Research 2 KAIST [email protected], {yooju.shin, jaegil}@kaist.ac.kr 1 Introduction Transformers have demonstrated impressive performance in time series forecasting, primarily due to their attention mechanisms ( Trirat et al. 2024 ; Shin et al. 2024 ; Nie et al. 2023 ; Zhang and Yan 2023 ; Wu et al. 2021 ; Zhou et al. 2021 ) . Traditionally, most methods have employed temporal tokenization, treating all variates at a given timestamp as a single token. However, recent studies reveal that variate tokenization —where each variate is embedded as a separate token—outperforms temporal tokenization in capturing inter-variate dependencies thereby increasing forecasting accuracy ( Liu et al. 2024b ) . Due to its broad applicability to Transformers, variate tokenization has been adopted in recent advancements in multivariate time series forecasting ( Liu et al. 2024c ; Wang et al. 2024 ; Han et al. 2024 ) . Despite its advantages, the feasibility of variate tokenization in real-world applications is hindered by the increasing number of variates ( N ). Many public benchmark datasets demonstrate the high dimensionality often encountered in real-world applications. For instance, the Electricity dataset includes 321 variates, each corresponding to a customer, and the Traffic dataset comprises 862 variates, each representing a sensor ( Wu et al. 2021 ) . The large N introduces inefficiency, as the computational cost of attention mechanisms increases quadratically with N . This inefficiency results in a superfluous carbon footprint during the model training process. In addition, the large N can lead to overfitting and reduced attention performance by diluting important information ( Peysakhovich and Lerer 2023 ; Liu et al. 2024a ) . Therefore, reducing N while retaining the essential information is critical for the success of variate tokenization. (a) Electricity dataset. (b) Traffic dataset. Figure 1: Variate redundancy in public periodic time series datasets. Most variates are highly correlated with others. For more detailed analysis, please refer to Appendix A. To address this issue, we propose removing variate redundancy in multivariate time series. As different variates often share the same characteristics such as trends and seasonality at some timestamps, excessive correlations between them are frequently observed in periodic time series. Figure 1 shows the variate redundancy in the term of maximum Pearson correlation coefficient values (refer to Eq. (4)) between the variates in two datasets. For the Electricity and Traffic datasets, 79.4% and 94.7% of all variates exhibit strong correlations with other variates exceeding 0.9, respectively. By selecting a few informative and distinguishing variates, we can significantly reduce computational cost while preserving the original periodic information of the data. However, efficiently picking out the representative variates is challenging for two reasons. First, the correlation between variates fluctuates due to inherent distribution shifts in time series data. This correlation shift in sliding windows generates a unique set of correlations for each batch as shown in Appendix B. We should filter out redundant variates in an online manner due to this correlation shift. Second, computing similarity between variates requires extensive computational cost. Popular similarity metrics, such as the Pearson correlation coefficient and dynamic time warping (DTW), have computational complexity in the order of O(N^{2}T^{2}) where T denotes the window length. In summary, we need a simple selection process with a fast similarity metric to leverage variate redundancy effectively. To this end, we propose a simple yet efficient training strategy for variate tokens called VarDrop . Given a multivariate time series, VarDrop performs fast Fourier transform for each variate and identifies k -dominant frequencies whose amplitude values are top- k in frequency domain. By ranking k frequencies in the descending order of amplitude, k -dominant frequency hashing ( k -DFH) generates a meaningful hash for each variate. VarDrop groups the variates with the same hash and select representative variate tokens for each group through stratified random sampling. Scaled dot-product attention is then computed with selected tokens where each represents unique temporal patterns in each batch. As a result, VarDrop resolves variate redundancy and successfully reduces the number of tokens for each batch without incurring significant computational overhead. Here we summarized our key contributions as follows: • We propose a simple yet efficient training strategy, VarDrop , which significantly improves computational efficiency by disregarding redundant variates in variate-tokenized Transformers. • VarDrop leverages k -dominant frequency hashing that efficiently identifies similar tokens in each batch by applying the fast Fourier transform. • We demonstrate the effectiveness of VarDrop through extensive experiments conducted on four benchmark datasets, comparing to state-of-the-art methods. 2 Related Work 2.1 Tokenization Strategies for Time Series Data Inspired by the success of Transformers in natural language processing, numerous Transformers have been proposed for multivariate time series forecasting. Previous studies, such as Autoformer ( Wu et al. 2021 ) , Fedformer ( Zhou et al. 2022 ) , Crossformer ( Zhang and Yan 2023 ) , and PatchTST ( Nie et al. 2023 ) , adopted a temporal tokenization method similar to language models, treating the values of all variates at a specific timestamp as a single token. The multiple variates are not considered individually, but processed as a whole when generating representations for each temporal token. As a result, these embedded temporal tokens fail to properly capture the correlations between different variates in multivariate time series. Moreover, temporal tokens could not consider relevant contexts due to the narrow receptive field ( Shin et al. 2023 ) . To address the limitations in temporal tokenization, iTransformer introduced variate tokenization ( Liu et al. 2024b ) . It is a special type of tokenization as the various tokenization methods used in Flowformer ( Wu et al. 2022 ) and treats each variate as one token, aiming to better model the variate dependencies. After the success of iTransformer, variate tokenization became a prevalent technique in forecasting models. Timer merges multiple variates from different domain into a single time series and treats the time series as a single token ( Liu et al. 2024c ) . MCformer tokenizes each variate and mixes the variates to capture inter-variate correlations ( Han et al. 2024 )

Read the complete text