Enhancing Masked Time-Series Modeling via Dropping Patches
Tianyu Qiu, Yi Xie, Yun Xiong, Hao Niu, Xiaofeng Gao · 2024
Read Enhancing Masked Time-Series Modeling via Dropping Patches by Tianyu Qiu, Yi Xie, Yun Xiong, Hao Niu, Xiaofeng Gao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15315.txt.
Introduction
Tianyu Qiu and co-authors present a 2024 computing paper on teaching models to read time series more efficiently. Masked modeling hides chunks of a series and trains the model to fill them in, a standard pretraining trick. The authors ask what happens if whole patches are dropped instead of only masked.
Their method, DropPatch, randomly removes patches of the series during pretraining. Training speeds up sharply, since the model processes far less data each round. Gains also appear in same-domain tests, new domains, few-shot learning, and cold starts. Experiments link the gains to stronger attention, less redundant information, and a data-augmentation effect. A proof shows dropping patches slows the collapse of the model's internal features toward a single line, keeping them richer.
Catalog record
| CREATOR | Tianyu Qiu, Yi Xie, Yun Xiong, Hao Niu, Xiaofeng Gao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15315
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Masked Time-Series Modeling via Dropping Patches Tianyu Qiu, Yi Xie, Yun Xiong, Hao Niu, Xiaofeng Gao Abstract This paper explores how to enhance existing masked time-series modeling by randomly dropping sub-sequence level patches of time series. On this basis, a simple yet effective method named DropPatch is proposed, which has two remarkable advantages: 1) It improves the pre-training efficiency by a square-level advantage; 2) It provides additional advantages for modeling in scenarios such as in-domain, cross-domain, few-shot learning and cold start. This paper conducts comprehensive experiments to verify the effectiveness of the method and analyze its internal mechanism. Empirically, DropPatch strengthens the attention mechanism, reduces information redundancy and serves as an efficient means of data augmentation. Theoretically, it is proved that DropPatch slows down the rate at which the Transformer representations collapse into the rank-1 linear subspace by randomly dropping patches, thus optimizing the quality of the learned representations. 1 Shanghai Key Lab of Data Science, School of Computer Science, Fudan University, Shanghai, China {tyqiu22, yixie18, yunx, hniu18}@fudan.edu.cn 2 MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China [email protected] Introduction In recent years, masked modeling has emerged as a prevalent self-supervised method in various fields, including natural language processing ( Devlin et al. 2018 ; Liu et al. 2019 ) and computer vision ( Baevski et al. 2022 ; He et al. 2022 ; Bao et al. 2021 ) . This technique improves representation learning by reconstructing masked content based on unmasked parts. Masked modeling has also been adapted for time-series analysis. A notable advancement involves segmenting time-series into patches (sub-sequence) and applying a patch-level masking strategy, which has received considerable attention since its inception ( Nie et al. 2022 ) . This method not only shows promising performance in transfer learning, but also significantly enhances supervised forecasting by employing self-supervised pre-training to initialize model parameters, consistent with recent findings ( Amos, Berant, and Gupta 2023 ) . Building upon the patching technique, numerous time-series foundation model works have emerged and achieve significant performance in time-series forecasting ( Goswami et al. 2024 ; Woo et al. 2024 ) . Despite its potential, we observed that masked time-series modeling, represented by PatchTST ( Nie et al. 2022 ) , faces a dilemma. A relatively low mask ratio reduces effectiveness in learning useful features ( He et al. 2022 ; Zhang, Wang, and Wang 2022 ) . Given the characteristic of periodicity and repetitive pattern of time-series data, the masked patch can be recovered with little high-level understanding of the underlying patterns, leading to superficial learning and over-fitting as shown in Figure 1 (A). A natural idea is to increase the mask ratio, but another issue emerges: the presence of an excessive number of masked patches can further dilute the attention mechanism’s capacity to concentrate on the relevant and informative parts of data, termed as scattered attention as shown in Figure 1 (C). It can lead to the degradation of downstream task performance as the representations gradually lose their distinctiveness ( Noci et al. 2022 ; Dong, Cordonnier, and Loukas 2021 ; Zhai et al. 2023 ) . We introduce a simple yet effective strategy, DropPatch , to encourage learning useful features and improve the overall performance. Building on foundational time-series pre-training techniques ( Nie et al. 2022 ) , DropPatch randomly removes a predefined proportion of patches. The remaining patches are subsequently processed for masking and reconstruction. It is crucial to distinguish between dropping and masking in the context of pre-training. For a given time-series sample, the dropping operation is applied prior to masking and reconstruction. Removed patches are entirely excluded from all training processes during the current epoch. In contrast, masked patches, represented as zero tensors overlaid with positional encoding, remain part of the training process throughout the epoch. In our empirical study, DropPatch demonstrates clear advantages in mitigating over-fitting (Figure 1 (B)), enhancing attention focus (Figure 1 (C)), and improving forecasting performance (Figure 1 (D)). The reduction in the number of patches due to the dropping operation leads to significant improvements in computational efficiency and reduced memory consumption. Figure 1: (A) The loss curve of PatchTST with lower mask ratio 0.4 (official implementation); (B) The loss curve of DropPatch (unless otherwise stated, the drop ratio and mask ratio is 0.6 and 0.4 throughout this paper); (C) The Kullback-Leibler (KL) divergence between the attention coefficients of the final encoder layer and a uniform distribution, where each dot represents an individual attention head. A larger KL divergence indicates that this set of attention distributions is farther from a uniform distribution and thus more focused. PatchTST(0.78) refers to the PatchTST configured with a mask ratio of 0.78, matching the number of visible patches in DropPatch. (D) Comparison of MSE metrics between PatchTST and DropPatch with forecasting steps T\in\{96,720\} on ETTm1. Extensive experiments validate the effectiveness of DropPatch. Through detailed experimental analysis, we uncover the underlying mechanisms driving these improvements. The DropPatch strategy enhances the attention mechanism by enabling a sharper focus on multi-scale and diverse information. It strengthens the model’s ability to capture critical patterns while reducing redundancy in representation. Furthermore, our theoretical findings indicate that the random dropping of patches effectively slows the convergence of the Transformer’s representations toward a rank-1 linear subspace, thereby promoting the feature diversity. Overall, our contributions can be summarized as follows: • We introduce DropPatch, a simple yet effective strategy that enhances masked time-series modeling. • Extensive experiments demonstrate that the DropPatch strategy improves pre-training efficiency and delivers substantial performance gains across diverse downstream tasks. Additionally, we compile comprehensive synthesized datasets to evaluate its role as a core component in foundational models for time-series analysis. • Through rigorous empirical and theoretical analysis, we validate the effectiveness of DropPatch and provide insights into the mechanisms driving these improvements. Method In this section, we describe the details of our proposed pre-training method, DropPatch, as shown in Fig. 2 . We denote that DropPatch is an effective pre-training strategy, the model does not perform the dropping operation during the fine-tuning stage. Figure 2: The overall pre-training framework of DropPatch. Patching and Channel-Independence For each sample of multivariate time-series \mathbf{X}\in{\mathbb{R}^{L\times{c}}} , where L represents the length of time-series, and c denotes the number of channels (variates). We first split the entire time-series sample into non-overlapping subseries-level patches, which are served as input tokens to Transformer, like PatchTST ( Nie et al. 2022 ) . We permute the original data of time-series into \mathcal{X}\in{\mathbb{R}^{c\times{P\times{L_{P}}}}} , where L_{P} denotes the length of each subseries-level patch, and P denotes the total number of patches. Dropping Patches After the patching operation, we will first conduct the positional encoding for these patches. The positional encoding process is designed to preserve the positional information during the self-attention computation and following the dropping operation. It should be noted that the positional encoding of each token is computed prior to dropping operation, ensuring that the original sequence position of each token is maintained after the removal. We randomly drop patches in the patched time-series, which is the core idea of our proposed DropPatch. Let r denotes the ratio of dropping with condition 0\leq r\leq 1 , implying that only (1-r)P patches remains for further