FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain
Zhengnan Li, Haoxuan Li, Hao Wang, Jun Fang, Zhichao Chen, Yuting Tan, Xilong Cheng, Yunxiao Qin · 2024
Read FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain by Zhengnan Li, Haoxuan Li, Hao Wang, Jun Fang, Zhichao Chen, Yuting Tan, Xilong Cheng, Yunxiao Qin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01654.txt.
Introduction
Zhengnan Li and co-authors build a forecasting model that curbs overfit when channels in time series data interact in 2024. Simple neural layers are light and good at time patterns. They overfit badly when used across channels. A theory check shows extreme values in the data make the flaw worse. Their new layer pins weights inside a simplex shape.
That shape forces the model to learn simpler patterns. The full model pairs a channel module with a time module that works in the frequency domain. Theory shows a lower bound on model complexity than standard layers. Tests on seven data sets show gains in accuracy and speed. The layer also lifts other models that use channel wise layers.
Catalog record
| CREATOR | Zhengnan Li, Haoxuan Li, Hao Wang, Jun Fang, Zhichao Chen, Yuting Tan, Xilong Cheng, Yunxiao Qin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01654
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain Zhengnan Li, Haoxuan Li, Hao Wang, Jun Fang, Zhichao Chen, Yuting Tan, Xilong Cheng, Yunxiao Qin Abstract Time series forecasting (TSF) plays a crucial role in various domains, including web data analysis, energy consumption prediction, and weather forecasting. While Multi-Layer Perceptrons (MLPs) are lightweight and effective for capturing temporal dependencies, they are prone to overfitting when used to model inter-channel dependencies. In this paper, we investigate the overfitting problem in channel-wise MLPs using Rademacher complexity theory, revealing that extreme values in time series data exacerbate this issue. To mitigate this issue, we introduce a novel Simplex-MLP layer, where the weights are constrained within a standard simplex. This strategy encourages the model to learn simpler patterns and thereby reducing overfitting to extreme values. Based on the Simplex-MLP layer, we propose a novel F requency S implex MLP (FSMLP) framework for time series forecasting, comprising of two kinds of modules: S implex C hannel- W ise MLP (SCWM) and F requency T emporal M LP (FTM). The SCWM effectively leverages the Simplex-MLP to capture inter-channel dependencies, while the FTM is a simple yet efficient temporal MLP designed to extract temporal information from the data. Our theoretical analysis shows that the upper bound of the Rademacher Complexity for Simplex-MLP is lower than that for standard MLPs. Moreover, we validate our proposed method on seven benchmark datasets, demonstrating significant improvements in forecasting accuracy and efficiency, while also showcasing superior scalability. Additionally, we demonstrate that Simplex-MLP can improve other methods that use channel-wise MLP to achieve less overfitting and improved performance. Code are available here . I Introduction Time series forecasting (TSF) is crucial across various fields, including web data analysis [ 1 ] , electricity consumption [ 2 , 3 , 4 ] , and weather forecasting [ 5 , 6 , 7 , 8 ] . Accurate predictions from historical data are essential for decision-making, policy development, and strategic planning. Recent advances in deep learning have significantly enhanced TSF capabilities [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] . Fig. 1: Comparison of overfitting behavior among FSMLP, TimesNet, TSMixer, and Autoformer on the ETTh1 dataset, with both look-back window and prediction length set to 96. TSMixer, TimesNet, and Autoformer show a rapid decline in training loss, but their validation loss remains high, indicating overfitting. In contrast, FSMLP maintains lower validation loss, demonstrating that the Simplex MLP effectively mitigates the overfitting issues typically encountered by MLPs when modeling channel dependencies. Deep learning-based TSF methods can be categorized into channel-independent and channel-mix approaches. Channel-independent methods [ 17 , 18 ] focus on capturing temporal dependencies by using a shared model across different channels. These methods often exhibit robust performance. For example, PatchTST [ 19 ] segments time series into patches and employs attention mechanisms to capture relationships between these patches, thereby effectively modeling temporal dependencies. However, the absence of explicit inter-channel modeling can result in suboptimal performance when the relationships between channels contain valuable information [ 5 , 12 , 20 ] . Unlike channel-independent methods, channel-mix models excel at capturing inter-channel relationships by explicitly modeling the dependencies among channels. There exist various approaches to model these dependencies, among which one of the simplest and most intuitive methods is the use of MLPs (Multi-Layer Perceptrons). This approach has been employed in early Transformer models [ 21 , 1 , 10 , 22 , 23 ] , such as Fedformer [ 21 ] , Autoformer [ 1 ] , Nonstationary-Transformer [ 10 ] , and MLP-Mixer-like models such as TSMixer [ 24 ] . However, despite success in modelling temporal dependencies,these channel-wise MLP models often suffer from overfitting and display sub-optimal performance compared to channel-wise attention methods [ 5 ] . As shown in Fig. 1 , TimesNet, TSMixer, and Autoformer all exhibit signs of overfitting. In this paper, we leverage Rademacher complexity theory [ 25 ] to analyze this phenomenon and find that this overfitting issue may be attributed to the presence of extreme values in time series data , as shown in Table I . Rademacher complexity measures the ability of a function class to fit random noise, with lower Rademacher complexity indicating a reduced tendency to overfit. The Rademacher complexity \mathcal{R}_{S}(\mathcal{H}) for the hypothesis class \mathcal{H} of the MLP in linear regression is bounded as follows: \mathcal{R}_{S}(\mathcal{H})\leq\frac{B}{m}\sqrt{\sum_{i=1}^{m}\|x^{(i)}\|_{2}^{2}} where w\in\mathbb{R}^{d} are the weight parameters of the model, d is the input dimensionality, m is the number of training data points, x^{(i)}\in\mathbb{R}^{d} represents the i -th input data point, \|x^{(i)}\|_{2} is the \ell_{2} -norm of the i -th data point, and B is an upper bound on the norm of the weight vector w , typically assumed to be B=\|w\|_{2} and much greater than 1. The presence of extreme values in time series data can result in a large B when modeling channel dependencies with MLPs, thereby increasing the Rademacher complexity and making these models more prone to overfitting. TABLE I: In mainstream time series datasets, extreme values are observed, with the majority of values falling within \sigma , while a few outliers exceed 3\sigma . These extreme values significantly impact the performance of MLPs in capturing channel dependencies. ETTh1 ETTh2 ETTm1 ETTm2 Traffic Weather ECL \leq\sigma 87.47% 87.22 12.46% 87.30% 88.44% 86.69% 84.38% \geq 3\sigma 0.35% 0.99% 0.35% 0.99% 1.59% 0.8% 0.39% To address the overfitting oriented from the extreme values, we propose a novel operator: Simplex-MLP, inspired by the theory of the Standard n -simplex [ 26 ] . A standard n -simplex is defined as the set of points in \mathbb{R}^{n+1} that satisfy two conditions: the sum of the coordinates of each point equals 1 , and each coordinate is greater than or equal to zero. We apply the n -simplex theory to traditional MLP and constrain the weights of MLP to lie within the standard n -simplex. We call such a novel MLP as Simplex-MLP and the Rademacher complexity of it can be bounded by \mathcal{R}_{S}(\mathcal{H}_{\Delta})\leq\frac{1}{m}\sqrt{\sum_{i=1}^{m}\|x^{(i)}\|_{2}^{2}} , much smaller than that of traditional MLPs. This indicates that the Simplex-MLP reduces the influence of redundant noise among channels and thereby improving generalization and reducing overfitting. Based on Simplex-MLP, we introduce a novel channel-mix framework F requency S implex MLP (FSMLP) for time series forecasting. Specifically, FSMLP is composed of two kinds of modules: Simplex Channel-Wise MLP (SCWM) and Frequency Temporal MLP (FTM). SCWM can effectively extract information among channels, while FTM excels at extracting temporal information within each channel. Furthermore, our models capture both temporal and inter-channel dependencies in the frequency domain, enhancing the overall performance. Each frequency component represents a period in the time domain. Therefore, modeling inter-channel dependencies in the frequency domain involves modeling the dependencies between different periods across channels. This approach introduces less noise compared to directly modeling inter-channel dependencies in the time domain [ 27 , 14 , 28 , 29 ] . In summary, our contributions are as follows: • We utilized Rademacher complexity to analyze and identify that the use of MLPs for explicitly modeling inter-channel dependencies leads to overfitting, primarily due to the presence of extreme values in time series dataset. •