CASA: CNN Autoencoder-based Score Attention for Efficient Multivariate Long-term Time-series Forecasting Minhyuk Lee, HyeKyung Yoon, MyungJoo Kang Abstract Multivariate long-term time series forecasting is critical for applications such as weather prediction, and traffic analysis. In addition, the implementation of Transformer variants has improved prediction accuracy. Following these variants, different input data process approaches also enhanced the field, such as tokenization techniques including point-wise, channel-wise, and patch-wise tokenization. However, previous studies still have limitations in time complexity, computational resources, and cross-dimensional interactions. To address these limitations, we introduce a novel CNN Autoencoder-based Score Attention mechanism (CASA), which can be introduced in diverse Transformers model-agnosticically by reducing memory and leading to improvement in model performance. Experiments on eight real-world datasets validate that CASA decreases computational resources by up to 77.7%, accelerates inference by 44.0%, and achieves state-of-the-art performance, ranking first in 87.5% of evaluated metrics. Our code is available at https://github.com/lmh9507/CASA . † † footnotetext: ∗ Equal contribution. † † footnotetext: † Corresponding author. 1 Introduction Figure 1 : The validation loss of iTransformer, Transformer, PatchTST, and our model on the Traffic dataset is evaluated. Point-wise and patch-wise implemented models exhibit lower performance compared to channel-wise models. However, while the iTransformer model rapidly saturates, CASA demonstrates consistent learning and achieves the lowest loss value. Multivariate Long-Term Time Series Forecasting (LTSF) plays a pivotal role in real-world applications, including weather prediction, traffic flow analysis Ji et al. (2023) , and solar energy forecasting Lai et al. (2018) . LTSF has seen rapid advancements driven by the emergence of Transformer model Vaswani (2017) . Subsequent Transformer variants have further demonstrated the effectiveness of the multi-head self-attention mechanism in capturing temporal dependencies and cross-dimensional correlations. Zhou et al. (2021) ; Wu et al. (2021) ; Liu et al. (2022b) ; Zhou et al. (2022) ; Liu et al. (2022c) ; Zhang and Yan (2023) ; Nie et al. (2022) . Despite numerous efforts, Transformer-based models have not consistently outperformed CNN- or MLP-based architectures in the LTSF domain Wu et al. (2023) ; Ekambaram et al. (2023) ; Das et al. (2023) ; Zeng et al. (2023) . Notably, DLinear Zeng et al. (2023) , a model constructed with simple linear layers, raises critical questions about the effectiveness and necessity of Transformer-based architectures in this field, especially considering their demanding computational and time resources. To address the aforementioned challenges, diverse tokenization techniques have been introduced into the core architecture of Transformer-family models Liu et al. (2023) ; Nie et al. (2022) . We evaluate the effectiveness of three models, each employing a distinct tokenization technique, as illustrated in Figure 2 . Figure 1 demonstrates that channel-wise tokenization achieves the best performance among the three, as highlighted in Yu et al. (2024) . However, it still leads to rapid saturation during training. Moreover, computational cost and memory usage remain significant challenges. Our objective is to develop a more efficient model that delivers superior predictive performance while mitigating these drawbacks. We tackle these issues by refining the self-attention mechanism, the cornerstone of Transformer-based models. Figure 2 : (a) point-wise token (b) patch-wise token (c) channel-wise token In this paper, we propose CNN Autoencoder-based Score Attention (CASA) , a simple yet novel module that addresses the aforementioned gap by effectively capturing correlations and avoiding saturation, thereby facilitating consistent learning, designed to serve as an alternative to the conventional self-attention mechanism. We retain the vanilla Transformer encoder while substituting the attention mechanism with a CNN-based module. CASA approximates \frac{QK^{T}}{\sqrt{d_{k}}} rather than directly calculating it, as done in traditional multi-head self-attention. This design addresses a critical limitation of conventional methods by sufficiently accounting for significant correlation between variates in the calculation of attention scores (see Section 3.3 for details). Our key contributions are as follows: • We present a simple yet effective CNN Autoencoder-based Score Attention (CASA) module as an alternative to self-attention. It scales linearly with the number of variates, input length, and prediction length. Compared to Transformer-based variants, CASA reduces memory usage by up to 77.7% and improves computational speed by 44.0% , depending on the dataset. • CASA can be agnostically integrated into Transformer models, regardless of the tokenization technique used (e.g., point-wise, patch-wise, or channel-wise). To the best of our knowledge, this is the first module validated across individual tokenization techniques, successfully enhancing cross-dimensional information capture and improving prediction performance. • CASA achieved first place in 54 out of 64 metrics across 8 real-world datasets and ranked highest in 14 out of 16 average metrics , establishing itself as a highly competitive solution for multivariate LTSF. Figure 3 : (a) Conventional Self-Attention. (b) Overall architecture of our CASA block. The time-series data is embedded using channel-wise tokenization. The 1D CNN Autoencoder is then used to compute cross-dimensional information. The softmax output and the value are multiplied element-wise. Our CASA places a strong emphasis on capturing essential cross-dimensional information by calculating high-dimensional spatial relationships before compressing the channel information. 2 Related Works Transformer variants The vanilla Transformer model Vaswani (2017) , widely recognized for its success in natural language processing, has also achieved notable advancements in time-series forecasting. Diverse Transformer variants have been introduced to enhance forecasting performance, which can be broadly grouped into three approaches. The first approach modifies the traditional self-attention mechanism with alternatives by incorporating specialized modules, or pyramidal attention Liu et al. (2022b) , to reduce memory requirements while capturing multi-resolution representations. Additional modifications, including the trapezoidal architecture Zhou et al. (2021) and de-stationary attention Liu et al. (2022c) , aim to improve robustness and address issues like over-stationarization. The second approach leverages frequency-domain techniques, such as Fast Fourier Transform (FFT) Zhou et al. (2022) and auto-correlation mechanisms Wu et al. (2021) , to better extract temporal patterns. The third approach introduces hierarchical encoder-decoder frameworks Zhang and Yan (2023) with routing mechanisms to capture cross-dimensional information, although these methods sometimes encounter challenges such as slower learning and higher computational demands. Alternatives of Transformers While Transformer variants have significantly advanced the time-series forecasting domain, CNN-based models present promising alternatives. These approaches include methods that model segmented signal interactions Liu et al. (2022a) and those that reshape 1D time-series data into 2D tensors Wu et al. (2023) , enabling the capture of both inter-period and intra-period dynamics. Similarly, linear models Zeng et al. (2023) have demonstrated simplicity while achieving high prediction performance. However, these methods generally fall short of explicitly addressing cross-dimensional interactions, which are crucial for improving multivariate time-series forecasting. Other methods have been developed to modify aspects of the Transformer architecture, particularly focusing on tokenization techniques. For instance, PatchTST Nie et al. (2022) segments input data into patches to extract local information within individual variates, while iTransformer Liu et al. (2023) treats each variate as a token, enabling the self-attention mechanism to capture multivariate correlations. However, a common drawback of these methods is their reliance on self-attention, which demands substantial computational resources. Furthermore, their prediction performance remains suboptimal, with both models struggling to effectively capture multivariate dependencies, which critically impacts their predictive accuracy. In contrast, our proposed model, CASA, addresses these challenges by significantly reducing resource consumption while achieving superior prediction performance, offering an efficient and effective alternative to traditional self-attention-based approaches. 3 Method This section provides an overview of CASA, including the motivation behind its architecture, a detailed explanation of its structure, and a complexity analysis. Section 3.1, we formulate the problem we aim to solve and define the necessary notations. Section 3.2 provides an explanation of the overall architecture. Section 3.3 discusses the limitations of conventional Transformers in capturing cross-dimensional interactions. Section 3.4 introduces CNN Autoencoder-based Score Attention (CASA), an improved self-attention module designed to address these issues. We confirm theoretical efficiency of CASA through complexity analysis. 3.1 Problem Formulation In multivariate LTSF, given an input length L , the number of variates N , the number of layers M , and a prediction length H , denote the input and the output X\in\mathbb{R}^{N\times L} and Y\in\mathbb{R}^{N\times H} respectively. The hidden dimension is denoted as D , and the intermediate feature after embedding is represented as Z_{i}\in\mathbb{R}^{N\times D}\;(i\in\{0,1,\cdots M\}) . Since this is not a univariate time-series forecasting problem, the input is consistently represented as a matrix. CASA SOFTS iTransformer PatchTST Transformer Complexity O(NL+NH) O(NL+NH) O(N^{2}+NL+NH) O(NL^{2}+NH) O(NL+L^{2}+HL+NH) Memory (MB) 1684 1720 10360 21346 4772 Inference (s/iter) 0.05 0.042 0.162 0.441 0.087 Table 1: A complexity comparison conducted on the Traffic dataset among baseline models, including the vanilla Transformer, PatchTST, iTransformer, and SOFTS, with respect to window length L, number of channels N, and forecasting horizon H. Notably, the complexity of CASA scales linearly with N, L, and H. Detailed implementation information is provided in Appendix D. 3.2 Overall Architecture The overall framework is depicted in Figure 3 . Following prior works Liu et al. (2023) ; Yu et al. (2024) , we adopt a channel-wise tokenization approach, utilizing the vanilla Transformer encoder as the backbone. To address the challenges outlined in Section 3.3, we enhance the Transformer by replacing only the attention mechanism with a 1D CNN Autoencoder Score Attention (CASA) . Although this modification alters only a very small portion of the overall model, it effectively reduces computational cost and memory usage while improving prediction performance. The input X is linearly embedded to produce the intermediate feature Z_{0} . The final feature Z_{M} , obtained by passing Z_{0} through M CASA blocks, is then fed into the predictor where the output becomes Y . The pipeline is summarized by the following equation. \displaystyle Z_{0}=\mathrm{Embedding}(X) (1) \displaystyle Z_{i+1}=\mathrm{CASA\;block}(Z_{i}) (2) \displaystyle Y=\mathrm{Predictor}(Z_{M}) (3) 3.3 Limitation of Self-Attention We demonstrate that the existing self-attention mechanism does not sufficiently consider cross-dimension information when embedding queries and keys. the structure of the self-attention mechanism in the conventional Transformer is as follows ( f_{j} : affine map): \mathrm{Attention}(Z_{i+1})=\mathrm{softmax}\left(\frac{Q_{i+1}K_{i+1}^{T}}{\sqrt{d_{k}}}\right)*V_{i+1} (4) Q_{i+1}=f_{1}(Z_{i}),\;\,K_{i+1}=f_{2}(Z_{i}),\;\,V_{i+1}=f_{3}(Z_{i}) (5) At this point, since f_{j} is an affine map, queries and keys are computed through the following operations: \displaystyle Q_{i+1}=Z_{i,1}W_{i,1}+b_{i,1},\quad K_{i+1}=Z_{i,2}W_{i,2}+b_{i,2} (6) \mathrm{where}\;W_{i,j}\in\mathbb{R}^{D\times D}\;\mathrm{and}\;b_{i,j}\in\mathbb{R}^{N\times D} (7) Proposition 1 . Query and key embeddings are variate-independent operations in the conventional Transformer using channel-wise tokenization. Proof. See Appendix E.1. ∎ Proposition 2 . Query and key embeddings are time-independent operations in the conventional Transformer using point-wise and patch-wise tokenization. Proof. See Appendix E.2. ∎ Proposition 1 and Proposition 2 imply that each tokenization method does not consider the correlation between variates or time points when embedding the key and query. Since multivariate time series exhibit correlations both between variates and across time points, this reduces the potential of the Transformer architecture. Especially, based on the Proposition 1 , the Transformer using channel-wise tokenization does not directly incorporate cross-dimensional information when embedding the r -th variate into queries and keys. In other words, the self-attention mechanism embeds queries and keys through variate-independent feature refinement operations and then computes the attention map using \frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}} . In the LTSF domain, tokens (i.e., variates) exhibit inherent correlations (see Section 4.4 ), which reduce the effectiveness of variate-independent operations during feature refinement. This limitation can hinder performance in the multivariate LTSF domain, where capturing correlations between variates is important. 3.4 CNN Autoencoder-based Score Attention To address the issue posed by the self-attention mechanism’s variate-independent operation, we treat each variate as a channel and apply a convolution instead of using the affine map from the conventional self-attention mechanism. This approach ensures that the operation becomes variate-dependent. In more detail, instead of directly computing \frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}} , we designed a score network Score to approximate this operation using 1D CNN Autoencoder. The modified structure of self-attention is as follows ( f : affine map, \circledast : element-wise product): \displaystyle\mathrm{Attention}(Z_{i+1})=\mathrm{softmax}(\texttt{Score}(Z_{i}))\circledast V_{i+1} (8) V_{i+1}=f(Z_{i}) (9) By approximating \frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}} via a CNN architecture instead of direct computation, we reduce complexity compared to the affine map (see the paragraph below), addressing the limitations of self-attention. This facilitates the development of a linear complexity model with enhanced performance (Section 4.1). To explain the score network in more detail, we adopted an inverted bottleneck autoencoder structure, inspired by previous research Wilson et al. (2016) ; Bengio et al. (2013) , which demonstrated that embedding low-dimensional features into a high-dimensional latent space can improve expressiveness. In summary, we leverage CNN operations to incorporate information across all variates, embedding them into a high-dimensional feature space before compressing the channels to retain only essential cross-variable information. Consequently, despite its simple architecture, the proposed module outperforms existing self-attention mechanisms while maintaining efficiency, constituting a significant contribution. Dataset CASA (ours) SOFTS Han et al. , 2024 iTransformer Liu et al. , 2023 PatchTST Nie et al. , 2022 TSMixer Ekambaram et al. , 2023 Crossformer Zhang and Yan, 2023 TiDE Das et al. , 2023 DLinear Zeng et al. , 2023 FEDformer Zhou et al. , 2022 MSE( \mathbb{\downarrow} ) / MAE( \mathbb{\downarrow} ) MSE / MAE MSE / MAE MSE / MAE MSE / MAE MSE / MAE MSE / MAE MSE / MAE MSE / MAE ETTm1 0.386 / 0.393 0.393 / 0.403 0.407 / 0.410 0.396 / 0.406 0.398 / 0.407 0.513 / 0.496 0.419 / 0.419 0.474 / 0.453 0.543 / 0.490 ETTm2 0.276 / 0.319 0.287 / 0.330 0.288 / 0.332 0.287 / 0.330 0.289 / 0.333 0.757 / 0.610 0.358 / 0.404 0.350 / 0.401 0.305 / 0.349 ETTh1 0.438 / 0.434 0.449 / 0.442 0.454 / 0.447 0.453 / 0.446 0.463 / 0.452 0.529 / 0.522 0.541 / 0.507 0.456 / 0.452 0.440 / 0.460 ETTh2 0.374 / 0.397 0.373 / 0.400 0.383 / 0.407 0.385 / 0.410 0.401 / 0.417 0.942 / 0.684 0.611 / 0.550 0.559 / 0.515 0.437 / 0.449 ECL 0.168 / 0.259 0.174 / 0.264 0.178 / 0.270 0.189 / 0.276 0.186 / 0.287 0.244 / 0.334 0.251 / 0.344 0.212 / 0.300 0.214 / 0.327 Traffic 0.421 / 0.261 0.409 / 0.267 0.428 / 0.282 0.454 / 0.286 0.522 / 0.357 0.550 / 0.304 0.760 / 0.473 0.625 / 0.383 0.610 / 0.376 Weather 0.243 / 0.267 0.255 / 0.278 0.258 / 0.278 0.256 / 0.279 0.256 / 0.279 0.259 / 0.315 0.271 / 0.320 0.265 / 0.317 0.309 / 0.360 Solar 0.221 / 0.244 0.229 / 0.256 0.233 / 0.262 0.236 / 0.266 0.260 / 0.297 0.641 / 0.639 0.347 / 0.417 0.330 / 0.401 0.291 / 0.381 1^{st}/2^{nd} count 14 / 2 2 / 13 0 / 1 0 / 2 0 / 0 0 / 0 0 / 0 0 / 0 0 / 1 Table 2 : Multivariate forecasting results with horizon H\in\{96,192,336,720\} and fixed lookback window length L=96 . Red values represent the best performance, while underlined values represent the second-best performance. Results are averaged from all prediction horizons. Full results are listed in Table 6. (Appendix B) Figure 4 : Results of prediction of Ours and baseline models on Weather dataset. (a) CASA (b) iTransformer (c) SOFTS (d) PatchTST Complexity Analysis CASA is an efficient algorithm that exhibits linear complexity not only with respect to the number of tokens, i.e., the number of variates N , but also with respect to the input length L and prediction length H . The detailed complexity calculation is as follows. Let the kernel size of the score network be denoted as k . The complexities of the Reversible Instance Normalization (RevIN), series embedding, and MLP are O(NL) , O(NLD) , and O(ND^{2}) , respectively. Additionally, the complexity of the score network, composed of CNN autoencoder blocks, is O(NkD^{2}) . The predictor has a complexity of O(NDH) . Thus, the overall complexity of our method is O(NL+NLD+ND^{2}+NkD^{2}+NDH) , which scales linearly with respect to N , L , and H . Since the hidden dimension and kernel size are constants in the algorithm, they can be ignored. Consequently, N is dominated by NL and NH (Since L and H typically take on large values in LTSF), leading to the final complexity summarized in Table 1 . In addition, the results for memory usage and inference time are included in the table, empirically demonstrating the efficiency of CASA. For details of the implementation, refer to the Appendix D. Model Comp ECL Traffic Weather ETTm1 ETTm2 ETTh1 ETTh2 MSE( \mathbb{\downarrow} ) MAE( \mathbb{\downarrow} ) MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE Transformer Attention 0.203 0.292 0.655 0.359 0.245 0.296 0.407 0.417 0.369 0.398 0.482 0.465 0.522 0.481 CASA 0.201 0.287 0.645 0.354 0.242 0.293 0.390 0.411 0.368 0.388 0.465 0.461 0.512 0.490 PatchTST Attention 0.189 0.276 0.454 0.286 0.256 0.279 0.396 0.406 0.287 0.330 0.453 0.446 0.385 0.410 CASA 0.186 0.273 0.440 0.280 0.253 0.277 0.386 0.402 0.285 0.332 0.452 0.443 0.365 0.399 iTransformer Attention 0.178 0.270 0.428 0.282 0.258 0.278 0.407 0.410 0.288 0.332 0.454 0.447 0.383 0.407 CASA 0.168 0.259 0.421 0.261 0.244 0.267 0.386 0.393 0.276 0.319 0.438 0.434 0.374 0.397 Table 3: The performance of CASA across three distinct Transformer-based models, each employing different tokenization techniques. The standard self-attention module is replaced with our CASA. Among the 42 metrics assessed, CASA demonstrated improvements in 40 of them. 4 Experiments Dataset We conduct our comprehensive experiment on 8 benchmark datasets Zhou et al. (2021) , such as Traffic, ETT series including 4 subsets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Solar, Electricity. More detailed information on Dataset is described in Appendix A. Baselines We chose totally 8 comtemporarlly baseline models, including SOFTS Han et al. (2024) , iTransformer Liu et al. (2023) , PatchTST Nie et al. (2022) , TSMixer Ekambaram et al. (2023) , Crossformer Zhang and Yan (2023) , TiDE Das et al. (2023) , DLinear Zeng et al. (2023) , FEDformer Zhou et al. (2022) . Figure 5 : Experimental results on the ETTm1, Electricity, and Traffic datasets (with 7, 321, and 862 variates, respectively). Our CASA remains robust across varying input and prediction lengths (48 to 720). Unlike PatchTST, which struggles as the number of variates increases, models like iTransformer and SOFTS, which tokenize variates, exhibit stronger performance. Setup Our comprehensive experiments results are based on MSE (Mean Squared Error) and MAE (Mean Absolute Error) metrics. Our main experiments are conducted on the conditions with L=96 and the H\in\{96,192,336,720\} . 4.1 Multivariate Forecasting Results The main results are presented in Table 2 , where red-bold text indicates the best score and blue-underlined text represents the second-best score. CASA demonstrates the lowest MSE and MAE losses across 8 benchmark datasets, surpassing the previous state-of-the-art model, SOFTS, by a substantial margin. Additionally, the second-lowest performance scores exhibit a smaller gap from the best score compared to the others. Notably, the proposed model showcases its robustness on relatively large datasets, such as Traffic, Weather, and Solar, highlighting its ability to capture complex correlations, which significantly enhances the model’s predictive performance. Figure 6 : (a) Memory usage scaling with the number of tokens, demonstrating CASA’s linear growth and reduced memory consumption compared to SOFTS. (b, c) Experimental results on Traffic and Electricity datasets (batch size: 16, input/prediction length: 96), highlighting CASA’s low memory usage and balanced trade-off between speed and performance. Figure 4 visualizes the prediction performance of weather dataset from CASA, SOFTS, iTransformer, and PatchTST models against the ground truth. CASA shows the closest alignment with the label, while iTransformer shows similar tendency along the ground truth with large deviation. SOFTS and PatchTST prediction slightly detours the label. The full results of different prediction lengths and the visualization results on the rest of the datasets are demonstrated in Appendix B and Appendix C, respectively. 4.2 Superiority Analysis of CASA Replacing Self-Attention with CASA To ensure the proposed model’s adaptability to diverse tokenization techniques, we integrate CASA into various Transformer variants including the vanilla Transformer, PatchTST, and iTransformer. Especially for the vanilla Transformer, we only use the encoder architecture to appropriately compare the effectiveness of CASA. The results on seven benchmark datasets are presented in Table 3 . With the exception of two specific cases—ETTm2 compared to PatchTST and ETTh2 compared to the vanilla Transformer—CASA consistently enhances the performance of the original and variants models, achieving improvements in 40 out of 42 results across the benchmarks. These findings not only demonstrate that replacing self-attention with CASA significantly boosts forecasting accuracy, but also highlight its flexibility and adaptability, as it can seamlessly integrate with diverse tokenization techniques, making it a versatile enhancement for Transformer-based architectures. Figure 7 : (a) Correlation matrix among the Ground Truth variates, computed on the Weather Dataset. (b) PDFs of the Ground Truth and each model, computed using KDE. Model MSE( \mathbb{\downarrow} ) Cosine Similarity( \mathbb{\uparrow} ) SSIM( \mathbb{\uparrow} ) CASA(ours) 1.1204 0.9965 0.9912 SOFTS 1.2520 0.9957 0.9889 iTransformer 1.8675 0.9910 0.9791 PatchTST 1.2393 0.9960 0.9896 Table 4 : Metrics for the correlation matrices of the Ground Truth and each model. Boldface indicates the best performance, and underlining indicates the second-best performance. Robustness of CASA under varying conditions To validate the robustness of CASA with respect to input length and the number of variates, we conducted experiments on the ETTm1, Electricity, and Traffic datasets, which contain 7, 321, and 862 variates, respectively, with prediction lengths ranging from 48 to 720. As shown in Figure 5 , the performance of models utilizing non-channel-wise tokenization declines as the number of variates increases. Specifically, PatchTST experiences a significant performance drop on the Traffic dataset, recording the highest MSE losses. In contrast, models such as iTransformer and SOFTS, which employ channel-wise tokenization, demonstrate greater resilience to increases in the number of variates. However, both models exhibit elevated MSE losses on the ETTm1 dataset, while their performance improves on the Electricity and Traffic datasets. In comparison, our proposed model maintains stable MSE losses across all three datasets, achieving notably lower MSE losses on the ETTm1 dataset. This underscores CASA’s ability to deliver high predictive accuracy under varying conditions, ensuring consistent performance even as the input data length increases. 4.3 Model Efficiency Analysis In this section, we empirically validate the efficiency of CASA, as theoretically outlined in Section 3.4 . For comparison, we use iTransformer and SOFTS as baselines. Figure 6 (a) depicts memory usage, revealing that CASA exhibits linear complexity and effectively leverages practical computational resources. This performance is comparable to SOFTS, which also demonstrates similar complexity but increases memory usage significantly. Notably, CASA significantly outperforms iTransformer, which suffers from quadratic memory growth. Figures 6 (b) and (c) present memory footprints, inference time, and MSE for the Traffic and Electricity datasets, using a batch size of 16 and input/inference sequence lengths of 96. CASA consumes fewer computational resources than Transformer variants such as iTransformer, Crossformer, and PatchTST. Regarding MSE, CASA achieves the second-lowest value on Traffic and the lowest on Electricity, all while maintaining fast inference and efficient resource usage. Although CASA slightly exceeds TSMixer in memory usage, it delivers stronger overall performance, striking an effective balance between accuracy and resource efficiency. 4.4 Investigating Cross-Dimensional Interactions: A Correlation Matrix Analysis of CASA We evaluate CASA’s capacity to capture cross-dimensional interactions by examining the correlation matrices derived from each model’s predictions (i.e., correlations among all variates) on the Weather Dataset, comparing outcomes from CASA, SOFTS, iTransformer, and PatchTST against the Ground Truth. The Ground Truth correlation matrix shows distinct positive and negative correlation blocks, indicating a clear spatial structure with a grid-like arrangement (Figure 7 (a)). Furthermore, kernel density estimation (KDE) using a Gaussian kernel reveals that correlation values are mostly concentrated in the positive domain (Figure 7 (b)), suggesting many variates rise or fall in sync. Notably, CASA’s correlation matrix most closely approximates the Ground Truth, as evidenced by the lowest MSE loss between their probability density functions (PDFs) (Figure 7 (b)). This underscores CASA’s ability to preserve intricate relationships essential for capturing temporal and spatial dependencies in weather data. For a more comprehensive evaluation, we compare each model’s correlation matrix to the Ground Truth using Mean Squared Error (MSE), cosine similarity, and the Structural Similarity Index Measure (SSIM). CASA outperforms all other models across these metrics, validating its effectiveness in capturing cross-dimensional interactions (Table 4 ). 5 Conclusion In this study, we introduce the CASA model, which demonstrates remarkable effectiveness and solid predictive performance, as confirmed by a broad range of experiments. By delivering state-of-the-art results while requiring notably fewer computational resources and less processing time than existing approaches. Moreover, its CNN-based autoencoder module successfully captures cross-dimensional interactions throughout the compress-and-decompress procedure, which contributes to its outstanding performance. Additionally, CASA shows strong potential as a versatile alternative to conventional attention mechanisms in various Transformer configurations, remaining unaffected by different tokenization methods. This further highlights its adaptability, practicality, and efficiency across diverse use cases. Appendix A Dataset A.1 Details of Each Dataset Data preprocessing is largely based on previous work SOFTS. We provide the dataset descriptions: • ETT (Electricity Transformer Temperature) : This dataset includes two hourly (ETTh) and two 15-minute (ETTm) variations, containing seven features related to oil and load measurements of electricity transformers, collected from July 2016 to July 2018. • Traffic : This dataset represents hourly road occupancy rates gathered by sensors monitoring San Francisco freeways from 2015 to 2016. • Electricity : It consists of hourly electricity usage data for 321 consumers, spanning from 2012 to 2014. • Weather : it contains 21 weather-related variables, such as temperature and humidity, recorded every 10 minutes throughout 2020 in Germany. • Solar-Energy : It captures solar power generation data from 137 photovoltaic plants in 2006, sampled every 10 minutes. Further details of these datasets are summarized in Table 5 . Dataset Timesteps Sample Frequency Dimension Weather 52,696 10 min 21 Electricity 26,304 1 hour 321 Traffic 17,544 1 hour 862 Solar 52,560 10 min 137 ETTh1 17,420 1 hour 7 ETTh2 17,420 1 hour 7 ETTm1 69,680 15 min 7 ETTm2 69,680 15 min 7 Table 5 : Characteristics of the datasets including timesteps, sample frequency, and dimensions. Dimensions denotes the number of variate in each dataset. A.2 Properties of datasets The limited size of datasets in the LTSF domain reflects the inherent constraints of real-world benchmark datasets. However, the use of synthetic data is not feasible due to critical concerns about data reliability and authenticity Hao et al. [2024] ; Gonzales et al. [2023] . These limitations highlight the importance of studying LTSF under such constraints, as it addresses practical challenges and ensures that the models remain applicable. B Full Results We fix the input length to L=96 and evaluated the prediction results across 8 real-world datasets for H\in\{96,192,336,720\} . For each dataset, four MSE and MAE is calculated, resulting in a total of 64 metrics. Among these, CASA achieves the best performance in 54 cases (84.4%). Detailed results are presented in Table 6 . C Visualization Results Figure 8 to Figure 14 visualize the prediction performance of weather dataset from CASA, SOFTS, iTransformer, and PatchTST models against the ground truth. The both input length and the prediction length are fixed to 96. Figure 8 : Results of prediction of Ours and baseline models on Etth1 dataset Figure 9 : Results of prediction of Ours and baseline models on Etth2 dataset Figure 10 : Results of prediction of Ours and baseline models on Ettm1 dataset Figure 11 : Results of prediction of Ours and baseline models on Ettm2 dataset Figure 12 : Results of prediction of Ours and baseline models on Traffic dataset Figure 13 : Results of prediction of Ours and baseline models on Solar dataset Figure 14 : Results of prediction of Ours and baseline models on Electricity dataset D Implementation Details in Section 3.4. In the experiments conducted on the Traffic dataset, N is set to 862. The parameters L and H are configured as 512 and 96, respectively, with a batch size of 16. CASA and SOFTS exhibit the same complexity with respect to N , L , and H , but their experimental results show slight variations due to differences in hyperparameter settings, such as the hidden dimension and kernel size. Except for CASA and SOFTS, all models include a quadratic term. However, iTransformer employs channel-wise tokenization, resulting in a quadratic term with respect to N , whereas PatchTST and Transformer exhibit a quadratic term with respect to L . Given that 1.5\times L\leq N in the training setup, iTransformer is less efficient compared to Transformer. Additionally, PatchTST involves an NL^{2} term, leading to the highest computational complexity as the input length and the number of variates increase. Metric Models CASA(ours) SOFTS iTransformer PatchTST TSMixer Crossformer TiDE TimesNet Stationary DLinear SCINet FEDformer MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE ETTm1 96 0.313 0.358 0.325 0.361 0.334 0.368 0.329 0.365 0.323 0.363 0.404 0.426 0.364 0.387 0.338 0.375 0.386 0.398 0.345 0.372 0.418 0.438 0.379 0.419 192 0.375 0.382 0.377 0.391 0.380 0.394 0.376 0.392 0.450 0.451 0.398 0.404 0.374 0.387 0.426 0.441 0.459 0.444 0.380 0.389 0.439 0.450 0.426 0.441 336 0.397 0.398 0.426 0.420 0.400 0.410 0.407 0.413 0.532 0.515 0.428 0.425 0.410 0.411 0.445 0.459 0.495 0.46 0.413 0.413 0.490 0.485 0.445 0.459 720 0.459 0.437 0.466 0.447 0.491 0.459 0.475 0.453 0.485 0.459 0.666 0.589 0.487 0.461 0.478 0.450 0.585 0.516 0.474 0.453 0.595 0.550 0.543 0.490 Avg 0.386 0.393 0.393 0.403 0.407 0.410 0.396 0.406 0.398 0.407 0.513 0.496 0.419 0.419 0.400 0.406 0.481 0.456 0.474 0.453 0.595 0.550 0.543 0.490 ETTm2 96 0.173 0.252 0.180 0.261 0.180 0.264 0.184 0.264 0.182 0.266 0.287 0.366 0.207 0.305 0.187 0.267 0.192 0.274 0.193 0.292 0.286 0.377 0.203 0.287 192 0.240 0.298 0.246 0.306 0.250 0.309 0.246 0.306 0.249 0.309 0.414 0.492 0.290 0.364 0.249 0.309 0.280 0.339 0.284 0.362 0.399 0.445 0.269 0.328 336 0.296 0.334 0.319 0.352 0.311 0.348 0.308 0.346 0.309 0.347 0.597 0.542 0.377 0.422 0.321 0.351 0.334 0.361 0.369 0.427 0.637 0.591 0.325 0.366 720 0.395 0.392 0.405 0.401 0.412 0.407 0.409 0.402 0.416 0.408 1.730 1.042 0.558 0.524 0.408 0.403 0.417 0.413 0.554 0.522 0.960 0.735 0.421 0.415 Avg 0.276 0.319 0.287 0.330 0.288 0.332 0.287 0.330 0.289 0.333 0.757 0.610 0.358 0.404 0.291 0.333 0.306 0.347 0.350 0.401 0.571 0.537 0.305 0.349 ETTh1 96 0.376 0.397 0.381 0.399 0.386 0.405 0.394 0.406 0.401 0.412 0.423 0.448 0.479 0.464 0.384 0.402 0.513 0.491 0.386 0.400 0.654 0.599 0.376 0.419 192 0.427 0.424 0.435 0.431 0.441 0.436 0.440 0.435 0.452 0.442 0.471 0.474 0.525 0.492 0.436 0.429 0.534 0.504 0.437 0.432 0.719 0.631 0.420 0.448 336 0.469 0.445 0.480 0.452 0.487 0.458 0.491 0.462 0.492 0.463 0.570 0.546 0.565 0.515 0.491 0.469 0.588 0.535 0.481 0.459 0.778 0.659 0.459 0.465 720 0.479 0.468 0.499 0.488 0.503 0.491 0.487 0.479 0.507 0.490 0.653 0.621 0.594 0.558 0.521 0.500 0.643 0.616 0.519 0.516 0.836 0.699 0.506 0.507 Avg 0.438 0.434 0.449 0.442 0.454 0.447 0.453 0.446 0.463 0.452 0.529 0.522 0.541 0.507 0.458 0.450 0.570 0.537 0.456 0.452 0.747 0.647 0.440 0.460 ETTh2 96 0.290 0.339 0.297 0.347 0.297 0.349 0.288 0.340 0.319 0.361 0.745 0.584 0.400 0.440 0.340 0.374 0.476 0.458 0.333 0.387 0.707 0.621 0.358 0.397 192 0.366 0.388 0.373 0.394 0.380 0.400 0.376 0.395 0.402 0.410 0.877 0.656 0.528 0.509 0.402 0.414 0.512 0.493 0.477 0.476 0.860 0.689 0.429 0.439 336 0.416 0.425 0.410 0.426 0.428 0.432 0.440 0.451 0.444 0.446 1.043 0.731 0.643 0.571 0.452 0.452 0.552 0.551 0.594 0.541 1.000 0.744 0.496 0.487 720 0.420 0.439 0.411 0.433 0.427 0.445 0.436 0.453 0.441 0.450 1.104 0.763 0.874 0.679 0.462 0.468 0.562 0.560 0.831 0.657 1.249 0.838 0.463 0.474 Avg 0.374 0.397 0.373 0.400 0.383 0.407 0.385 0.410 0.401 0.417 0.942 0.684 0.611 0.550 0.414 0.427 0.526 0.516 0.559 0.515 0.954 0.723 0.437 0.449 ECL 96 0.138 0.231 0.143 0.233 0.148 0.240 0.164 0.251 0.157 0.260 0.219 0.314 0.237 0.329 0.168 0.272 0.169 0.273 0.197 0.282 0.247 0.345 0.193 0.308 192 0.157 0.248 0.158 0.248 0.162 0.253 0.173 0.262 0.173 0.274 0.231 0.322 0.236 0.330 0.184 0.289 0.182 0.286 0.196 0.285 0.257 0.355 0.201 0.315 336 0.175 0.267 0.178 0.269 0.178 0.269 0.190 0.279 0.192 0.295 0.246 0.337 0.249 0.344 0.198 0.300 0.200 0.304 0.209 0.301 0.269 0.369 0.214 0.329 720 0.200 0.290 0.218 0.305 0.225 0.317 0.230 0.313 0.223 0.318 0.280 0.363 0.284 0.373 0.220 0.320 0.222 0.321 0.245 0.333 0.299 0.390 0.246 0.355 Avg 0.168 0.259 0.174 0.264 0.178 0.270 0.189 0.276 0.186 0.287 0.244 0.334 0.251 0.344 0.192 0.295 0.193 0.296 0.212 0.300 0.268 0.365 0.214 0.327 Traffic 96 0.386 0.243 0.376 0.251 0.395 0.268 0.427 0.272 0.493 0.336 0.522 0.290 0.805 0.493 0.593 0.321 0.612 0.338 0.650 0.396 0.788 0.499 0.587 0.366 192 0.410 0.255 0.398 0.261 0.417 0.276 0.454 0.289 0.497 0.351 0.530 0.293 0.756 0.474 0.617 0.336 0.613 0.340 0.598 0.370 0.789 0.505 0.604 0.373 336 0.426 0.263 0.415 0.269 0.433 0.283 0.450 0.282 0.528 0.361 0.558 0.305 0.762 0.477 0.629 0.336 0.618 0.328 0.605 0.373 0.797 0.508 0.621 0.383 720 0.461 0.282 0.447 0.287 0.467 0.302 0.484 0.301 0.569 0.380 0.589 0.328 0.719 0.449 0.640 0.350 0.653 0.355 0.645 0.394 0.841 0.523 0.626 0.382 Avg 0.421 0.261 0.409 0.267 0.428 0.282 0.454 0.286 0.522 0.357 0.550 0.304 0.760 0.473 0.620 0.336 0.624 0.340 0.625 0.383 0.804 0.509 0.610 0.376 Weather 96 0.155 0.197 0.166 0.208 0.174 0.214 0.176 0.217 0.166 0.210 0.158 0.230 0.202 0.261 0.172 0.220 0.173 0.223 0.196 0.255 0.221 0.306 0.217 0.296 192 0.206 0.245 0.217 0.253 0.221 0.254 0.221 0.256 0.215 0.256 0.206 0.277 0.242 0.298 0.219 0.261 0.245 0.285 0.237 0.296 0.261 0.340 0.276 0.336 336 0.265 0.286 0.282 0.300 0.278 0.296 0.275 0.296 0.287 0.300 0.272 0.335 0.287 0.335 0.280 0.306 0.321 0.338 0.283 0.335 0.309 0.378 0.339 0.380 720 0.347 0.340 0.356 0.351 0.358 0.347 0.352 0.346 0.355 0.348 0.398 0.418 0.351 0.386 0.365 0.359 0.414 0.410 0.345 0.381 0.377 0.427 0.403 0.428 Avg 0.243 0.267 0.255 0.278 0.258 0.278 0.256 0.279 0.256 0.279 0.259 0.315 0.271 0.320 0.259 0.287 0.288 0.314 0.265 0.317 0.292 0.363 0.309 0.360 Solar 96 0.187 0.218 0.200 0.230 0.203 0.237 0.205 0.246 0.221 0.275 0.310 0.331 0.312 0.399 0.250 0.292 0.215 0.249 0.290 0.378 0.237 0.344 0.242 0.342 192 0.223 0.243 0.229 0.253 0.233 0.261 0.237 0.267 0.268 0.306 0.734 0.725 0.339 0.416 0.296 0.318 0.254 0.272 0.320 0.398 0.280 0.380 0.285 0.380 336 0.237 0.257 0.243 0.269 0.248 0.273 0.250 0.276 0.272 0.294 0.750 0.735 0.368 0.430 0.319 0.330 0.290 0.296 0.353 0.415 0.304 0.389 0.282 0.376 720 0.239 0.258 0.245 0.272 0.249 0.275 0.252 0.275 0.281 0.313 0.769 0.765 0.370 0.425 0.338 0.337 0.285 0.200 0.356 0.413 0.308 0.388 0.357 0.427 Avg 0.221 0.244 0.229 0.256 0.233 0.262 0.236 0.266 0.260 0.297 0.641 0.639 0.347 0.417 0.301 0.319 0.261 0.381 0.330 0.401 0.282 0.375 0.291 0.381 1^{st}/2^{nd} count 54 10 11 34 0 7 1 11 0 1 1 2 0 3 0 0 0 0 0 0 0 0 3 0 Table 6 : Performance comparison across 8 real-world datasets with L=96 and varying H : CASA achieves the best results in 54 out of 64 metrics (84.4%) E Proofs E.1 Proof of Proposition 1. Proof. For matrix A , define the r -th row as \mathbf{row}_{r}(A) and the s -th column as \mathbf{col}_{s}(A) . Furthermore, for the vector v , define its l -th component as v(l) . Then, the feature for the r -th variate in Q_{i} , denoted as \mathbf{row}_{r}(Q_{i}) is calculated as follows: \displaystyle\mathbf{row}_{r}(Q_{i+1}) \displaystyle=\mathbf{row}_{r}(Z_{i,1})*W_{i,1}+\mathbf{row}_{r}(b_{i,1}) (10) Take any \Delta Z_{i,1}\in\mathbb{R}^{N\times D} such that \;\mathbf{row}_{r}(\Delta Z_{i,1})=\vec{0} . Define \widetilde{Z}_{i,1}\coloneqq Z_{i,1}+\Delta Z_{i,1} . Then \mathbf{row}_{r}(\widetilde{Z}_{i,1})=\mathbf{row}_{r}(Z_{i,1}) . Therefore, \displaystyle\mathbf{row}_{r}(Q_{i+1}) \displaystyle=\mathbf{row}_{r}(Z_{i,1})*W_{i,1}+\mathbf{row}_{r}(b_{i,1}) (11) \displaystyle=\mathbf{row}_{r}(\widetilde{Z}_{i,1})*W_{i,1}+\mathbf{row}_{r}(b_{i,1}) (12) \displaystyle=\mathbf{row}_{r}(\widetilde{Q}_{i+1}) (13) Thus, for any r , the value of \mathbf{row}_{r}(Q_{i+1}) depends solely on the columns of W_{i,1} without interacting with other rows, that is, other variates. Similarly, key embeddings reach the same conclusion in the same way. ∎ E.2 Proof of Proposition 2. Proof. By transposing Z_{i,1} , it can be proven in the same way as Proposition 1 ∎