Hybrid preprocessing for neural network-based stock price prediction☆ Authors: Jian-Lei Li, Wei-Kang Shi Published: 2024 ABSTRACT In the domain of stock price prediction, the intricate interdependencies within multivariate time series data present significant challenges for accurate forecasting. This paper introduces a groundbreaking hybrid preprocessing technique to tackle this issue. By leveraging the Empirical Wavelet Transform (EWT), we adeptly extract both low-frequency and high-frequency components from the time series. We then apply Dynamic Time Warping (DTW) and Differential Dynamic Time Warping (DDTW) to measure component similarities, identifying correlated patterns within the stock price series. High-frequency components are managed using sliding windows and Principal Component Analysis (PCA), while PCA is directly applied to low-frequency components. Integrating these techniques into neural network models, our approach yields a substantial 30% improvement in prediction accuracy compared to traditional methods. This significant advancement underscores the potential of our hybrid preprocessing method in enhancing stock price prediction accuracy, offering valuable insights for financial market analysis. Introduction Time series forecasting [1] is a crucial research area focusing on analyzing and processing data sequences that vary over time [2], to predict future trends [3], patterns [4], or values [5]. Research in this field is vital for various industries, including financial market analysis [6], weather forecasting, energy demand prediction [7], and inventory management. Accurate time series forecasting can help businesses and organizations make more informed decisions [8], optimize resource allocation [9], reduce uncertainty, and enhance efficiency. In recent years, neural networks have become prevalent in stock price prediction due to their powerful nonlinear modeling capabilities. Despite their strengths, neural networks often require substantial computational resources and may sometimes fall short in prediction accuracy. Stock price prediction is inherently challenging due to the noise and volatility of financial markets. Many models have been proposed to tackle this issue, but they often struggle to capture the complex patterns in stock price data, especially without adequate preprocessing. Recent studies have integrated preprocessing techniques with advanced models to enhance prediction accuracy. For instance, Quilty et al. [10] utilized classification algorithms with data preprocessing to reduce prediction errors. Ma et al. [11] introduced the Multi-source Aggregated Classification (MAC) method, which uses graph convolutional networks and pre-trained embedding feature generators to outperform state-of-the-art baselines in prediction accuracy and financial metrics. Similarly, Sonkamble et al. [12] employed Principal Component Analysis (PCA) to simplify data before applying linear regression, which improved predictive precision for Tesla stock data. Wang [13] compared machine learning models for Apple stock, finding that linear regression and Long Short-Term Memory (LSTM) networks were most effective, with LSTM showing robustness for time-series data. Other notable studies include Rajpurohit et al. [14], who integrated LSTM with sentiment analysis for data preprocessing to enhance prediction accuracy. Liang et al. [15] combined LSTM with wavelet transform-based preprocessing, showing significant improvements in performance. Chen et al. [16] proposed the XCEEMDAN-Bidirectional LSTM-Spline model, which effectively utilized feature extraction for superior performance. Cai [17] introduced a CNN-GRU-attention model employing various decomposition methods like EMD, EEMD, and CEEMDAN, which outperformed conventional models. Furthermore, Jiang et al. [18] proposed a model integrating affinity propagation clustering with convolutional neural networks (CNN) for feature extraction, and LSTM for multifactor analysis, achieving superior prediction performance. Rezaei et al. [19] combined empirical mode decomposition (EMD) techniques with CNN and LSTM models, significantly enhancing feature extraction and prediction accuracy. Lastly, Zarandi et al. [20] developed a hybrid fuzzy intelligent agent-based system for stock price prediction, demonstrating the effectiveness of combining fuzzy logic with intelligent agents for financial forecasting. These studies highlight the continuous evolution of methodologies aimed at improving stock price prediction, emphasizing the importance of innovative preprocessing and modeling techniques. Our proposed hybrid preprocessing technique aligns with these advancements, offering a novel combination of Empirical Wavelet Transform (EWT), Dynamic Time Warping (DTW), Derivative Dynamic Time Warping (DDTW), and PCA. This approach addresses both low-frequency trends and high-frequency fluctuations, significantly enhancing prediction accuracy and model robustness. In conclusion, our hybrid preprocessing technique demonstrates superior performance in stock price prediction, as evidenced by closer predicted-actual curves and lower error indicators. This method not only captures complex patterns in stock data but also improves the overall robustness of the prediction models. Future research should continue optimizing preprocessing techniques and model hyperparameters to further enhance prediction accuracy and reliability for investors and decision-makers. Image 1 This paper introduces a novel hybrid preprocessing technique for stock price prediction [21], utilizing Empirical Wavelet Transform (EWT) [22] for decomposing time series into low-frequency trends and high-frequency details. Dynamic Time Warping (DTW) [23] and Derivative Dynamic Time Warping (DDTW) are combined for advanced similarity measurement, capturing both global similarity and local trends, which improves the identification of stock price patterns. Applying these techniques to neural network models enhances their ability to learn complex patterns, significantly boosting prediction accuracy and robustness. By integrating EWT, DTW, and DDTW with neural networks, the method achieves superior precision in stock price prediction. Key contributions of this paper include:•Enhanced Predictive Accuracy: Demonstrating significant improvement in predictive accuracy and trend-tracking capabilities of neural network models through hybrid preprocessing.•Comprehensive Model Comparison: Providing a detailed comparison of model performances with and without preprocessing.•Future Research Directions: Highlighting the importance of preprocessing in multivariate time series forecasting and suggesting future optimization of preprocessing techniques and model hyperparameters. •Enhanced Predictive Accuracy: Demonstrating significant improvement in predictive accuracy and trend-tracking capabilities of neural network models through hybrid preprocessing. Enhanced Predictive Accuracy: Demonstrating significant improvement in predictive accuracy and trend-tracking capabilities of neural network models through hybrid preprocessing. •Comprehensive Model Comparison: Providing a detailed comparison of model performances with and without preprocessing. Comprehensive Model Comparison: Providing a detailed comparison of model performances with and without preprocessing. •Future Research Directions: Highlighting the importance of preprocessing in multivariate time series forecasting and suggesting future optimization of preprocessing techniques and model hyperparameters. Future Research Directions: Highlighting the importance of preprocessing in multivariate time series forecasting and suggesting future optimization of preprocessing techniques and model hyperparameters. The paper is structured as follows: Section 2 covers basic concepts like EWT, DTW, and PCA [24]. Section 3 details the hybrid preprocessing model. Section 4 presents experimental results and discussions. Section 5 concludes the paper. Basic concepts This section primarily discusses time series preprocessing techniques, mainly including decomposition techniques, similarity calculation, and principal component extraction. Time series decomposition In choosing EWT for decomposition, the main consideration is its ability to effectively handle nonlinear and non-stationary signals. Traditional wavelet transforms are less adaptive to signals and require predefined basis functions, whereas EWT dynamically adjusts basis functions based on actual data characteristics, making it more suitable for complex financial market data analysis. The original signal is analyzed using FFT (Fast Fourier Transform), and according to the Shannon criterion, the signal's Fourier spectrum is normalized within [0,π] and divided into continuous N segments [A1,A2,…,AN], resulting in N+1 boundary lines on the spectrum. Let ωn be the interval boundary line, then An=[ωn−1,ωn], where ω0=0 and ωN=π are the interval boundary lines. The schematic diagram of the signal spectrum division is shown in Fig. 1.Figure 1Schematic diagram of the signal spectrum division.Figure 1 Schematic diagram of the signal spectrum division. The width of the transition phase is given by the equation:(1)τn=γ×ωn,0<γ