Evaluating the effectiveness of self-attention mechanism in tuberculosis time series forecasting Authors: Zhihong Lv, Rui Sun, Xin Liu, Shuo Wang, Xiaowei Guo, Yuan Lv, Min Yao, Junhua Zhou Published: 2024 ABSTRACT BackgroundWith the increasing impact of tuberculosis on public health, accurately predicting future tuberculosis cases is crucial for optimizing of health resources and medical service allocation. This study applies a self-attention mechanism to predict the number of tuberculosis cases, aiming to evaluate its effectiveness in forecasting.MethodsMonthly tuberculosis case data from Changde City between 2010 and 2021 were used to construct a self-attention model, a long short-term memory (LSTM) model, and an autoregressive integrated moving average (ARIMA) model. The performance of these models was evaluated using three metrics: root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE).ResultsThe self-attention model outperformed the other models in terms of prediction accuracy. On the test set, the RMSE of the self-attention model was approximately 7.41% lower than that of the LSTM model, MAE was reduced by about 10.99%, and MAPE was reduced by approximately 9.87%. Compared to the ARIMA model, RMSE was reduced by about 28.86%, MAE by about 32.22%, and MAPE by approximately 29.89%.ConclusionThe self-attention model can effectively improve the prediction accuracy of tuberculosis cases, providing guidance for health departments optimizing of health resources and medical service allocation.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12879-024-10183-9. Introduction Tuberculosis (TB) is a highly infectious disease caused by Mycobacterium tuberculosis, primarily transmitted through the air, and bring a serious threat to global public health [1]. According to the “Global Tuberculosis Report 2023” published by the World Health Organization, there were approximately 10.6 million new cases of TB worldwide in 2022, with an incidence rate of 133 per 100,000 people [2]. China is classified as a high-burden country by the World Health Organization, ranking third globally in the estimated number of TB cases in 2022 [3]. Due to the large population base of China, TB brings a severe burden on public health and the economy [4]. Therefore, it is crucial to predict the future number of TB cases to optimize health resources and medical service allocation. Achieving these goals requires accurate models for predicting TB cases, serving as an early warning system. Predicting TB cases based on historical data, known as time series forecasting (TSF), is the focus of this study. Time series forecasting (TSF) methods for diseases can be broadly categorized into statistical-based and machine learning-based approaches. Statistical-based methods, such as the Grey model [5], the exponential smoothing model [6], and the autoregressive integrated moving average (ARIMA) model [7], are widely used due to their effectiveness in capturing the linear characteristics of time series. For example, studies have shown that seasonal ARIMA can be used as a predictive tool for TB cases in China [8]. However, these methods are not suitable for time series with nonlinear characteristics, such as TB case data [9, 10]. To capture the nonlinear features of time series, machine learning methods such as random forests [11], support vector machines [12], and back-propagation neural networks [13, 14] have been introduced into TSF for diseases [15, 16]. Machine learning models offer higher predictive accuracy when dealing with time series that exhibit nonlinear characteristics. For example, a comparative study indicated that the back-propagation neural network model outperformed the ARIMA model in a TSF task for acquired immunodeficiency syndrome [17]. However, traditional machine learning models struggle to capture long-term dependencies [18]. Long-term dependency refers to the reliance on earlier data in the time series for predicting future cases. Early data in the time series often contain critical information, such as seasonal patterns, epidemiological characteristics, and the latency of disease transmission [19]. For example, in China, TB cases typically peak in spring and tend to decline in autumn and winter [20]. If a model cannot effectively utilize the early data in the time series, the accuracy of its predictions may be significantly reduced. With advances in computer technology, deep learning models such as recurrent neural networks and long short-term memory (LSTM) networks have been successfully applied to time series forecasting (TSF) for TB [21], influenza [22], and other respiratory diseases [23], largely due to their ability to capture long-term dependencies in time series data. Studies have shown that, compared to ARIMA models, LSTM models reduced the prediction error for TB by 12.92–36.79% [24]. However, as the length of the time series increases, LSTM models may “forget” earlier data, leading to a decline in performance [25, 26]. To address this issue, the Google research team proposed a deep learning model based on the self-attention mechanism in 2017. Unlike LSTM, the self-attention network retains early data in the time series, making it suitable for time series of any length. While this approach has been widely applied in fields such as natural language processing [27] and speech recognition [28], its application in TSF tasks for diseases remains limited. Therefore, this study proposes a model based on the self-attention mechanism to predict TB cases. The core question of this research is: Can the self-attention mechanism effectively improve the accuracy of TB TSF? First, we collected data on TB cases and performed data description and preprocessing. Next, we constructed prediction models for TB cases, including a self-attention model, an LSTM model, and an ARIMA model. Finally, the structures of the three models were optimized and evaluated, with a particular focus on examining the interpretability and generalizability of the self-attention model. Figure 1 presents the main framework of the study. Fig. 1The main framework of the research in this paper The main framework of the research in this paper Methods Data Collection and Preprocessing Data Collection and description In this study, data on TB cases were obtained from the Disease Surveillance Information Reporting Management System of the Changde Center for Disease Control and Prevention. All TB diagnoses adhered to the Diagnostic Criteria for Tuberculosis (WS 288–2017 edition), issued by the National Health and Family Planning Commission of China. The data spans from January 2010 to December 2021, covering the monthly number of TB cases. This study collected data on TB cases spanning a total of 144 months. The dataset is complete, with no outliers or missing values. We strictly adhered to the temporal sequence when dividing the dataset, ensuring that future data were not accessed prematurely during model training, thereby maintaining the reliability of the evaluation results. Specifically, the training set included data from January 2010 to December 2019, the validation set contained data from January 2020 to December 2020, and the test set used data from January 2021 to December 2021. Table 1 presents the basic statistical description of TB cases for each dataset. The results show that a total of 61,061 TB cases were reported in Changde City from 2010 to 2021, with the highest number of cases in a single month being 654 and the lowest being 176. Figure 2 illustrates the time series of TB cases. Table 1Monthly tuberculosis cases in Changde City (2010–2021): training, testing, and validation dataset descriptionDatasetTime rangeTotal casesMean\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}SDMinP25MedianP95MaxTrain2010–201951,716431\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}90176370427498654Validation20205035419\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}72304365434474539Test20214310359\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}432913473533764502010–202161,061424\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}88176360421488654 Table 1Monthly tuberculosis cases in Changde City (2010–2021): training, testing, and validation dataset descriptionDatasetTime rangeTotal casesMean\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}SDMinP25MedianP95MaxTrain2010–201951,716431\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}90176370427498654Validation20205035419\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}72304365434474539Test20214310359\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}432913473533764502010–202161,061424\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\pm\:$$\end{document}88176360421488654 Monthly tuberculosis cases in Changde City (2010–2021): training, testing, and validation dataset description Fig. 2Monthly time series of tuberculosis cases in Changde City, 2020–2021 Monthly time series of tuberculosis cases in Changde City, 2020–2021 To analyze the trend in TB cases, this study employed the widely used Seasonal and Trend decomposition using Loess (STL) technique to break down the time series into three components: trend, seasonality, and residuals [29]. Figure 3 illustrates the decomposition of the TB time series. The trend component shows an overall downward trend in TB cases in Changde City from 2010 to 2021. The seasonal component reveals that January to February is the low period for TB cases, March to June represents the peak period, and July to December marks the declining period. Fig. 3Decomposition of monthly tuberculosis time series in Changde City, 2020–2021 Decomposition of monthly tuberculosis time series in Changde City, 2020–2021 Data preprocessing To facilitate faster model convergence, the data needed to be preprocessed [30]. First, the TB cases data were scaled to a range between 0 and 1 using a normalization formula to eliminate differences in variable magnitude. The normalization formula is as follows:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{X}_{norm}=\frac{X-{X}_{min}}{{X}_{max}-{X}_{min}}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{X}_{norm}$$\end{document}​ represents the normalized value, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:X$$\end{document} represents the number of TB cases, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{X}_{min}$$\end{document}​ represents the minimum number of TB cases, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{X}_{max}$$\end{document} represents the maximum number of TB cases. Next, the normalized data were transformed into input-output pairs using a fixed time window. Figure 4 illustrates the data transformation process. First, the size of the time window was determined. Then, using a sliding window approach with a time step of 1, data were extracted from the TB time series. The data within each time window served as the model’s input, while the subsequent data point was used as the model’s output. These input-output pairs were then used to train the model, enabling it to predict future TB cases. Fig. 4Flowchart of the data transformation process Flowchart of the data transformation process Models for Predicting tuberculosis cases Self-attention model The self-attention mechanism, which forms a key part of the Transformer model proposed by Ashish Vaswani et al. in 2017, relies on an attention weight matrix to allocate focus within a time series, thereby enhancing the model’s ability to capture long-term dependencies [31]. The attention weight matrix reflects the importance of each input data point in generating the output. In this study, the input data consist of a sequence of historical TB cases over \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:t$$\end{document} time windows, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:X=\left[{\chi\:}_{1},{\chi\:}_{2},\cdots\:,{\chi\:}_{t}\:\right]$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\chi\:}_{t}$$\end{document} represents the number of TB cases in month \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:t$$\end{document}. Figure 5 illustrates the calculation principle of the self-attention mechanism. First, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:X$$\end{document} is mapped through a linear transformation into three vectors: the query vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Q$$\end{document}, the key vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:K$$\end{document}, and the value vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:V$$\end{document}. Then, the query vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Q$$\end{document} is dotted with the transpose of the key vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:K$$\end{document} to calculate a similarity score, known as the attention weight matrix. Subsequently, the attention weight matrix is multiplied by the value vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:V$$\end{document} to obtain the self-attention output, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Attention\left(X\right)$$\end{document}. Fig. 5Schematic diagram of self-attention mechanism Schematic diagram of self-attention mechanism Multi-head self-attention [31] divides the self-attention computation into multiple “heads,” with each head responsible for learning different subspace representations of the data. Since multi-head self-attention can analyze input data from multiple perspectives simultaneously, it can capture diverse features and patterns in the data. Therefore, this study employed multi-head self-attention to construct the prediction model. The calculation formula is as follows:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Q}_{i}={W}_{i}^{q}\cdot\:X$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{K}_{i}={W}_{i}^{k}\cdot\:X$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{V}_{i}={W}_{i}^{v}\cdot\:X$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\>hea{d_i} = Attention\left( {{Q_i},\,{K_i},\,{V_i}} \right) = softmax\left( {{{{Q_i}{K_i}^T} \over {\sqrt {{d_k}} }}} \right){V_i}$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:MultiHead\left(Q,\,K,\,V\right)=Concat\left({head}_{1},\cdots\:,{head}_{i}\right){W}^{o}$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}_{i}^{q}$$\end{document}​, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}_{i}^{k}$$\end{document}​, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}_{i}^{v}$$\end{document}​, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{W}^{o}$$\end{document} represent the learnable weight matrices, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document} represents the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document}-th attention head. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{head}_{i}$$\end{document} represents the output of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document}-th attention head. The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:softmax$$\end{document} function is used to normalize the similarity scores into a probability distribution. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{k}$$\end{document}​ represents the length of the key vector for each attention head and dividing by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\sqrt{{d}_{k}}$$\end{document}​​ is used to maintain numerical stability of the model. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:MultiHead\left(Q,K,V\right)$$\end{document} represents the output of the multi-head self-attention mechanism. Figure 6 presents the framework of the self-attention model used in this study. The model consists of three hidden layers: the self-attention layer, the global average pooling layer, and the fully connected layer. The self-attention layer captures the long-term dependency features in the data, the global average pooling layer reduces dimensionality, and the fully connected layer generates the prediction results. Fig. 6Structural framework of the self-attention model Structural framework of the self-attention model In addition, the model’s hyperparameters, which are preset by the developers, determine the structure of the model. We set the ranges of these hyperparameters based on relevant TSF literature and model reports [32, 33]. In this study, the time window was set to 4/6/8/12, the number of attention heads to 2/4/8, the dimension per head to 16/32/64, the number of neurons to 4/8/16/32/64, the batch size to 32/64, the learning rate to 0.001, and the maximum number of epochs to 1000. Since the Adam optimization algorithm combines the advantages of momentum and an adaptive learning rate, it effectively facilitates the model’s rapid convergence and stable learning process [34]. Therefore, we used Adam to optimize the model parameters in this study. LSTM Model The Long Short-Term Memory (LSTM) network was proposed by Sepp Hochreiter and Jürgen Schmidhuber in 1997, specifically designed to address the issues of gradient explosion or vanishing that recurrent neural networks encounter when processing long sequence data [35]. The LSTM network controls the storage, updating, and output of information through the introduction of three gates: the input gate, forget gate, and output gate. The forget gate mainly determines which information should be discarded from the cell state. The input gate decides which new information should be stored in the cell state. The cell state passes information through the entire chain, using the forget gate to remove unimportant data and the input gate to add new information. The output gate is responsible for reviewing the values in the cell state and determining which part of the information should be output. In this study, the LSTM model consists of two hidden layers: an LSTM layer and a fully connected layer. The hyperparameter settings for the model are as follows: the time window is set to 4/6/8/12, with a time step of 1. The number of neurons is set to 4/8/16/32/64, the batch size is set to 32/64, the learning rate is 0.001, and the maximum number of epochs is set to 1000. The Adam algorithm was used to optimize the model parameters. ARIMA Model The Autoregressive Integrated Moving Average (ARIMA) model was proposed by Box and Jenkins in the early 1970s, is primarily used for analyzing and forecasting time series data [36]. The ARIMA model combines three elements: autoregression, integration, and moving average. The model is represented as ARIMA (p, d, q), where p represents the order of the autoregressive terms, indicating the linear relationship between the current observation and its past p observations; d represents the order of differencing needed to make the series stationary; and q represents the order of the moving average terms, indicating the linear relationship between the current error term and its past q error terms. When the research data exhibit seasonal trends, the seasonal ARIMA model is derived as an extension of the ARIMA model. The seasonal ARIMA model is typically represented as ARIMA (p, d, q) (P, D, Q) S​, where P represents the seasonal autoregressive order, D represents the seasonal differencing order, Q represents the seasonal moving average order, and S represents the length of the seasonal cycle. In this study, the establishment of the ARIMA model involved the following steps: First, the time series was visualized, and the Augmented Dickey-Fuller test was used to check if the series was stationary. If the series was not stationary, differencing was applied until stationarity was achieved. Then, the Autocorrelation Function and Partial Autocorrelation Function plots of the stationary series were generated to determine the range of p, q, P, and Q values. Finally, the auto_arima function in Python was used to fit models with different parameters, and the optimal model was selected based on the Akaike Information Criterion, Bayesian Information Criterion, and Ljung-Box Q test. Model optimization The self-attention model and LSTM model were optimized using random search for hyperparameter tuning. Compared to grid search, random search is more efficient in larger search spaces, allowing it to find high-performing hyperparameter combinations more quickly [37]. First, multiple hyperparameter combinations were randomly sampled from a predefined hyperparameter space, and the model was trained with each combination. During each training process, the convergence of the model and possible overfitting was assessed by observing the loss curves for both the training and validation sets. Next, the validation set was used to evaluate the performance of each model configuration, with mean squared error used as the evaluation metric. The effects of different hyperparameter combinations were compared, and the optimal configuration was determined based on the combination that performed best on the validation set. In contrast, ARIMA models have a relatively small parameter space, which makes them suitable for exhaustive exploration using grid search. The optimal parameter combination was selected by thoroughly examining all possible configurations and combining statistical metrics such as the Akaike Information Criterion, Bayesian Information Criterion, and Ljung-Box Q test. Model evaluation metrics Currently, root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE) are widely used to evaluate time series forecasting models [17, 38]. Therefore, this study uses these three metrics to evaluate the performance of each model. RMSE is the square root of the average of the squared deviations between actual values and predicted values. The calculation formula is:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:RMSE=\sqrt{\frac{1}{n}{\sum\:}_{i=1}^{n}{\left({y}_{i}-\widehat{{y}_{i}}\right)}^{2}}$$\end{document} MAE is the average of the absolute deviations between the actual values and the predicted values. The calculation formula is:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:MAE=\frac{1}{n}{\sum\:}_{i=1}^{n}\left|{y}_{i}-\widehat{{y}_{i}}\right|$$\end{document} MAPE is the average proportion of the deviation between actual values and predicted values relative to the actual values. The calculation formula is:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:MAPE=\frac{100\%}{n}{\sum\:}_{i=1}^{n}\left|\frac{{y}_{i}-\widehat{{y}_{i}}}{{y}_{i}}\right|$$\end{document} Where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:n$$\end{document} is the number of samples, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{y}_{i}$$\end{document} is the actual TB cases in month \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\widehat{{y}_{i}}$$\end{document}​ is the predicted TB cases in month \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document}. In this study, RMSE is used to measure the overall deviation of the model when predicting TB cases, MAE evaluates the average deviation, and MAPE measures the relative size of the error, representing the proportion of the error relative to the actual TB cases. Tools and libraries used Python 3.10 was used to build the models in this study. TensorFlow 2.15.0 was employed for constructing the self-attention and LSTM models, while the ARIMA model was built using the statsmodels 0.14.4 and pmdarima 2.0.4 libraries. The statistical significance level was set at 0.05. Results Optimal models Through experimental comparisons, the optimal configuration for the self-attention model was achieved with a time window of 12, 8 attention heads, 32 dimensions per head, a batch size of 64, and 280 training epochs. At this point, the loss curve exhibited a stable decline, reaching its minimum, which indicated the best-performing self-attention model. Similarly, for the LSTM model, the optimal configuration was found with a time window of 12, 16 neurons, a batch size of 64, and 280 training epochs. The loss curve for the LSTM model also showed a stable decline, reaching its minimum, signifying the optimal configuration. For the ARIMA model, the ARIMA (2, 1, 0) (0, 1, 1)12​ model had the lowest Akaike Information Criterion and Bayesian Information Criterion values. Additionally, the Ljung-Box Q test (P = 0.92 > 0.05) indicated that the residuals were white noise, identifying this as the optimal ARIMA model. Detailed optimization information and residual diagnostics for the three models are provided in Additional file 1 (Supplementary Tables 1 to 4 and Supplementary Fig. 1 to 7). Table 2 presents a comparison between the predicted and actual values on the test set for the three optimal models. Figure 7 illustrates the fitting and prediction curves of the three models. From the figure, it is evident that during the low TB period in January and February, the self-attention model’s predictions were closest to the actual values (with an MAE of 19 cases), followed by the LSTM model (MAE of 30 cases), while the ARIMA model showed a larger deviation (MAE of 39 cases). During the peak period from March to June, both the self-attention model and the LSTM model produced predictions relatively close to the actual values (MAE of 39 cases), whereas the ARIMA model’s error increased significantly (MAE of 66 cases). In the declining period from July to December, the self-attention model’s predictions remained highly consistent with the actual values (MAE of 23 cases), followed by the ARIMA model (MAE of 25 cases), while the LSTM model showed a slightly larger deviation (MAE of 26 cases).In summary, the self-attention model provided the most stable predictions across all time periods, while the LSTM and ARIMA models demonstrated greater fluctuations in performance. Table 2Comparison of predicted and actual monthly tuberculosis cases in Changde City, 2021MonthActualPredictedSelf-Attention(AE)LSTM(AE)ARIMA(AE)January357325(32) 331(26) 291(66)February291 285(6) 325(34)302(11)March411 398(13) 396(15)500(89)April375431(56) 368(7) 446(71)May381424(43)352(29) 405(24) June450 405(45) 347(103)371(79)July351425(74) 375(24) 408(57)August349340(9)328(21) 348(1) September355 349(6) 316(39)364(9)October298336(38)317(19) 284(14) November343 335(8) 318(25)299(44)December349 350(1) 320(29)326(24)Note: The comparison is based on predictions from three different models. AE represents the absolute error of the model’s prediction Table 2Comparison of predicted and actual monthly tuberculosis cases in Changde City, 2021MonthActualPredictedSelf-Attention(AE)LSTM(AE)ARIMA(AE)January357325(32) 331(26) 291(66)February291 285(6) 325(34)302(11)March411 398(13) 396(15)500(89)April375431(56) 368(7) 446(71)May381424(43)352(29) 405(24) June450 405(45) 347(103)371(79)July351425(74) 375(24) 408(57)August349340(9)328(21) 348(1) September355 349(6) 316(39)364(9)October298336(38)317(19) 284(14) November343 335(8) 318(25)299(44)December349 350(1) 320(29)326(24)Note: The comparison is based on predictions from three different models. AE represents the absolute error of the model’s prediction Comparison of predicted and actual monthly tuberculosis cases in Changde City, 2021 Note: The comparison is based on predictions from three different models. AE represents the absolute error of the model’s prediction Fig. 7Fitting and prediction plots of the three models (a: self-attention model, b: LSTM model, c: ARIMA model) Fitting and prediction plots of the three models (a: self-attention model, b: LSTM model, c: ARIMA model) Model performance evaluation Table 3 presents the performance of the three models on both the training and test sets. The results indicate that, On the training set, the self-attention model achieved an RMSE of 49.72, an MAE of 36.82, and a MAPE of 9.22%, outperforming the LSTM model (RMSE: 57.59, MAE: 44.49, MAPE: 11.28%) and the ARIMA model (RMSE: 62.73, MAE: 47.50, MAPE: 11.96%). On the test set, the self-attention model also demonstrated superior performance, with an RMSE of 35.74, an MAE of 27.54, and a MAPE of 7.58%, compared to the LSTM model (RMSE: 38.60, MAE: 30.94, MAPE: 8.41%) and the ARIMA model (RMSE: 50.23, MAE: 40.63, MAPE: 10.81%). Additionally, the 95% confidence intervals for the absolute error of the predictions on the test set are [12.42, 42.66] for the self-attention model, [15.62, 46.66] for the LSTM model, and [21.04, 60.23] for the ARIMA model. These results clearly indicate that the self-attention model provides the highest prediction accuracy, followed by the LSTM model, with the ARIMA model exhibiting the lowest accuracy. Figure 8 visually compares the performance of the three models on both the training and test sets, highlighting the superior performance of the self-attention model. Table 3Performance comparison of three models on training and test setsModelTrain setTest setRMSEMAEMAPERMSEMAEMAPEAE (95%CI)Self-Attention 49.72 36.82 9.22% 35.74 27.54 7.58% [12.42,42.66]LSTM57.5944.4911.28%38.6030.948.41%[15.62,46.66]ARIMA62.7347.5011.96%50.2340.6310.81%[21.04,60.23]Note: AE (95% CI) represents the absolute error with the 95% confidence interval Table 3Performance comparison of three models on training and test setsModelTrain setTest setRMSEMAEMAPERMSEMAEMAPEAE (95%CI)Self-Attention 49.72 36.82 9.22% 35.74 27.54 7.58% [12.42,42.66]LSTM57.5944.4911.28%38.6030.948.41%[15.62,46.66]ARIMA62.7347.5011.96%50.2340.6310.81%[21.04,60.23]Note: AE (95% CI) represents the absolute error with the 95% confidence interval Performance comparison of three models on training and test sets Note: AE (95% CI) represents the absolute error with the 95% confidence interval Fig. 8Performance comparison of the three models Performance comparison of the three models Interpretability analysis of the self-attention model The interpretability of the self-attention model is primarily achieved by analyzing the attention weight matrix. During each prediction, the self-attention layer generates an attention weight matrix, which represents how TB cases in each month of the input sequence influence TB cases in other months. Figure 9 illustrates the distribution of the attention weight matrix when predicting TB cases for January and February 2021. In the attention matrix, yellow-colored areas indicate greater attention from the model, while blue-colored areas indicate less attention. Specifically, Figure. 9a shows that when predicting TB cases for January 2021, the model primarily focused on data from January, February, and December 2020. Figure 9b reveals that when predicting TB cases for February 2021, the model placed more emphasis on data from February and December 2020, as well as January 2021. Notably, in both predictions, the model particularly focused on data from February 2020. This suggests that the model considers TB cases from February 2020 to have a significant impact on the predictions for both months. Fig. 9Attention matrices of the self-attention model for predicting cases: (a) January 2021, (b) February 2021 Attention matrices of the self-attention model for predicting cases: (a) January 2021, (b) February 2021 To further analyze the model’s attention allocation during predictions on the test set, we performed a longitudinal summation of the attention matrix to observe how the model focused on different months throughout the entire prediction process. Figure 10 illustrates the model’s attention distribution when predicting TB cases for the entire year of 2021. The results indicate that the model particularly focused on data from February 2020 when predicting TB cases for January and February 2021. During predictions from March to October, the model notably focused on data from February 2021. For the predictions in November and December, the model focused primarily on data from February and October 2021. These findings suggest that the model is effective at focusing on early case numbers in the sequence when making future predictions, thereby accurately capturing the long-term dependencies in TB case data. Additionally, Supplementary Figure. 8, included in Additional File 1, presents the model’s attention distribution on the training set. Fig. 10Attention of the self-attention model in predicting monthly tuberculosis cases in 2021 Attention of the self-attention model in predicting monthly tuberculosis cases in 2021 Generalization performance of the self-attention model To evaluate the self-attention model generalization capability, an external dataset was used for validation. This dataset was sourced from the Chinese Center for Disease Control and Prevention and includes monthly TB cases in Hunan Province from 2004 to 2020. Specifically, data from January 2004 to December 2018 was used as the training set, data from January to December 2019 as the validation set, and data from January to December 2020 as the test set. The self-attention model, LSTM model, and ARIMA model were reconstructed using the same procedures, and their performances were evaluated based on three metrics: RMSE, MAE, and MAPE. The model construction and prediction results are presented in Additional File 2. The results show that when predicting TB cases in Hunan Province from January to December 2020, the self-attention model achieved an RMSE of 433.27, an MAE of 274.66, and a MAPE of 6.72%, outperforming the ARIMA model (RMSE: 459.45, MAE: 359.95, MAPE: 8.34%) and the LSTM model (RMSE: 439.23, MAE: 311.06, MAPE: 7.23%). The optimal parameters for the self-attention model were a time window of 12, 8 attention heads, a dimension of 32 per head, a batch size of 64, and 300 epochs. These results indicate that the self-attention model demonstrates strong generalization ability, adapting stably to different regions and time series while maintaining high prediction accuracy. Discussion In this study, we applied three time series prediction models to forecast TB cases in Changde City, Hunan Province, China: the self-attention model, the LSTM model, and the ARIMA model. The results indicate that, on the test set, the RMSE of the self-attention model was reduced by approximately 7.41%, the MAE by around 10.99%, and the MAPE by about 9.87% compared to the LSTM model. When compared to the ARIMA model, the RMSE decreased by approximately 28.86%, the MAE by around 32.22%, and the MAPE by about 29.89%. In terms of specific models, the ARIMA model primarily analyzes the linear characteristics of TB data using statistical methods. However, the TB data also exhibit significant nonlinear characteristics, which the ARIMA model struggles to capture effectively, leading to its suboptimal performance in this study. This finding is consistent with [9], which highlighted that ARIMA performs poorly when dealing with complex nonlinear time series. The LSTM model, on the other hand, has an advantage in handling nonlinear dependencies and retaining useful historical information. However, when dealing with long sequences of data, it may gradually forget earlier information in the sequence [26]. In our predictions of TB cases, the LSTM model did not fully utilize the early data, limiting its prediction performance and reducing its accuracy. In contrast, the self-attention model, through the calculation of attention weights, can capture the potential impact of earlier TB data on later developments, resulting in superior prediction accuracy. This result is consistent with [39], which demonstrated that the Transformer model, based on the self-attention mechanism, outperformed both the ARIMA and LSTM models in predicting influenza incidence. To understand how the self-attention model predicts TB cases, we conducted an interpretability analysis. Figure 10 illustrates the specific attention patterns of the model across different time periods. When predicting TB cases for various months in 2021, the model primarily referenced data from February 2020, as well as February and October 2021. Notably, these months coincide with traditional Chinese holidays such as the Spring Festival and National Day, during which reported TB cases tend to be lower. Increased social mobility during these holidays may accelerate the spread of Mycobacterium tuberculosis, while reduced medical services lead to fewer reported cases. Considering that the average period from TB infection to diagnosis is 72 days (with a standard deviation of 28 days) [40], this explains the increase in confirmed cases in the months following these holidays [41]. Therefore, the model’s focus on data from these months likely aims to capture the delayed effects of holidays on TB incidence and reporting patterns. This demonstrates that the model can effectively allocate attention to time series data, enhancing its ability to capture long-term dependencies. In addition, we evaluated the generalizability of the self-attention model using an external dataset. The results showed that the self-attention model outperformed both the ARIMA and LSTM models on the external dataset, indicating that it can be effectively used to predict TB case numbers in different regions. While the self-attention model demonstrated strong predictive performance, it does have some limitations. First, the model’s high complexity results in long training times and significant resource demands. Second, increased complexity introduces a risk of overfitting, especially when the training data is limited or lacks diversity. Additionally, although this study provided an in-depth analysis of the time series data, it did not account for factors such as climate change, population migration, or improvements in healthcare facilities, all of which may also influence TB incidence. To address these limitations, future research will focus on reducing the risk of overfitting by increasing the diversity of the dataset. Additionally, future studies will aim to integrate factors like climate and environmental conditions to enhance the self-attention model, creating a more comprehensive and accurate prediction tool. Conclusion This study is the first to apply the self-attention mechanism to TB time series forecasting, providing an analysis of both the interpretability and generalization of the model. Compared to ARIMA and LSTM models, the self-attention mechanism can capture the influence of earlier data on future predictions, leading to improved prediction accuracy. This approach can help public health departments implement intervention measures earlier, improving the effectiveness of prevention and control efforts. Additionally, this study opens new possibilities for applying self-attention models to the prediction of other infectious diseases, potentially improving the accuracy of forecasting for other infectious diseases as well. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary Material 1: Table S1. Comparison of Self-Attention Models with Various Parameters. Table S2. Comparison of LSTM Models with Various Parameters. Table S3. Comparison of ARIMA Models with Various Parameters. Table S4. Parameter Test Results for the ARIMA (2,1,0)(0,1,1)12 Model. Figure S1. Training Loss Curves of Self-Attention Models with Various Parameters. Figure S2. Residual Diagnostic Plots for the Optimal Self-Attention Model. Figure S3. Training Loss Curves of LSTM Models with Various Parameters. Figure S4. Residual Diagnostic Plots for the Optimal LSTM Model. Figure S5. Tuberculosis Time Series in Changde City from 2010 to 2020 after First-Order and Seasonal Differencing. Figure S6. ACF Plot (a) and PACF Plot (b) of the Differenced Series. Figure S7. Residual Diagnostic Plots for the Optimal ARIMA Model. Figure S8. Attention of the Self-Attention Model on the Training Set. Additional File 2: Validation of Monthly Tuberculosis Data in Hunan Province. Additional File 3: Code for model building. Supplementary Material 1: Table S1. Comparison of Self-Attention Models with Various Parameters. Table S2. Comparison of LSTM Models with Various Parameters. Table S3. Comparison of ARIMA Models with Various Parameters. Table S4. Parameter Test Results for the ARIMA (2,1,0)(0,1,1)12 Model. Figure S1. Training Loss Curves of Self-Attention Models with Various Parameters. Figure S2. Residual Diagnostic Plots for the Optimal Self-Attention Model. Figure S3. Training Loss Curves of LSTM Models with Various Parameters. Figure S4. Residual Diagnostic Plots for the Optimal LSTM Model. Figure S5. Tuberculosis Time Series in Changde City from 2010 to 2020 after First-Order and Seasonal Differencing. Figure S6. ACF Plot (a) and PACF Plot (b) of the Differenced Series. Figure S7. Residual Diagnostic Plots for the Optimal ARIMA Model. Figure S8. Attention of the Self-Attention Model on the Training Set. Additional File 2: Validation of Monthly Tuberculosis Data in Hunan Province. Additional File 3: Code for model building.