Random Attention Mechanism for Deep Quality Modeling and Online Prediction in Chemical Processes Authors: Xie Ma, Lingjian Ye, Zhiqiang Ge Published: 2026 ABSTRACT Many deep learning methods have recently been introduced for quality prediction in industrial processes, among which the deep latent variable model should be one of the most promising methods due to its simple and explainable model structure. However, how to efficiently extract features from the latent space for quality prediction remains unclear. This paper develops a random attention mechanism upon the basic deep latent variable regression modeling framework. After the latent variables have been extracted from different layers of the deep model, various random attentions are made from the pool of latent variables, based on which different submodels are constructed for the same quality prediction purpose. Then, an information fusion strategy is designed to combine the quality prediction results of different submodels. Results on a debutanizer column and a CO2 absorption column show that the proposed random attention mechanism can reduce the average prediction index (RMSE) by 18% and 31%, respectively, compared to the regular methods. Introduction To ensure high product quality in industrial processes, effective closed-loop control systems are typically required for those quality-related variables. In practice, however, continuous measurements of quality variables are often very difficult due to limitations in instrumentation technologies, measurement environments, investment budget, and so on. As a result, successful closed-loop quality control systems are quite limited in practical industrial applications. To serve as a backbone in quality control systems, online measurements of quality-related variables are highly desirable. With the advent of both artificial intelligence and data science, more and more data-driven modeling methods have been developed and introduced for the purpose of quality prediction in industrial processes. − Compared to traditional hardware-based instrumentation tools, data-driven quality prediction methods provide software-based alternatives to continuously estimate quality variables from other related process variables, which are much easier to measure in the industrial plant. − In the past years, some intensively studied and widely applied quality prediction models include principal component regression (PCR), partial least-squares (PLS), artificial neural networks, support vector machines, Gaussian process regression, and so on. − More recently, deep learning techniques have also been introduced for industrial quality modeling and prediction, including stacked autoencoders, convolutional neural networks, recurrent neural networks, and long–short memory networks, to name a few. − Among those well-developed quality prediction models, the most useful ones in practical industrial applications are probably those simple models such as PCR and PLS, due to their transparent model structures, being well-fitted to the nature of industrial data, and being easy to integrate into quality control and optimization systems. However, subjected to their model capacities, the quality prediction performances of those simple models may be restricted to a limited space, particularly compared to deep-learning-based methods. Fortunately, some basic latent variables such as PCA and PLS have recently extended to deep forms, with the inspiration of deep learning ideas, i.e., layer-by-layer modeling, in-model information transformation, and more complex model structures. − For example, the deep form of the PCA model has obtained comparative performance for industrial process monitoring to those traditional deep learning methods such as deep autoencoders; however, much less computational burden is required and there is no cumbersome parameter tuning procedures. , A lightweight and highly efficient deep PLS model has been formulated for the purpose of regression modeling such as quality prediction. According to the results from detailed industrial case studies, the effectiveness of the deep PLS model has been well confirmed, such as prediction accuracy, computational efficiency, and applicability under small training data sets. While those extracted latent features from different layers of the deep model can be well utilized and combined together to serve as new secondary variables for downstream tasks such as quality prediction, designing more effective attention mechanisms may have significant impacts on the deep model, similar to the advancements seen in traditional deep learning application areas. The motivation of this paper is to propose a random attention mechanism for deep latent variable models, taking the basic PCR model as an example. Aiming at providing a more effective industrial quality prediction model, the deep residual learning form of the PCR model is first constructed through layer-by-layer training and latent feature extraction procedures. Then, on the basis of the latent components extracted from different layers, a random attention mechanism is designed. To do this, various additional PCR models are developed by randomly selecting latent variables from the whole latent component pool to serve as input variables for each PCR model. For the purpose of online quality prediction, the results of various PCR models driven by different random attention mechanisms can be fused together to provide a unified prediction value for quality variables. The remainder of this paper is organized as follows. The detailed methodology of the random attention mechanism-driven deep PCR model is illustrated in Section , including the construction of the deep residual PCR model, the design of the random attention mechanism, and the online quality prediction scheme. In Section , two industrial case studies are provided for performance evaluation and comparative studies among different methods. Finally, conclusions were made. Random Attention Mechanism for Deep Quality Prediction In this section, the methodology of the deep PCR model with a random attention mechanism is illustrated in detail, including the development of the deep latent variable model, the formulation of the random attention mechanism-driven quality prediction model, and the online application procedures. Deep Learning of PCR Model As a classical latent variable regression model, PCR has been widely used in various industrial application scenarios. For example, PCR often provides as a software algorithm for continuous sensing some key variables in chemical processes, makes early predictions of final product quality in batch processes, or serves as virtual instruments in measuring critical parameters in semiconductor manufacturing systems. − Besides, PCR is also a quite popular chemometrics method, which has been intensively studied in the area of multivariate spectroscopic calibration modeling and data analytics. − There are two main steps in the PCR modeling process: (1) latent variable or principal component extraction through the PCA algorithm; (2) least squares regression between latent variables and the target variables (quality variables). In the first step, the use of the PCA algorithm is to find those latent variables that can retain the most amount of data information from the training data set. Given the data set of process variables as X ∈ R n× m , where n is the number of data samples, m is the number of measured process variables, the decomposition of the PCA algorithm is as follows: 1 X=TPT+E where T ∈ Rn × k is latent variable matrix or called score matrix; K is the number of retained latent variables, which usually have the largest eigenvalues through the PCA decomposition process, P ∈ Rm × k is the loading matrix, and E ∈ Rn ×m is the residual data set. In the second step of the PCR model, a simple least-squares regression is implemented between the extracted latent variables and the quality variables Y ∈ Rn ×r, where r is the number of quality variables. The whole PCR model structure is given as 2 X=TPT+E⇒[Rpcr=(TTT)−1TTY=(PTXTXP)−1PTXTY] To formulate the deep learning form of the basic PCR model, a layer-by-layer residual learning strategy can be introduced here. First, an ordinary PCR model is developed as shown in eq . For the purpose of description consistency, this first layer-wise PCR model is rewritten as follows: 3 X=T(1)P(1)T+E(1)⇒[Rdpcr(1)=(T(1)TT(1))−1T(1)TY=(P(1)TXTXP(1))−1P(1)TXTY] Starting from the second layer of the deep model, the residual data information from the last layer-wise PCR model is served as the input data set for developing the layer-wise PCR model in the current layers. Here, if we suppose that a total of L layers are used to formulate the deep learning model, the general layer-wise PCR model can be described as follows 4 E(l−1)=T(l)P(l)T+E(l)⇒[Rdpcr(l)=(T(l)TT(l))−1T(l)TY=(P(l)TE(l−1)TE(l−1)P(l))−1P(l)TE(l−1)TY] where l = 2,3,···,L. As a result, various patterns of latent variables have been extracted in different layers of the deep model. Compared to the shallow PCR model, much more inherent features are able to be extracted from the original data set, which are related to the quality variables. A simple flowchart of the deep PCR modeling process is depicted in Figure . Deep residual learning structure of PCR model. Quality Modeling with Random Attention Mechanism Based on the deep learning structure of the PCR model, different layer-wise models have extracted different patterns of latent variables, which may have quite different contributions in predicting the quality variables. In order to make full use of those extracted latent variables, the idea is to construct a regression model between the whole latent variables and the quality variables. For example, an additional PCR model can be developed for the purpose of final quality prediction, given as 5 Twhole={[T(1)][T(2)][T(3)]⋯[T(L)]}↓Twhole=T(w)P(w)T+E(w)⇒[R(w)=(T(w)TT(w))−1T(w)TY=(P(w)TTwholeTTwholeP(w))−1P(w)TTwholeTY] In order to improve the quality prediction performance, a random attention mechanism is proposed in this section. To do this, a subset of latent variables is randomly selected from the whole latent variable set T whole , based on which a sub-PCR model can be constructed between this subset and the quality variables, given as follows 6 rand{Twhole}⇒{Tcsub}c=1,2,⋯,C↓Tcsub=Tc(s)Pc(s)T+Ec(s)⇒[Rc(s)=(Tc(s)TT(w))−1Tc(s)TY=(Pc(s)TTcsubTTcsubPc(s))−1Pc(s)TTcsubTY] where C is the total number of sub-PCR models, for each of which the input features are randomly selected from the whole latent variable set T whole . By randomly putting attentions on some particular latent variables and repeating it for many times, the potential ability of the regression model can be fully explored. Consequently, the performance of the final quality prediction model will be probably improved to a certain extent. Online Quality Prediction Scheme Given a new data sample X new obtain from the industrial process, the first step of the online quality prediction scheme is to extract the latent variables from different layers of the deep model, which are calculated as follows 7 {tnew(1)=P(1)Txnewtnew(2)=P(2)T[I−P(1)P(1)T]xnewtnew(3)=P(3)T[I−P(2)P(2)T][I−P(1)P(1)T]xnew⋮tnew(L)=P(L)T∏l=1L−1[I−P(l)P(l)T]xnew}⇒[tnew(1)tnew(2)tnew(3)⋮tnew(L)]=tnewwhole Then, those sub-PCR models developed through the random attention mechanism are used to predict the values of quality variables, given as 8 {ŷ1,newsub=R1(s)t1,new(s)=R1(s)[tnewwhole(index1)]ŷ2,newsub=R2(s)t2,new(s)=R2(s)[tnewwhole(index2)]ŷ3,newsub=R3(s)t3,new(s)=R3(s)[tnewwhole(index3)]⋮ŷC,newsub=RC(s)tC,new(s)=RC(s)[tnewwhole(indexC)]} where indexc (c = 1,2,···,C) is index vector of each randomly selected subset of latent variables in the whole latent variable set T whole , which has been used for the development of each corresponding sub-PCR model in the training stage. In order to provide a final quality prediction result for the new data sample X new , those prediction values of quality variables by different sub-PCR models need to be combined or fused together through an appropriate way. A general idea is to make a simple averaging implementation, provided as follows 9 ŷnew=1C∑c=1Cŷc,newsub It is noted that we have only used the simplest information fusion strategy to demonstrate the idea of an online quality prediction scheme driven by the random attention mechanism, which means other more complicated and maybe more effective fusion algorithms can also be employed. However, we leave those straightforward extensions for possible future work, which may further improve the quality prediction performance of the proposed method, as shown in Figure . Online quality prediction based on deep PCR model with random attention mechanism. Industrial Case Studies In this section, two industrial examples are employed for performance evaluation of the developed method, and the rooted mean square error (RMSE) criterion is used as the performance index, which is given as follows 10 RMSE=∑i=1N∥yi−ŷi∥2Nte where y i and ŷ i are real and predicted values of the quality variables, respectively, Nte is the total number of new data samples in the testing data set. Debutanizer Column In the past two decades, the debutanizer column has been widely used for performance evaluations of various soft-sensing and quality-prediction algorithms, which is actually a key part of the desulfurizing and naphtha splitter plant. A simple illustration of the debutanizer column is shown in Figure . To carry out quality control in this column, real-time measurement of the butane content in the production process is an important step, which can be considered a quality variable. However, compared to other normally recorded process variables such as temperatures, pressures, and steam flows, the butane content is much more difficult to measure online. In order to construct an effective quality control system for this column, the content of butane in the process must be predicted online. A total of seven related process variables are used as the secondary variables for the development of the quality prediction model, detailed information on which can be found in Table . Flowchart of the debutanizer column. 1Process Variables in the Debutanizer ColumnProcess variablesDescriptionsx1Top temperaturex2Top pressurex3Reflux flowx4Flow to next processx5sixth tray temperaturex6Bottom temperaturex7Bottom temperature Process Variables in the Debutanizer Column To formulate the random attention mechanism-based deep PCR model, a training data set that contains 500 data samples has been collected in the column. For performance evaluation, another testing data set, which contains 1500 data samples, has also been collected. For fair comparisons, the basic deep PCR model without the random attention mechanism is also developed for quality prediction in this process. For both of the two deep latent variable models, the number of retained latent variables in each single PCR model is determined through the CVP method, which denotes the ratio of explained variances to the retained latent variables. The numbers of hidden layers in both two deep models are initially set as 5 in the simulation studies, and for the random attention mechanism, a total of 20 random seeds are used for the generation of the sub-PCR models. First, the prediction results of the butane content in the column by the two deep learning models are presented in Figure , in which one can see that the prediction accuracy of the quality variable has been significantly improved with the introduction of the random attention mechanism. To explore more detailed information on the sub-PCR models under different random seeds, the RMSE values of those 20 sub-PCR models are shown in Figure , together with the final prediction model. As can be seen from this figure, the performance of the final quality prediction model is better than most sub-PCR models, which means that the step of information fusion across different random attentions on the latent variables should be effective to improve the prediction performance. Based on the multiple simulation studies, the average RMSE values of the basic deep PCR model and the deep PCR model with random attention mechanism are 0.2274 and 0.1856, respectively. As a result, the RMSE value has been reduced by 18.38%. Quality prediction results of the testing data set, (a) deep PCR; (b) deep PCR with RAM. RMSE values of different sub-PCR models compared to the final prediction model. To examine the sensitivities of the quality prediction models under different parameter settings, the results of the two deep-learning models under different numbers of hidden layers are presented in Figure . As can be seen from this figure, the performance was significantly improved with the introduction of the random attention mechanism under different settings of the hidden layers. Figure investigates different numbers of random seeds used in the proposed method, where one observes that the quality prediction performance becomes smoother when more than 10 random seeds are selected in the information fusion step. When more and more sub-PCRs are generated with different random attention seeds, more considerations need to be paid on the information fusion step, which also means that more competitive strategies may be designed to further improve the prediction performance. However, at the same time, the computational burden of the online quality prediction model should be increased, which may restrict the real-time performance of the model. Because the sub-PCR models are inherently lightweight, the inference latency remains on the order of milliseconds even with 35 or more submodels, making it highly feasible for real-time online process modeling and prediction. RMSE values of the two deep learning models under different numbers of hidden layers. RMSE values of the RAM-based deep PCR model under different numbers of random seeds. Quality Prediction in CO2 Absorption Column The CO2 absorption column is a part of the ammonia synthesis production plant, the simple flowchart of which is shown in Figure . In this column, an important quality variable is the concentration of residual CO2 in the process gas, which can be found in the top-right corner of Figure . A simple flowchart of the CO2 absorption column. Compared with other process variables that can be routinely recorded from the column, the concentration of CO2 is relatively difficult and expensive to measure online. In order to provide timely estimation values for the concentration of CO2 in the process gas, a data-driven quality prediction model can be developed by using those easy-to-measure process variables. On the basis of process knowledge and correlation analyses, a total of 11 process variables are selected as the input variables of the predictive model. Detailed information on those selected process variables is listed in Table , including liquid levels, flows, pressures, and temperatures. 2Descriptions of Process VariablesVariablesDescriptionsU1Pressure of process gas into E3U2Liquid level of Separator 2U3Temperature of barren liquor at E1 exitU4Flow rate of barren liquor to CO2 Absorption ColumnU5Flow rate of half deficient liquor to CO2 Absorption ColumnU6Temperature of process gas at exit of Separator 2U7Differential pressure of process gas at entrance of CO2 Absorption ColumnU8Temperature of rich liquor at exit of CO2 Absorption ColumnU9Liquid level of CO2 Absorption ColumnU10High level alarming of Separator 1U11Pressure of process gas at the exitYConcentration of residual CO2 in process gas Descriptions of Process Variables For the construction of the quality prediction model, a total of 5000 data samples have been collected as the training data set. Similarly, the numbers of hidden layers in both of the two deep latent variable models are set as 5, and the initial number of random seeds is set as 20 in the attention mechanism. For performance evaluation and comparative studies between the basic deep PCR model and the one with the random attention mechanism, a testing data set, which also consists of 5000 data samples, is obtained through the process. Detailed prediction results of both deep models are shown in Figure , in which one can see very clearly that the online prediction accuracy for the quality variable has been greatly improved by the introduction of the random attention mechanism. Again, the average RMSE values of the basic deep PCR model and the deep PCR model with a random attention mechanism from multiple simulation studies are 0.0089 and 0.0061, respectively. Therefore, the reduction of the RMSE value in this case was 31.46%. Quality prediction results of the testing data set, (a) deep PCR; (b) deep PCR with RAM. To examine more detailed results of different sub-PCR models, Figure provides the value of RMSE for each of the 20 sub-PCR models, together with that of the final prediction model, which is highlighted by the red line. As can be seen from this figure, based on the information fusion process, the prediction performance has been significantly boosted, i.e., the final result is better than all of those results of 20 sub-PCR models. RMSE values of different sub-PCR models compared to the final prediction model. In order to explore how many random attention we really need to obtain a prediction result, different numbers of random seeds are evaluated through both training and testing stages, detailed RMSE values of which are demonstrated in Figure . In this case, the best number and random seeds should be around 35, under which the smallest value of RMSE was obtained. However, this is probably subject to the nature of data samples, size of latent variable subset, and the strategy of information fusion. Finally, the sensitivities of the number of layers in both of the two deep models are evaluated under different settings, detailed results of which are illustrated in Figure . As can be expected, the deep model with the introduction of the random attention mechanism has obtained much better results in all setting cases, while the best prediction performance has been determined when the number of layers is set as 3 or 4 in this case study. RMSE values of the RAM-based deep PCR model under different numbers of random seeds. RMSE values of the two deep learning models under different numbers of hidden layers. Conclusions In this paper, a random attention mechanism was introduced into the deep learning framework of the basic latent variable regression model. Based on the latent variable pool obtained from the layer-wise feature extraction step, various random attentions have been made to generate different sub-PCR models for the same quality prediction purpose. Through information fusion among different random attention-driven regression models, the quality prediction performance has been well boosted. On the basis of those results provided by the two industrial examples, both feasibility and effectiveness of the random attention mechanism have been confirmed under the deep latent variable modeling framework. It is worth noting that only the very basic form of the model has been presented in this paper, which means some extensions can be made to further enhance the quality prediction performance, such as heterogeneous latent variable model structures and different formulations of the random attention mechanism. Another interesting issue is to learn the structure of the deep latent variable model automatically. Although several structure learning algorithms have already been proposed, , how to effectively integrate the random attention mechanism into the structure learning process may require further investigation. Besides, we would like to highlight that more advanced information fusion strategies such as weighted averaging (based on submodel training variance) and stacking (using a meta-learner) could push the prediction boundaries even further for the proposed method. To further validate the generalizability of the proposed method in batch industrial processes, this deep PCR-based random attention learning framework can be adapted by utilizing multiway unfolding techniques prior to deep feature extraction. In this way, the three-dimensional batch process data set can be reformulated to fit the modeling structure of the proposed method, based on which the extracted deep latent features can be unfolded and fused together for conducting the random attention mechanism. In the online batch prediction stage, each time-slice data sample from the batch process will be applied to determine the result of target variables, supported by an ensemble form of the random attention-based deep latent variable models.