Convergence analysis and application for high-order neural networks based on gradient descent learning algorithm via smooth regularization Authors: Khidir Shaib Mohamed, Alawia Adam, Yousif Shoaib Mohammed, Yan Xiong Published: 2025 ABSTRACT AbstractThe pi-sigma network (PSN), as a high-order network, has demonstrated its capacity for rapid learning and strong nonlinear processing. This paper proposes a type algorithm, for PSNs using a batch gradient method based on regularization. Direct application of regularization during network training presents two main drawbacks. There are numerical oscillations and theoretical challenges in computing the gradients at the origin. We then introduced smoothing functions by approximating the regularization to overcome these obstacles, resulting in a new gradient descent method based on smoothing regularization (GDS). Numerical results for the 4-dimensional parity problem and the nonlinear Gabor function problem demonstrate that the GDS algorithms perform better than the other four regularization methods in terms of generalization and pruning efficiency. Theoretical analysis and experimental verification strictly prove the monotonicity and strong and weak convergence of the network based on the GDS algorithms.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-29494-1.Keywords: Pi-sigma network, Gradient descent algorithm, Smoothing regularization, Convergence, Numerical resultsSubject terms: Computational science, Computer science Introduction Higher order networks (HONs) extend the traditional pairwise interactions between nodes to higher order interactions involving three or more nodes. This allows for the modeling of more complex relationships and dynamics within the network. Various fields such as dynamical systems, adaptation, and self-organizing systems have studied HONs1–3. Applications of HONs include modeling the spread of disease, understanding the structure of complex systems, and predicting the behavior of social networks4–6. A pi-sigma network (PSN) is an effective HON that can be used for pattern classification and function approximation. It retains the strong learning ability of a multilayer HON while mitigating the exponential growth in the number of weights and hidden units that occurs when the dimension of the input vectors increases. This advantage is discussed in references7–9. Gradient descent (GD) algorithm is a method for unconstrained mathematical optimization. First suggested in 184710, it is a first-order iterative algorithm for finding the local minimum of a differentiable multivariate function. In 1907, researchers proposed a similar method11. Researchers first studied its convergence properties for non-linear optimization problems in 194412, and continued to study and use the method in the following decades13. Two methods exist for training the weights, referred to as batch training and online training [14,15]. In the primary technique, the network modifies the weights after each training pattern is presented. In the second method, aggregate the cumulative error across an epoch and modify the weights subsequent to the presentation of the entire training set. This study adheres to the batch training update methodology. Overfitting is a prevalent issue in machine learning wherein a model grows excessively complicated and begins to accommodate the noise in the training data instead of the fundamental patterns [16]. In literature, numerous strategies are employed to mitigate the issue of overfitting, which can be summarized as follows: regularization17–19, dropout18, early halting20, and cross-validation21. Regularization is an effective method to mitigate overfitting in machine learning models. Regularization is a technique that incorporates a penalty term into a model’s loss function to deter excessive weights or overly complex models. Overfitting is a prevalent issue in machine learning and is hence of significance. Overfitting is the phenomenon where an analysis is excessively tailored to fit a specific set of data, resulting in a lack of generalizability. Overfitting occurs when a neural network excessively conforms to the training data, resulting in a diminished capacity to accurately predict fresh data. There are multiple strategies to mitigate the problem of overfitting in neural networks. A term proportional to the norm of the weight vectors is a fundamental penalty incorporated into the standard cost function; this method, known as regularization, is commonly utilized in parameter estimation and has recently been regarded as a viable approach to the variable selection challenge in machine learning. regularization, namely, assumes the following form23: where is the standard error function depending on the weights and and is regularization parameter. Let be a real number. The -norm (also called -norm) of vector is , If , we have , then this is regularized logistic regression. If , we have , this is regularized logistic regression. The purpose of network learning is to obtain a weight vector such that ). The weight vectors corresponding iteration by where is the learning rate. The regularization techniques developed recently are excellent for sparse network architectures. Although it is an NP hard issue, regularization is the most logical form of sparse optimization24. Put another way, regularization cannot be used directly by an optimization algorithm25. regularization is added to the network in PSN to lessen the redundancy of hidden layer neurons26. As a recent idea to encourage sparsity during training, smoothing regularization has been put forth27–29. By using a penalty term, regularization promotes tiny sums of absolute values for the parameters. The Lasso algorithm30 is what we term regularization, and it usually yields sparse feature vectors. We find another example of regularization learning in31. The absolute value of the coefficient is included as a penalty term in the regularization. The regularization proposed in32 is easier to solve, although being less sparse. Additionally, regularization for federated learning was suggested by the authors in33. This method is also applied in electrical impedance tomography34. The difficulties in theoretical research and potential for poor convergence in computing application stem from the difficulties in regularization and the inability to directly apply the gradient approach. A innovative approach to pruning FNNs was developed in22 by modifying the standard regularization term using a smoothing technique. In the sparsity effect, the regularization performs well35. However, oscillation will occur during the iterative numerical test process because this regular term involves an absolute value function that is not differentiable at the origin. Nonetheless, regularization smoothing is used36–39. Weight decay, sometimes referred to as regularization, is a popular regularization technique that optimizes networks without producing a sparse effect40–44. In this work, a gradient algorithm with smoothing regularization (GDS) for the PSN is proposed. The innovation in this work lies in the utilization of GS. As illustrated through numerical simulations (“Discussion”) section, GDS exhibits superior sparsity promotion compared to GD and GD in pruned weights. We mainly focus on two challenging, since the general regularization is no differentiable at the original point, this causes oscillation in the computation and difficulty in the convergence analysis. To overcome these challenges, we suggest replacing and approximating the non-differentiable function with a smooth function, resulting in a smoothing regularization method for PSN. According to22, the smoothing regularization method is also better at creating a sparse feedforward neural network structure than , , , and regularization methods, and even than the smoothing regularization method. The results of this study show that smoothing regularization accelerates convergence, enhances network structure, and creates stronger mapping. To achieve these ends and the benefits of our proposed algorithm, we shall use the smoothing regularization for the network regularization. We organize the remaining portions of this brief as follows: In “Pi-Sigma network” section, we give a literature review. In “Gradient descent algorithm based smoothing regularization (GDS ) for training PSN” section, we have provided an explanation of the pi-sigma network. In “Main results” section, we briefly describe a neural network model with the batch gradient method and smoothing regularization. “Numerical results” section presents the main convergence theorem. “Discussion” section provides a numerical example to bolster the convergence result. We summarize the findings and conclusions in “Conclusion” section. We have relegated the proof of the theorem to the Appendix. Pi-Sigma network Figure 1 shows a PSN with a single output. This is a fully connected, two-layer feedforward network. However, unlike the multilayered perceptron (MLP), the summing layer is not "hidden," as the weights from this layer to the outputs remain fixed at 1. This property drastically reduces training time. Let n, m, and 1 be the number of neurons in the input layer, summation layer, and quadrature layer of the PSN, respectively. Assume that is a given nonlinear activation function for the output unit, which is often, but not necessarily, selected as the logistic function. A pi-sigma network with structure. We denote by the weight vectors connecting the input and summing units, and write . In particular, for an input vector , the network’s final output is where is the internal product of and . Gradient descent algorithm based smoothing regularization (GDS) for training PSN Let the stander error function for the training sample . The network is supplied with a given set of training examples , then the standard error function can be defined by where . A very often used method to handle this kind of problem is the gradient method. Thus, the gradient of with respect to and the corresponding iteration formula for the weight vectors is written as and where is the learning rate, and . For non-convex loss functions, the gradient method can sometimes become trapped in local minima, leading to overfitting. This implies that complex models perform poorly in terms of generalization. Networks use regularization as a technique to prevent overfitting by adding a penalty to the loss function. The goal is to find a balance between fitting the training data and avoiding complex models that generalize poorly. To achieve this, we can generally incorporate an regularization into the standard error function (GD) in Eq. (4). We can then represent the new cost error function as follows: The function represents the standard error function, which is dependent on the weights . The value is the regularization parameter. The norm of the weights , denoted as , is given by the expression . Given that regularization lacks differentiability, our initial focus is on exploring optimization techniques that approach the non-differentiable objective as a non-smooth optimization issue. This involves utilizing sub-gradients of the objective function at locations where differentiability is absent. In the context of non-smooth optimization, a parameter vector can be considered a local minimum if and only if the zero-vector 0 is an element of the sub-differential . The sub-differential is a set that contains all the sub-gradients at 45. The sub-differential of the absolute value function is equal to the signum function . In optimization and statistical learning theory, the proximal gradient descent method (PGD) for learning is a field of study that examines methods for a broad class of convex regularization problems in which the regularization penalty might not be differentiable. Statistical learning theory regularization problems can be solved using PGD approaches, which provide a broad framework with penalties customized for a certain problem application46,47. regularization (sometimes called Lasso) of the form Eq. (7) is one such instance. We simplify the solution for the proximity operator in Eq. (7) by having , as is well known. This makes it simple to calculate , which is exactly Then, the proximal gradient descent method for regularization (PGD ) is given by In this paper, we consider to address the aforementioned non-derivative flaw in Eq. (7). First, provide a clear and concise definition of the smoothing function where smoothing parameter, and then, the derivate of with respect to vector as follows: Thus, the gradient descent method for smoothing regularization (GDS) is Thus, the gradient of in Eq. (12) with respect to and the corresponding iteration formula for the weight vectors is writ as and where is the learning rate, and is the regularization parameter. Main results Some propositions are needed for the convergent result. (C.1): The functions , , . (C.2): , , , and . (C.3): The learning rate and are the regularization parameter holds to satisfy: , where which defined below in Eq. (A.7). (C.4): There exists a closed bounded region such that the weight sequence , and the set contains only finite number of points. Remark 5.1 Activation functions commonly used in neural networks, like the function , fulfill condition (C.1). To ensure the weak convergence below, we need to have condition (C.2) in place. This is because boundedness in the learning iteration process implies (weak) convergence. The condition in condition (C.3) is not overly limiting, as the penalty parameter is typically expected to be both positive and small in practical applications. condition (C.3) is a common requirement for the learning rate to ensure the convergence of a batch gradient method. It is also necessary for the learning rate to be positive and constant. condition (C.3) guarantees that the error function will have a finite number of local minimums, which leads to strong convergence. It is straightforward to confirm that the absolute values of , and are all uniformly bounded by for all , as shown in (C.1) and (C.2). We will often refer back to these observations in our proofs. Theorem 5.1 Define the error function as specified in Eq. (12) and construct the weight sequence using the iterative procedure described in Eq. (14), starting with any beginning value. Given that conditions (C.1)–(C.3) are satisfied, the following inference can be drawn: for all . for all . There exists such that . There exists such that . for all . Moreover, if Assumption (A4) is also satisfied, the following strong convergence result holds: for all . Moreover, if Assumption (A4) is also satisfied, the following strong convergence result holds: There exists a point such that . There exists a point such that . Remark 5.2 The conclusions (a) and (b) demonstrate the monotonicity and limit of the error function sequence {}. The conclusions (c) demonstrate the weak convergence of and . The Conclusion (c) provides the precise information regarding the strong convergence of {}. Remark 5.3 From the point of view of mathematical theory, we obtain the convergence results with a constant learning rate for batch mode. Specifically, we will use the result, which is essentially the same as Theorem 14.1.5 in49, for the demonstration of the convergence results. However, we extend the convergence result including the strong convergence result as well in this paper in the Appendix. Note: The simulations support the convergence assertion made in “Discussion” section as well. To verify the validity of the proposal, the calculation procedures for three logarithms were carried out using the same input information based on Algorithm 1 for all three numerical experiments. Table 1 shows the learning parameters, and Table 2 and Table 3 shown the samples of the 4-dimensional and the 5-dimensional parity problems. Table 1The numbers of parameters.Learning methodsLearning parameters4-dimensional parity5-dimensional parityHigh-order Gaussian2-dimensional Gabor functionGD GD PGD 0.050.040.070.5 0.0010.0020.0030.0001GDS Network structure4-5-15-6-12-10-12-5-1Weight size [− 0.3,0.3] Max iteration1000120010004000 The numbers of parameters. Table 2Samples of 4-dimensional parity problem.InputOutputInputOutput11110− 11− 1− 11− 1111111− 1− 10111− 111− 1111− 111− 10− 1− 1110− 1− 1− 1111− 1− 1101− 11− 10− 1− 11− 11− 1− 1− 1− 1011− 1111− 1− 1− 11− 11− 110 Samples of 4-dimensional parity problem. Table 3Samples of 5-dimensional parity problem.InputOutputInputOutput− 11111− 1011111− 111111− 1− 10− 1111− 1− 111− 1− 1− 11− 10− 1− 1− 1− 11− 11− 1− 1− 1− 1− 1− 101− 1− 1− 1− 1− 11− 1− 11− 1− 1− 111− 11− 1− 1− 10− 11− 111− 1111− 111− 1011− 1− 11− 11− 11− 1− 11− 101− 1− 1− 11− 10− 1− 1− 1− 11− 11− 1− 1111− 111− 1111− 101− 111− 1− 11− 1− 111− 1− 10− 11− 1− 11− 1011− 1− 11− 1111− 1− 1− 1− 10− 11− 1− 1− 1− 11− 111− 1− 1− 10111− 1− 1− 111− 1− 111− 11− 1− 1− 111− 101− 1− 1− 11− 10− 1− 1− 1− 11− 1111− 1− 11− 11− 11− 1− 11− 10 Samples of 5-dimensional parity problem. GDS Numerical results To illustrate the convergence of gradient descent with batch update rule for GDS, numerical experiments are carried out for a function approximation problem and 4-dimational parity problem. We compare GDS method with two other regularization methods based on PSN: the gradient descent with batch update rule for regularization (GD), the gradient descent with batch update rule for regularization (GD), and the proximal gradient descent with batch update rule for regularization (PGD) methods. K-dimensional parity problems The N-dimensional parity problem involves a challenging classification task with input patterns in an K-dimensional vector space. A parity problem consists of many boolean input variables and one boolean output variable. The input/output rule states that the output should be true 1 if an odd number of the input values are true. Since the rule essentially allows either of the inputs to be true 1, we refer to the problem as "Exclusive-OR" (or XOR https://en.wikipedia.org/wiki/XOR_gate) when there are only two input variables. It is well known that parity issues are statistically neutral48. A parity bit, sometimes referred to as a check bit, is added to a string using binary code. In this example, we use 4- and 5- dimensional parity problems to illustrate the effectiveness of the proposed method. Table 1 shows the models parameters. Table 2 displays the inputs and the desired outputs of the training samples for the 4- dimensional parity problem, as applied in49. In22, Table 3 presents the inputs and desired outputs of the training samples for the 5- dimensional parity problem (Figs. 2, 3, 4). Four-dimensional parity problem: value of the error function with different algorithms (GD , GD , PGD , GDS). Four-dimensional parity problem: norm of gradient with different algorithms (GD , GD , PGD , GDS). Five-dimensional parity problem: value of the error function with different algorithms (GD , GD , PGD , GDS). The error functions for GDS, PGD , GD , and GD for the 4 and 5 dimensional parity issues decrease steadily, as seen in Figs. 2 and 4. According to Theorem 5.1, the gradient norms of the error functions approach tiny positive constants for both 4 dimensional and 5 dimensional parity situations. The Figs. 3 and 5 illustrate this. GDS has a lower error than GD , and GD with the same training parameters. The preponderance of sparseness and the chosen average of the smallest of all weights show that GDS has a much lower prevalence of sparseness than PGD , GD , and GD . Our GDS method actively reduces the size of weights. These findings also support Theorem 5.1. Table 4 displays the average number of neurons eliminated by three pruning methods during ten trials. Because it takes the least amount of time, our proposal is the fastest. These results also support Theorem 5.1. Table 4 shows the average number of neurons removed by three pruning algorithms during 10 trials. Our proposal stands out as the fastest due to its minimal time consumption. Five-dimensional parity problem: norm of gradient with different algorithms (GD , GD , PGD , GDS). Table 4Performance for GDS, GD, PGD and GD for 4-dimensional and 5-dimensional parity problems, respectively.ProblemsMethodsAverage training errorAverage norm of gradientAverage numbers of neurons eliminated4-dimensional parityGD3.3253e−050.00114.3GD2.2094e−057.8358e−045.2PGD2.4346e−056.3772e−045.7GDS1.6752e−058.68678e−046.25-dimensional parityGD2.3153e−059.7451e−044.1GD1.4518e−057.0661e−044.6PGD1.1480e−056.4062e−045.8GDS9.9957e−064.7212e−046.2 Performance for GDS, GD, PGD and GD for 4-dimensional and 5-dimensional parity problems, respectively. Higher order Gaussian function A more general formulation of a Gaussian function with a flat-top and Gaussian fall-off can be taken by raising the content of the exponent to a power : This function is referred to as a super-Gaussian function and is frequently utilized in Gaussian beam formulation. The figure below was generated utilizing A = 0.5, , , . The GDS in this experiment has 2 input units, 6 hidden layer units, and 1 output unit. The training concludes after 100 iterations. One hundred one input points are uniformly selected from the interval [− 4, 4]. Of the 101 points, 81 input points are randomly selected as training patterns, while the remaining 20 points are utilized for testing generalization. Table 1 shows the models parameters. First, this example will test how GDS works with the training data in Table 5 and the suggested way to look at how the super-Gaussian function affects the error function, the gradient norm, and the average number of neurons that are thrown away. The square error is clearly observable in Fig. 6. The mean square errors keep going down as the number of epochs goes up, and the gradient in Fig. 7 gets closer and closer to zero as the number of iterations goes up. The mean square errors consistently diminish as the number of epochs increases. Table 5 shows that as GD goes down from GDS, the average number of neurons that are eliminated shows that our proposal is better (GDS). This outcome corroborates and aligns with the hypotheses of Theorem 5.1. Table 5Performance for GDS, GD PGD and GD for Super-Gaussian function.MethodsAverage training errorAverage norm of gradientAverage numbers of neurons eliminatedGD4.2068e-040.05342.1GD4.5647e-040.06191.9PGD3.9217e-040.04473.4GDS3.4332e-040.04033.9 Performance for GDS, GD PGD and GD for Super-Gaussian function. Higher order Gaussian function: value of the error function with different algorithms (GD, GD, PGD, GDS). Higher order Gaussian function: Norm of gradient with different algorithms (GD , GD , PGD , GDS). Gabor function approximation The PSN, since it incorporates higher-order correlation, is computationally efficient for function approximation problems. To test GDS, we have constructed a Gabor function approximation problem (cf. Figure 8). Gabor function. We choose 144 and 16 samples from a uniformly spaced and grid within the range and for the purpose of training and testing, respectively. In order to prevent overfitting during the training phase, we introduce uniformly distributed noise within the range of the weights to the training data. However, the testing samples are kept free from any noise. We do 10 trials for each of the three methods (GDS, PGD, GD, GD). The results presented in Figs. 9, 10, 11, 12, and 13 unequivocally shows that the GDS algorithm surpasses the PGD , GD and GD algorithms in terms of predictive performance. Table 6 presents the mean values of the training and testing error functions, as well as the average number of neurons removed by three pruning algorithms during 10 trials. Again, from Table 6 we see that our method GDS has the best accuracy and the best sparsity promoting property. Gabor function approximation performance by GD . Gabor function approximation performance by GD . Gabor function approximation performance by P GD . Gabor function approximation performance by GDS . Table 6Performance for GDS, PGD, GD and GD for Gabor function.MethodsAverage training errorAverage training errorAverage numbers of neurons eliminatedGD0.03550.05031.9GD0.01910.02312.8PGD0.01040.01204.0GDS0.01030.01204.3 Performance for GDS, PGD, GD and GD for Gabor function. Classification problems The UCI Machine Learning Repository is a collection of datasets, domain theories, and data generators that people in the machine learning community use to test machine learning algorithms in the real world. The UCI Machine Learning Repository presently houses 677 datasets for the benefit of the machine learning community. This platform allows for donations and provides access to datasets utilized by millions globally (https://archive.ics.uci.edu/). This experiment used binary issues to assess the efficacy of the PSN. We have obtained the data sets for binary classification tasks, including Sonar, WDBC, Monks, Climate, Speeds, Iris, Olitos, and Ecoli from the UCI Machine Learning Repository. Table 7 indicates that 70% of the data sets are designated as training samples, and the other 30% are classified as test samples. Table 7Specification for classification problem datasets.Data setData sizeInput featuresClasses1. Sonar2086022. WDBC5692823. Monks432624. Climate5402025. Speeds210736. Iris160437. Olitos1202548. Ecoli336879. Vehicle94618410. Waveform Version 2500040311. Splice-junction319061312. Mushroom8124222 Specification for classification problem datasets. In this training, eight databases were selected to verify the effectiveness of the proposed method as shown in Table 7. To analyze the effectiveness of the experiments, Figs. 12 and 13 lists two performance metrics, namely training accuracy (%) and testing accuracy (%), to show the classification capability of the network. Figure 12, explains efficiency of the training accuracy of the GDS, and from Fig. 13 we can see the flexibility of testing accuracy of the GDS algorithm is higher than the PGD , GD and GD algorithms, it can be concluded that the proposed new algorithm performs well, with better generalization. This will be our interest in future studies. Discussion The recent and convincing comparison in22 shows that the smoothing regularized method (BGS ) for FNN works better and promotes sparsity more than the regularized method (BG ), the regularized method (BG ), and even the smoothing regularized method (BGS ). Mohamed22 demonstrates many promising properties of the smoothing regularization, indicating that this new method can also extend to other networks. In this paper, we propose GDS for PSN and compare it with PGD , GD and GD , using the 4 dimensional and 5 dimensional parity problem, Super-Gaussian function, Gabor function, and eighth classification problems. The simulation results demonstrate that GDS generates sparser weights and a more streamlined structure while also enhancing learning efficiency. The network’s convergence is swift and seemingly independent of the network size within the simulated parameters. Figures 2 and 3, respectively, illustrates that the gradient error and norm of GD closely resemble those of GDS and are more realistic than those of GD . Table 4 presents the numerical data for these three learning methods across a total of 10 trials. Figures 4 and 5 indicates that the gradient error and norm approximate GD , with GDS serving as the baseline and demonstrating enhanced convergence mapping. The squared error E(W) is vividly seen in Fig. 6. The mean square errors decrease as the number of epochs increases, and the gradient in Fig. 7 approaches zero with the rising number of iterations. The mean square errors continually decrease as the number of epochs increases. Table 5 indicates that as GD decreases from GD , the average number of deleted neurons demonstrates the superiority of our approach (GDS). Table 6 illustrates the performance of average error during training and testing for the Gabor function in simulation, utilizing three learning algorithms throughout ten trials, demonstrating that GDS outperforms GD and GD . These results are clear as in the Figs. 9, 10, and 11, which show the extent to which the mapping of our proposal is better than others in terms of its similarity to the Gabor function. The comparison clearly demonstrates that GDS exhibits superior efficiency and enhanced sparsity-promoting characteristics compared to PGD , GD and GD . Tables 4, 5, and 6 unequivocally indicate that GDS is the most rapid algorithm among all numerical outcomes. Finally, the comparative of real world data sets findings of training accuracy and testing accuracy (in Table 8) demonstrate that our approach significantly outperforms others, indicating its potential for handling huge datasets. Table 8Performance comparison for classification problems.NoDate setAlgorithmTraining accuracy (%)Testing accuracy (%)1SonarGD94.6490.03GD96.5490.27PGD96.9991.01GDS97.1191.452WDBCGD93.2892.31GD94.4191.50PGD93.7690.15GDS95.6692.873MonksGD95.3689.32GD95.6990.17PGD96.5589.99GDS96.6191.734ClimateGD98.5992.87GD97.7390.22PGD97.0891.33GDS98.9893.135SpeedsGD95.2990.88GD 95.3191.79GDS95.8992.176IrisGD94.1991.45GD94.5591.59PGD95.1191.77GDS95.1692.687OlitosGD95.9890.11GD94.2089.99PGD95.4790.36GDS95.8791.718EcoliGD95.7491.57GD95.8893.13PGD97.1594.34GDS97.6194.899VehicleGD76.8073.27GD77.9572.16PGD79.0273.99GDS79.6974.3610Waveform Version 2GD87.4680.90GD86.1381.27PGD90.4183.62GDS90.4184.1811Splice-junctionGD81.0978.44GD80.4778.14PGD86.6785.79GDS89.2587.2112MushroomGD78.5572.43GD78.6173.11PGD79.3976.60GDS79.7676.89 Performance comparison for classification problems. Conclusion The standard regularization term incorporates absolute values and is non-differentiable at the origin, frequently resulting in an NP-hard problem. This paper proposes a batch gradient method incorporating smoothing regularization. This is achieved by employing a smoothing function to approximate regularization and circumvent its NP-hard nature. We have demonstrated the method for selecting the smoothing regularization parameter, which effectively prunes the network and improves its generalization capability. We have established the weak convergence of the suggested algorithm, indicating that the gradient function approaches zero and the error function converges to a constant. We also theoretically establish strong convergence, signifying that the weight sequence converges to a fixed point. Two simulation examples validate the superiority of the proposed method and the theoretical conclusions presented in this paper. Furthermore, the generalization of neural networks is a crucial concern in the development of learning methods for these networks. This implies that neural network models should not only exhibit great accuracy in approximating the data samples used for training, but also have strong performance when applied to new, unknown data. One often employed method to enhance the generalization of neural networks is incorporating a penalty term that is directly proportional to the amount of the network weights into the traditional error function. Our objective was to investigate the convergence of the gradient approach with smoothing regularization for neural networks in future research. In addition, we will focus on mini-batch gradient descent as a potential alternative to batch gradient descent. Mini-batch gradient descent is a technique that includes partitioning the dataset into smaller batches and conducting updates for each batch. This distinguishes it from batch gradient descent. These developments are anticipated to enhance the performance and efficiency of neural network training and pruning. Additionally, one of our forthcoming significant goals is to guarantee the convergence of network statements by maintaining a constant learning rate and applying them in real-life situations. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Acknowledgements The Researchers would like to thank the Deanship of Graduate Studies and Scientific Research at Qassim University for financial support (QU-APC-2025). Author contributions Khidir Shaib Mohamed: Conceptualization, methodology, Software, formal analysis, data curation, project administration, funding acquisition. Alawia Ibrahim Hamed Adam: Writing original draft and editing, Writing – review and editing. Yousif Shoaib Mohammed: Writing original draft and editing, writing – review and editing. Yan Xiong: Software, formal analysis, resources and investigation. Funding The authors received no external grant funding for this research. Data availability The code used during the current study is available from the corresponding author on reasonable request, and the UCI Machine Learning Repository datasets are available. https://archive.ics.uci.edu/. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Supplementary Materials Supplementary Material 1 Data Availability Statement The code used during the current study is available from the corresponding author on reasonable request, and the UCI Machine Learning Repository datasets are available. https://archive.ics.uci.edu/.