Difference of Convex (DC) approach for neural network approximation with uniform loss function
2026
Read Difference of Convex (DC) approach for neural network approximation with uniform loss function (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05557.txt.
Introduction
Vinesha Peiris and Nadezda Sukhorukova test, in a 2026 paper, whether difference-of-convex programming can train neural networks under a uniform loss. Most network training minimizes average squared error. Uniform, or minimax, approximation instead controls the worst-case error, which is harder because the objective is nonsmooth.
A difference-of-convex method splits a difficult function into two convex parts that are easier to handle. Earlier work showed the approach works as an optimizer for neural networks. Here the authors apply it to minimax approximation and run numerical experiments. They compare its performance against AdaMax, a commonly used method for this task, and report the difference-of-convex approach is also efficient in the uniform-loss setting.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Difference of Convex (DC) approach for neural network approximation with uniform loss function Abstract Neural networks (NNs) can be viewed as approximation tools. Traditionally, NNs are relying on gradient and stochastic gradient (SG) methods. There are a number of available computational packages for constructing least squares approximations, while uniform (minimax) approximations are hard due to their nonsmooth nature. It was recently demonstrated that a difference convex (DC) programming approach is an efficient alternative optimiser for NNs. In this paper, we demonstrate that a DC programming approach is also efficient for minimax approximation. In our numerical experiments, we compare a DC-programming approach and ADAMAX, a commonly used method for minimax NN approximations. Vinesha Peiris, Nadezda Sukhorukova Mathematics Subject Classification (2020): 90C26, 65K05, 41A50, 90C47, 90C90 Keywords: minimax, Chebyshev approximation, DC-programming, convex analysis, neural networks 1 Introduction Neural Networks (NNs) are powerful Artificial Intelligence (AI) techniques. The theory of NNs is based on solid mathematical background, established in [ 1 , 2 , 3 , 4 ] . These works lead to the work of A. Kolmogorov and V. Arnold. Their celebrated Kolmogorov-Arnold Theorem [ 5 , 6 ] was an attempt to solve the 13th Hilbert’s problem. Essentially, NNs provide accurate approximations. These approximations are found as solutions of optimisation problems, where the objective functions represent the approximation error. An excellent overview of optimisation techniques for NNs can be found in [ 7 ] . The most popular choice for the loss function is least squares due to their smoothness. The numerical methods are based on gradient-type techniques, for example, Stochastic Gradient (SG). In the case of uniform approximation, the corresponding optimisation problems are nonsmooth and the applicability of SG is questionable. In [ 8 ] the authors demonstrated that Difference of Convex (DC) algorithms are an efficient alternative to the traditional SG techniques in the case of least squares networks. More details on DC-programming are provided in section 3 . Despite a significant progress in NN approximation in least squares criterion, its Chebyshev (uniform) counterpart is still behind due to its complexity: the corresponding optimisation problems are nonsmooth and non-convex. Chebyshev approximation is a fascinating mathematical problem itself and moreover, it has several practical applications. For example, in [ 9 ] the authors demonstrated that uniform approximation is a more reliable approach if the amount of data is limited. A comprehensive survey on necessary and sufficient optimality conditions for Chebyshev approximation (with a focus on cone constraints) can be found in [ 10 ] . See also references therein. The contribution of this paper is as follows. First of all, we formulate uniform-based NN as DC-programming problems. Then, we implement the numerical experiments with DC-algorithms. Finally, we compare the results with the performance of ADAMAX [ 11 ] , a popular technique for uniform networks. 2 Motivation DC-programming is a powerful tool in the area of modern optimisation, namely, global non-convex optimisation. The origins of DC-programming appear in [ 12 ] . Then there have been decades of fruitful development of the theory and algorithms (DCA, stands for DC-programming Algorithm). DCA and its variations have been successfully applied to real-world problems [ 13 , 14 ] just to name a few. A detailed modern review of DC-programming and the corresponding algorithms can be found in [ 15 , 16 ] . One of the applications, where DC-programming and DCA have been successfully implemented is deep learning and neural networks. In particular, in [ 17 ] the authors use DCA for the computation of the adversarial attack for adversarial robustness. Also, in [ 8 ] the authors demonstrated that DCA can be used as an alternative approach to the neural networks. These papers demonstrated the efficiency of DC-programming in this application. In the current paper, we extend the existing results in the following directions. 1. Instead of least squares approximation, we use uniform ( \max -norm) based loss function. This approach may be more efficient in the case when the available data are limited [ 9 ] . 2. We use ReLU as the activation function. This function is commonly used in neural networks, but its nonsmooth nature limits the application of gradient-based optimisation techniques, in particular, when the loss function is nonsmooth, which is the case for uniform approximation. It is mentioned in [ 16 ] that in the case of ReLU and Leaky ReLU activation functions, the implementation of DCA can be reduced to solving linear programming problems repeatedly. This is a great advantage, since modern optimisation packages are able to solve large-scaled linear programming problems efficiently. In this paper, we explicitly formulate these linear programming problems and solve them within DCA. In our numerical experiments, we compare the results obtained by DCA with ADAMAX [ 11 ] , a specially developed algorithm for uniform-based loss functions (minimax approximation). In the next section, we provide all the necessary mathematical constructions: DC decomposition (that is, the presentation as a difference of convex functions) of the networks and the implementation of DCA. 3 DC decomposition and DCA implementation 3.1 Neural network approximations Deep learning and neural networks are popular tools for function and data approximation. In this paper, we concentrate on the purely approximation nature of the problem. It was demonstrated in [ 3 , 4 ] that even neural networks with a single hidden layer are efficient approximation techniques. In most practical problems, the goal is to reconstruct (approximate) a function, whose values are known in a finite number of points (dataset Q ). Essentially, a single hidden layer neural network with ReLU activation function construct functions and data approximations in the following form: \sum_{i=1}^{n}\alpha_{i}\max\{0,{\bf a}_{i}^{T}{\bf T}\}, (1) where {\bf T}\in Q are the discrete points, {\bf T}\in\mathbb{R}^{d} , \alpha_{i} , i=1,\dots,n and the components of vectors, {\bf a}_{i}^{T}\in\mathbb{R}^{d} are the parameters of the neural network. The components of {\bf a}_{i}^{T}\in\mathbb{R}^{d} are also called weight. The number of components n in the sum of ( 1 ) is the number of nodes in the hidden layer. In most practical problems, this number is not known. When n increases, the accuracy of the approximations improves, while the computational complexity increases. Note that the activation function ReLU \sigma(x)=\max\{0,x\} is positively homogeneous of degree one. Therefore, if the sign of each \alpha_{i} , i=1,\dots,n is known, the absolute value of \alpha_{i} , i=1,\dots,n can be reconstructed through the components of vectors {\bf a}_{i}^{T} . Hence, the approximation in ( 1 ) can be constructed as follows: \sum_{i=1}^{n_{1}}\max\{0,{\bf a}_{i}^{T}{\bf T}\}-\sum_{i=n_{1}+1}^{n_{1}+n_{2}}\max\{0,{\bf a}_{i}^{T}{\bf T}\}, (2) where n_{1} and n_{2} are large enough. Similar decompositions have been proposed in [ 18 ] . If n_{1}=n_{2} in ( 2 ), one can talk about the number of pairs (one component comes with a plus and another with a minus). Therefore, the minimax optimisation problem studied in this paper is as follows: \text{minimise}\max_{i=1,\dots,N}\left|f({\bf T}_{i})-\left(\sum_{j=1}^{n_{1}}\max\{0,{\bf a}_{j}^{T}{\bf T}_{i}\}-\sum_{j=n_{1}+1}^{n_{1}+n_{2}}\max\{0,{\bf a}_{j}^{T}{\bf T}_{i}\}\right)\right|. (3) In this paper, for simplicity, we assume that n_{1}=n_{2}=n and denote the weights from the second group (for j\geq n_{1}+1 ) by {\bf b}_{j} , j=1,\dots,n . Then problem ( 3 ) is \text{minimise}\max_{i=1,\dots,N}\left|f({\bf T}_{i})-\left(\sum_{j=1}^{n}\max\{0,{\bf a}_{j}^{T}{\bf T}_{i}\}-\sum_{j=1}^{n}\max\{0,{\bf b}_{j}^{T}{\bf T}_{i}\}\right)\right|. (4) In the next subsection, we formulate a DC decomposition to the objective function in ( 4 ). 3.2 DC decomposition