Nonsmooth Optimisation and neural networks
Vinesha Peiris, Nadezda Sukhorukova · 2025
Read Nonsmooth Optimisation and neural networks by Vinesha Peiris, Nadezda Sukhorukova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01860.txt.
Introduction
Vinesha Peiris and Nadezda Sukhorukova look at neural networks through the lens of kinked mathematics in a 2025 paper. Network fitting is an optimization problem full of corners, where smooth calculus stalls. Quasidifferential calculus, a toolkit built for such corners, is their chosen lens.
They study the clean task of matching a curve with a bare network, one layer deep, with monotone activation shapes. A step-by-step algorithm builds the one-hidden-layer approximator piece by piece. The nonsmooth tools prove efficient, and they partly explain an oddity: the procedure can stall after a single step with no gain. The paper is a proof that corner-aware math belongs in network theory.
Catalog record
| CREATOR | Vinesha Peiris, Nadezda Sukhorukova |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01860
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nonsmooth Optimisation and neural networks Vinesha Peiris, Nadezda Sukhorukova Abstract In this paper, we study neural networks from the point of view of nonsmooth optimisation, namely, quasidifferential calculus. We restrict ourselves to the case of uniform approximation by a neural network without hidden layers, the activation functions are restricted to continuous strictly increasing functions. We develop an algorithm for computing the approximation with one hidden layer through a step-by-step procedure. The nonsmooth analysis techniques demonstrated their efficiency. In particular, they partially explain why the developed step-by-step procedure may run without any objective function improvement after just one step of the procedure. Keywords: nonsmooth optimisation, quasidifferentials, bisection for quaiconvex optimisation. MSC Classification 41A50, 26A27, 65D10, 65D12, 65D40 1 Introduction Deep learning is a branch of machine learning which evolve rapidly within the area of artificial intelligence. Recently, the study of deep learning has been an interesting area for many researchers due to its excellent performance in a variety of practical applications [ 14 , 20 , 24 , 28 ] , just to name a few. Deep learning focuses on using neural networks to model and solve complex problems. In general, a neural network consists of interconnected layers of nodes (input, hidden, output) that process and transform data. These networks are designed to learn patterns and representations directly from data through a process known as training, where the network adjusts its internal parameters (weights and biases) to minimise errors. These networks can also be interpreted as solving approximation problems, with their internal parameters effectively managed through optimisation techniques [ 14 , 16 , 31 ] . The least squares loss function (error function), widely used in neural networks, serves as a standard method to evaluate and optimise the performance of the network on a training dataset. Its popularity comes from the fact that it involves minimising a smooth quadratic objective function, making it well suited for basic optimisation techniques such as gradient descent. In some specific cases, the loss function can be adapted to a uniform (Chebyshev) approximation based model, which minimises the maximum error and outperforms least squares–based models when the training dataset is accurate, but limited in size [ 25 , 26 ] . In this research, we use a simple model of an artificial neural network that has no hidden layers and only one output node. In the case of deep neural networks, his model corresponds to the optimisation between two consecutive layers. We explore the use of two types of activation functions for the output layer: Leaky ReLU function and strictly increasing smooth functions. We utilise the uniform approximation based loss function which leads to a nonsmooth and nonconvex optimisation problem and therefore, nonsmooth optimisation techniques are needed to address them. One of the most comprehensive textbooks on nonsmooth optimisation is [ 2 ] . The quasidifferential is an appropriate tool for the construction of the solution of nonsmooth and nonconvex optimisation problems. It is a possible extension of the notion of gradient to the case of nonsmooth functions. Since its introduction by V.F. Demyanov, A. M. Rubinov and L. N. Polyakova [ 8 , 9 , 10 , 11 ] , it has been an interesting research direction for many mathematicians. By using the well-developed calculus rules for quasidifferential functions [ 6 , 12 ] , we give a characterisation of necessary optimality condition for our optimisation problems which proves that every local minimum is an inf-stationary point. Moreover, we propose applying the bisection algorithm from [ 26 ] step-by-step to solve the corresponding optimisation problem. This paper is organised as follows. The motivation for the current study is discussed in Section 2 . Section 3 explains the notion of quasidifferentials and related calculus rules. In Section 4 , we introduce the mathematical formulation of the networks that we use. In Section 5 , we discuss the step-by-step process to solve the optimisation problems and we provide the necessary condition for optimality. Section 6 provides numerical experiments and finally, Section 7 explains conclusions and future research directions. 2 Motivation The theory of Artificial Neural Networks (ANNs) is based on solid mathematical background. The origins come back to the work of A. Kolmogorov and his student V. Arnold. Their celebrated Kolmogorov-Arnold representation Theorem [ 1 , 19 ] is an attempt to solve the 13th problem of Hilbert. Then, further development of this result lead to a convenient and efficient adaptation to function and data approximation, established in [ 5 , 16 , 21 , 27 ] . An excellent overview of optimisation techniques for ANNs can be found in [ 31 ] . Essentially, ANNs construct approximation in the form of a sum \sum_{i=1}^{n}a_{i}\sigma((b_{1},b_{2},\dots,b_{m}){\bf T}+b_{0}), where {\bf T\in\mathbb{R}^{m}} and a_{1},\dots,a_{n},b_{0},\dots,b_{m} are the parameters of approximation and these parameters are the decision variable of the optimisation problems. The goal is to minimise the loss functions, which are the approximation error (least squares, maximal absolute deviation, sum of absolute deviations, etc.). Most ANN packages rely on least squares, since the corresponding optimisation problems are smooth and can be locally optimised. In our research, we optimise the maximal deviation (uniform norm) and the corresponding optimisation problems are nonsmooth and nonconvex. ANN are very popular techniques in modern numerical applications, in particular, for construction accurate approximation [ 15 , 22 , 23 , 26 ] just to name a few. Still there are a number of open problems and avenues for improvements [ 17 ] . Due to the nonsmoothness of the problem, one needs nonsmooth optimisation techniques to tackle them. One possible way is to do it using quasidifferential calculus [ 11 ] . This is just one of the possible extension of the notion of gradient to the case of nonsmooth functions. Quasidifferential calculus has been applied to a number of difficult nonsmooth optimisation problems, including best free knot polynomial spline approximation [ 29 ] . 3 Quasidifferentiable calculus Nonsmooth optimisation is a branch of optimisation where one deals with nonsmooth functions. The history of nonsmooth optimisation can be traced back to P. Chebyshev and his approximation problems [ 4 ] . A very detailed review of the development on this field can be found in [ 13 ] . In this study, we are using the notion of quasidifferentiability, one of the possible generalisation of smoothness, originally described in [ 8 , 10 ] and later developed to a rich and fruitful field [ 6 , 11 , 12 ] . This theory allows one to compute descent directions for nonsmooth functions. If no descent direction can be obtained, the point is inf-stationary in the sense of Demyanov-Rubinov . The formal definition is provided later in this section. This can be done through the so-called Quasidifferential Calculus, developed in [ 11 ] . In this paper, we only present the essential results, directly related to our study. For more information, one should refer to [ 11 , 12 ] . Definition 3.1 ( [ 12 , Chapter 1, page 7] ) . A function \Psi defined on an open set \Omega is quasidifferentiable at a point X\in\Omega if it is locally Lipschitz continuous, directionally differentiable at this point and there exists compact, convex sets \underline{\partial}\Psi(X) and \overline{\partial}\Psi(X)