Approximation theory for 1-Lipschitz ResNets
Davide Murari, Takashi Furuya, Carola-Bibiane Schönlieb · 2025
Read Approximation theory for 1-Lipschitz ResNets by Davide Murari, Takashi Furuya, Carola-Bibiane Schönlieb (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12003.txt.
Introduction
Davide Murari, Takashi Furuya, and Carola-Bibiane Schönlieb wrote a 2025 math paper on a special family of neural networks. These networks never stretch distances between inputs, a trait that helps in making images, solving inverse problems, and building sturdy classifiers. The paper asks how well such networks can match a target function.
The team proves the networks can match any target of the same type on a bounded region, given enough width and depth. They can also match certain jagged, straight-line functions exactly. With norm limits placed between blocks, the same power holds at fixed width. The results give the first full guarantees of this kind for these networks.
Catalog record
| CREATOR | Davide Murari, Takashi Furuya, Carola-Bibiane Schönlieb |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12003
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Approximation theory for 1-Lipschitz ResNets Davide Murari, Takashi Furuya, Carola-Bibiane Schönlieb Abstract 1 1 -Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on 1 1 -Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone–Weierstrass Theorem, we first show that these 1 1 -Lipschitz ResNets are dense in the set of scalar 1 1 -Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine 1 1 -Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for 1 1 -Lipschitz ResNets, laying a rigorous foundation for their practical use. 1 Introduction The flexibility of neural network parameterisations allows them to approximate any regular enough target function arbitrarily well [ 41 , 40 , 28 , 20 , 8 , 17 ] . Despite this desirable aspect, there are several reasons why one would not want a completely unconstrained parameterisation. For instance, unconstrained networks tend to be overly sensitive to input adversarial perturbations because their local Lipschitz constants can be large, making them unreliable classifiers [ 34 ] . There are also situations where one is purely interested in modelling specific sets of functions, for example, to turn a constrained optimisation problem into an unconstrained one [ 19 , 31 , e.g] . A prominent example is the critic in Wasserstein GANs [ 2 ] , which must be 1-Lipschitz to yield a valid estimate of the 1-Wasserstein distance via the Kantorovich-Rubinstein duality [ 37 ] . In this paper, we focus on scalar-valued neural networks that are constrained to be 1 -Lipschitz in the Euclidean \ell^{2} norm. These networks have found extensive applications in inverse problems [ 33 , 32 , 16 ] , generative modelling [ 14 , 24 , 21 ] , and as a means to improve network resilience to adversarial attacks [ 36 , 33 , 35 , 29 , 23 ] . The approximation properties of constrained networks are poorly understood, and different enforcement strategies can yield markedly different expressiveness. The commonly adopted strategies to constrain a network’s Lipschitz constant tend to reduce its expressiveness, leading to noticeable performance drops. This paper studies the approximation properties of Lipschitz-constrained residual networks. We propose new constrained networks that are theoretically able to approximate any scalar 1 -Lipschitz function arbitrarily well. Although our analysis is theoretical, the architectures we consider are readily implementable and can be trained like existing Lipschitz-constrained layers. We next review related work (Section 1.1 ), summarise our contributions (Section 1.2 ), and present an outline of the paper (Section 1.3 ). 1.1 Related work 1 -Lipschitz neural networks. We can identify three principal approaches to enforce or promote Lipschitz constraints: weight normalisation strategies, changes to the network layers, and regularisation strategies. Spectral normalisation or orthogonal weight matrices provide the most well-established constraining procedure for feed-forward neural networks [ 24 , 35 ] . When working with Residual Neural Networks (ResNets), more changes are necessary due to the skip connection. Results in dynamical systems, numerical analysis, and convex analysis lead to 1 -Lipschitz ResNets based on negative gradient flows [ 33 , 23 ] . These constrained ResNets are the backbone of most of the results in this paper, and we will describe them later. They have also proven to be more efficient than other constraining strategies in terms of performance and computational resource consumption [ 29 ] . Because of the empirical decrease in expressiveness with constrained networks, an alternative to hard constraining the Lipschitz constant is regularising the optimisation problem by penalising too large Lipschitz constants [ 21 , 14 , 38 , 42 ] . Lipschitz constraints are also considered in more modern architectures, such as Transformers [ 18 , 9 ] . Approximation theory for Lipschitz-constrained networks. Constraining the Lipschitz constant of a network naturally leads to restricted expressiveness, in the sense that a 1 -Lipschitz neural network can not approximate f(x)=x^{2} arbitrarily well, for example. However, a more relevant question is if all the 1 -Lipschitz functions can be accurately approximated by a given family of 1 -Lipschitz networks. In [ 1 ] , the authors present the Restricted Stone-Weierstrass Theorem and prove that feed-forward networks based on the \mathrm{GroupSort} activation function and norm-constrained weights are dense in the set of scalar 1 -Lipschitz functions. This theorem is fundamental for our derivations as well, and we state it in Section 2 . In [ 25 ] , the authors study feed-forward networks with splines as activation functions and show that they have the same expressiveness as the networks studied in [ 1 ] . To the best of our knowledge, no such results are available for 1 -Lipschitz ResNets. Closely related to ResNets, we mention [ 10 ] where the authors show that Neural ODEs with a constraint on the Lipschitz constant of the flow map are still universal approximators of continuous functions if the linear lifting and projection layers are left unconstrained. For further motivation of why we study 1 -Lipschitz ResNets and why the 1 -Lipschitz constraint is relevant, see Appendix F.1 . 1.2 Main contributions This paper studies the approximation capabilities of 1 -Lipschitz Residual Neural Networks (ResNets). We focus on \mathrm{ReLU}(x)=\max\{x,0\} as activation function. Relying on the Restricted Stone-Weierstrass Theorem, in Theorem 3.1 we show that a class of ResNets with residual layers based on explicit Euler steps of negative gradient flows is dense in the set of scalar 1 -Lipschitz functions. To achieve this approximation result, we allow for networks of arbitrary width and depth. We also provide an alternative proof by showing, in Theorem 3.2 , that this set of networks contains all the 1 -Lipschitz piecewise affine functions. This alternative derivation provides further insights into the considered networks. It also informs the other main result of the paper, where the network width is fixed, allowing us to connect these two main theorems. This density result is extended to vector-valued 1 -Lipschitz functions as well, see Lemma 5.1 . By interleaving the residual layers with suitably constrained linear maps, we show, in Theorem 4.1 , that the width can be fixed while preserving the density in the set of scalar 1 -Lipschitz functions. This second main result allows us to propose a new, practically implementable ResNet-like architecture that is flexible enough to approximate to arbitrary accuracy any scalar 1 -Lipschitz function. We remark that one of the main novelties of our analysis lies in proving the universality of 1 -Lipschitz ResNets without relying on the Restricted Stone-Weierstrass Theorem, and in providing two alternative viewpoints to our analysis, thereby gaining a deeper understanding of Lipschitz-continuous ResNets. Not directly relying on the Restricted Stone-Weierstrass Theorem forces us to understand that the set of considered networks contains a very well-studied set of functions: 1-Lipschitz piecewise affine functions. This