Convergence rates for Tikhonov regularization on compact sets: application to neural networks
Barbara Palumbo, Paolo Massa, Federico Benvenuto · 2025
Read Convergence rates for Tikhonov regularization on compact sets: application to neural networks by Barbara Palumbo, Paolo Massa, Federico Benvenuto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19936.txt.
Introduction
Barbara Palumbo, Paolo Massa, and Federico Benvenuto pace a cure for ill-posed back-asks in a 2025 paper. Such asks seek a hidden cause from blurred effects, and the hunt is wild without a brake. The stock brake adds a cost term, yet its pace of approach to truth can sag.
Their cure runs the brake on a chain of tight sets that fills the allowed zone step by step. The chain's step acts as a second dial. They prove the same pace as the stock brake, and the best pace when the map is straight. Neural webs fall under the same proof.
Catalog record
| CREATOR | Barbara Palumbo, Paolo Massa, Federico Benvenuto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Convergence rates for Tikhonov regularization on compact sets: application to neural networks Barbara Palumbo, Paolo Massa, Federico Benvenuto Abstract In this work, we consider ill-posed inverse problems in which the forward operator is continuous and weakly closed, and the sought solution belongs to a weakly closed constraint set. We propose a regularization method based on minimizing the Tikhonov functional on a sequence of compact sets which is dense in the intersection between the domain of the forward operator and the constraint set. The index of the compact sets can be interpreted as an additional regularization parameter. We prove that the proposed method is a regularization, achieving the same convergence rates as classical Tikhonov regularization and attaining the optimal convergence rate when the forward operator is linear. Moreover, we show that our methodology applies to the case where the constrained solution space is parametrized by means of neural networks (NNs), and the constraint is obtained by composing the last layer of the NN with a suitable activation function. In this case the dense compact sets are defined by taking a family of bounded weight NNs with increasing weight bound. Finally, we present some numerical experiments in the case of Computerized Tomography to compare the theoretical behavior of the reconstruction error with that obtained in a finite dimensional and non-asymptotic setting. The numerical tests also show that our NN-based regularization method is able to provide piece-wise constant solutions and to preserve the sharpness of edges, thus achieving lower reconstruction errors compared to the classical Tikhonov approach for the same level of noise in the data. keywords ill-posed inverse problems, regularization theory, convergence rates, deep neural networks † † runningheads: Regularization on compact sets / Barbara Palumbo, Paolo Massa, and Federico Benvenuto AMS 65J20, 65J22, 65K10 1 Introduction Regularization theory [ 11 ] provides a framework to overcome the ill-posedness of an inverse problem by introducing a family of operators that approximate the inverse of the forward operator in a stable way. Each operator in the family is indexed by a regularization parameter , which controls the trade-off between stability and accuracy. The most widely adopted regularization method is the Tikhonov strategy [ 38 ] , which defines the regularized solution as the minimizer of an objective functional. Depending on the specific problem, the Tikhonov method can be adapted to different noise models or incorporate additional prior knowledge. These include different data misfits, such as the L^{1} norm or the Kullback–Leibler divergence, and various penalty functionals. A central question in this context is the study of convergence rates , which quantify the speed at which the regularized solution converges to the exact solution as a function of the noise level, under smoothness assumptions on the true solution. Convergence rates for the Tikhonov method and its extensions have been thoroughly established; see, e.g., [ 16 , 15 , 35 , 27 , 4 , 6 , 18 , 33 , 22 , 21 , 34 , 17 ] . Another approach to define a family of regularization operators is to restrict the domain of the forward operator, thereby limiting the space of admissible solutions. A notable example of this approach is the Ritz regularization method which defines a dense sequence of finite-dimensional vector subspaces, and finds a Tikhonov-regularized solution within each subspace. Convergence rates for this method are the same as those of classical Tikhonov regularization, provided that the approximating subspaces become dense at a sufficiently fast rate [ 13 ] . An alternative strategy to obtain stable solutions to ill-posed inverse problems is to restrict the domain of the forward operator to a compact subset of the solution space. This approach is known as the selection method [ 38 ] . While this method do not define a regularization in the classical sense, since it does not introduce a family of approximating operators, it still ensures stability under certain conditions. In particular, if the forward operator is injective, restricting it to a compact domain yields a well-posed problem as the inverse operators is continuous. In this work, we consider a possibly non-linear inverse problem with a continuous and weakly closed forward operator. Unlike in previous studies, we assume that the sought solution belongs to a weakly closed constraint set. To provide a family of regularized solutions in this general case, we define a dense sequence of compact sets and we determine the Tikhonov solution within each set. This construction blends two distinct philosophies: on the one hand, it reflects the idea of the Ritz method of approximating the solution within a dense sequence of sets; on the other hand, by selecting compact sets, it aligns with the strategy of the selection method. Doing so, we reinterpret the selection method as a genuine regularization strategy. The main advantage of this reinterpretation is that the regularized solution can be defined as a neural network (NN) [ 12 ] , since the sequence of compact sets can be realized by neural networks with increasing bounded weights and number of parameters (e.g., the number of neurons), thanks to their universal approximation property which guarantees the ability to approximate the exact solution to any desired level of accuracy [ 24 ] . This leads to the additional benefit that in practice a regularized solution can be easily computed by optimizing the NN weights with widely-adopted gradient descent techniques [ 23 ] . Moreover, the use of NNs allows imposing constraints (e.g., non-negativity) on the inverse problems solution by applying an appropriate activation function at the last layer. This use of NNs substantially differs from the typical machine learning approaches adopted in the context of inverse problems. In this regard, machine learning has been used primarily for three purposes: (i) selecting or learning the regularization parameter, (ii) approximating the inverse operator, and (iii) designing data-adaptive regularization functionals or learned priors. In these settings as well, the literature offers an analysis of convergence rates; see, for example, [ 2 , 30 , 25 , 1 , 10 , 14 ] . However, our regularized solution can be viewed as a form of learned positional encoding, where spatial points are embedded via a NN, similarly to recent approaches [ 39 ] . A particularly relevant example of this technique is Neural Radiance Fields (NeRF), which was recently proposed and has been extensively applied to the problem of scene reconstruction and novel view synthesis from a limited number of viewpoints [ 28 , 3 , 41 , 19 ] . We prove that our proposed strategy is a regularization method based on two parameters: in addition to the classical Tikhonov regularization parameter, the second parameter corresponds to the index of the compact sets. Our proofs follow the analysis of non-linear inverse problems presented in Chapter 10 of [ 11 ] . We obtain convergence rates for classical source conditions on solutions belonging to a weakly closed constraint set, in line with those for regularization methods under non-negativity constraints [ 9 ] , or closed and convex constraints [ 8 ] . Analogously to the Ritz regularization method, our convergence rates depend on the rate at which the sequence of sets becomes dense within the solution space. We also prove that, when applied