Feature Qualification by Deep Nets: A Constructive Approach
Feilong Cao, Shao-Bo Lin · 2025
Read Feature Qualification by Deep Nets: A Constructive Approach by Feilong Cao, Shao-Bo Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18676.txt.
Introduction
Deep networks can approximate almost anything, which paradoxically makes it hard to say which features of the data do the work. Feilong Cao and Shao-Bo Lin take a constructive route in 2025. Rather than probing a trained network, they build one by hand. Their linear operator, made of deep sigmoid nets, is tuned to expose feature quality. Its product gates and local fitting power do the heavy lifting.
The built operator approximates smooth and radial functions at the best possible rate. Better, it can certify several features at once, smoothness and radialness among them. The authors offer the construction as a theoretical yardstick for judging what a target function demands.
Catalog record
| CREATOR | Feilong Cao, Shao-Bo Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Feature Qualification by Deep Nets: A Constructive Approach Feilong Cao, Shao-Bo Lin Abstract The great success of deep learning has stimulated avid research activities in verifying the power of depth in theory, a common consensus of which is that deep net are versatile in approximating and learning numerous functions. Such a versatility certainly enhances the understanding of the power of depth, but makes it difficult to judge which data features are crucial in a specific learning task. This paper proposes a constructive approach to equip deep nets for the feature qualification purpose. Using the product-gate nature and localized approximation property of deep nets with sigmoid activation (deep sigmoid nets), we succeed in constructing a linear deep net operator that possesses optimal approximation performance in approximating smooth and radial functions. Furthermore, we provide theoretical evidences that the constructed deep net operator is capable of qualifying multiple features such as the smoothness and radialness of the target functions. 1 Introduction The great success of deep learning in practice [ 1 ] significantly stimulates the theoretical understanding of deep neural networks (deep nets), including to pursue the excellent performances in approximating numerous functions [ 2 ] , to derive the capacity of deep nets under different metrics [ 3 ] , to show their tractability in learning high-dimensional data [ 4 ] , to analyze the global landscapes for some convex losses concerning deep nets [ 5 ] , and to settle certain convergence issues for gradient-based algorithms [ 6 ] . These encouraging developments not only build a meaningful springboard to discover the running mechanism of deep learning, but also provide promising guidance in designing new deep learning schemes. However, several issues such as the inconsistency between generalization and optimization on the size of networks [ 7 ] and the structure selection for given learning tasks [ 8 ] still remain open. More importantly, there is a crucial dilemma between the versatility and feature qualification for deep nets in the sense that the versatility prohibits deep learning to figure out which data features are extracted in the approximation and learning processes. The aim of this paper is to settle the dilemma between the versatility and feature qualification by constructing deep net operators. Our study is motivated by three interesting observations. At first, inverse theorems that qualifies the smoothness via the approximation rates have been successfully derived for polynomials [ 9 ] , radial basis functions [ 10 ] , and shallow nets [ 11 ] . It is highly desired to establish similar inverse theorems for deep nets. Then, the versatility of whole deep net sets makes it impossible to qualify a single data feature by approximation rates. For example, if the target functions are smooth and spatially sparse [ 12 , 13 , 14 ] , then the approximation rates are essentially products of two quantities measuring the sparseness and smoothness. Without additional a-priori information, it is extremely difficult to factorize these products to reflect the detailed smoothness and sparsity. A preferable way to conquer this obstacle is to narrow the range of deep nets to compromise their versatility. Finally, constructing rather than training deep nets succeeds in avoiding the inconsistency between optimization and generalization in learning smooth functions [ 15 ] , spatially sparse functions [ 16 ] and piecewise smooth functions [ 14 ] . The construction approach presented in [ 11 , 17 ] is also capable of yielding an inverse theorem for shallow nets in terms that the smoothness of the target functions can be qualified by the approximation rates. Based on these observations, we devote to constructing deep nets that are capable of breaking through bottlenecks of shallow nets, and qualifying the data features by approximation rates, simultaneously. As it has been verified in [ 18 , 19 , 20 ] that both classical algebraic polynomials and shallow nets are incapable of extracting the radial property of the target functions, we aim to construct deep nets to approximate smooth and radial functions to embody the power of depth. We adopt the sigmoid function to be the nonlinear activation since its differentiability permits to establish the so-called Bernstein-type inequality that is crucial in feature qualification [ 9 , 10 , 11 ] . Utilizing the square-gate [ 20 ] and localized approximation properties of deep sigmoid nets [ 21 ] , we succeed in constructing a linear deep-neural-net-operator (DNO) for the approximation and feature qualification purposes. From the approximation perspective, we derive almost optimal rates for the constructed DNO in approximating smooth and radial functions which is beyond the capability of shallow nets with arbitrary activation functions, demonstrating the power of depth in our construction. Furthermore, the derived approximation rates are essentially the same as the those for training-based deep sigmoid nets [ 20 ] , showing that the construction approach does not degenerate the approximation performance of deep nets. From the feature qualification perspective, we prove that the construction DNO is an optimal conditional qualification operator with respect to the smoothness and radialness. In particular, we verify that if the smoothness of target the target functions is known, then the radialness information can be sufficiently reflected by the approximation rates of DNO, and vice-verse. The rest of the paper is organized as follows. In the next section, we introduce some basic definitions concerning deep sigmoid nets. In Section 3 , we construct DNO and study its approximation performance. In Section 4 , the feature qualification property of the constructed DNO is analyzed. In Section 5 , we present the stepping stone of our theoretical analysis and the proofs of main results are given in the last section. 2 Deep Nets: Approximation and Feature Qualification A deep net with depth L\in\mathbb{N} and width vector (d_{1},\dots,d_{L})^{T} for d_{\ell}\in\mathbb{N} is mathematically defined by \mathcal{N}_{d_{1},\dots,d_{L}}(x)=a^{L}\cdot\sigma\circ\mathcal{J}_{L,W^{L},b^{L}}\circ\sigma\circ\dots\circ\sigma\circ\mathcal{J}_{1,W^{1},b^{1}}(x),\qquad x\in\mathbb{R}^{d}, (2.1) where f_{1}\circ f_{2}(x)=f_{1}(f_{2}(x)) , {a}^{L}\in\mathbb{R}^{d_{L}} , \sigma:\mathbb{R}\rightarrow\mathbb{R} is the activation function with \sigma(\vec{u}) acting on vectors componentwise of the vector \vec{u} , and \mathcal{J}_{\ell,W^{\ell},b^{\ell}}:\mathbb{R}^{d_{\ell-1}}\rightarrow\mathbb{R}^{d_{\ell}} is the affine operator given by \mathcal{J}_{\ell,W^{\ell},b^{\ell}}(x):=W^{\ell}x+b^{\ell} with d_{\ell}\times d_{\ell-1} weight matrix W^{\ell} and d_{\ell} dimensional bias vector b^{\ell} . Deep nets defined by ( 2.1 ) possess at most \tilde{n}_{L}:=d_{L}+\sum_{k=1}^{L}(d_{k-1}d_{k}+d_{k}) (2.2) tunable parameters with d_{0}=d . Denote by \mathcal{H}_{d_{1},\dots,d_{L}} the set of all deep nets formed as ( 2.1 ). If L=1 , \mathcal{N}_{d_{1}} defined by ( 2.1 ) is the classical shallow net. The structure of the deep net is determined by the weight matrices W^{1},\dots,W^{L} . In particular, sparse matrix W^{\ell} , full matrix W^{\ell} and Toeplitz matrix correspond to deep sparse connected networks (DSCNs) [ 22 ] , deep fully connected networks (DFCNs) [ 23 ] and deep convolutional neural networks (DCNNs) [ 24 ] , respectively. The study of advantages of deep nets can date back to the 1990s, when [ 21 ] proved that deep nets can provide localized approximation but shallow nets fail. After then, limitations of shallow nets have been widely discovered, including their bottlenecks in approximating smooth functions [ 25 ] , extracting the radialness [ 18 ] , capturing the sparseness in the frequency domain [ 26 ] and spatial domain [ 12 ] , etc.. We refer the readers to two fruitful review papers [ 27 , 2