Explicit integral representations and quantitative bounds for two-layer ReLU networks

Anthony Lee · 2026

Read Explicit integral representations and quantitative bounds for two-layer ReLU networks by Anthony Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23260.txt.

Introduction

Lee constructs explicit integral forms for two-layer ReLU networks, giving simple forms for any multivariate polynomial. Quantitative bounds are given for a sharpened form using a harmonic extension and a projection. The bounds show L2 errors free of dimension and degree, resting instead on monomial coefficients and the distribution. He also links the work to the reproducing kernel Hilbert space of the exponential kernel. A simpler form using multiplication by a fixed function carries better bounds.

Catalog record

CREATORAnthony Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64000
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Explicit integral representations and quantitative bounds for two-layer ReLU networks Anthony Lee Abstract An approach to construct explicit integral representations for two-layer ReLU networks is presented, which provides relatively simple representations for any multivariate polynomial. Quantitative bounds are provided for a particular, sharpened ReLU integral representation, which involves a harmonic extension and a projection. The bounds demonstrate that functions can be approximated with L 2 ​ ( 𝒟 ) L^{2}(\mathcal{D}) errors that do not depend explicitly on dimension or degree, but rather the coefficients of their monomial expansions and the distribution 𝒟 \mathcal{D} . 1 Introduction The capability of trained neural networks to accurately approximate some high-dimensional functions is now well established empirically, and has driven spectacular progress in many application areas. A robust theoretical explanation for this phenomenon, particularly for deep networks, remains outstanding. In this paper, we focus on an explicit representation of arbitrary multivariate polynomials in two-layer ReLU networks, which is relatively simple both to describe and to bound the performance of. The approach follows the perspective of two-layer networks adopted in 3 , i.e. to view a random two-layer network as an average of independent and identically distributed random functions \phi_{1},\ldots,\phi_{n} , or neurons, that are pointwise unbiased for the function of interest f . The resulting \mathcal{O}(n^{-1/2}) bound on the expected L^{2}(\mathcal{D}) error of the function then implies the existence of a deterministic two-layer network with at most that L^{2}(\mathcal{D}) error. As discussed in 10 , the general idea does not provide bounds on the number of neurons required without making assumptions on the function to be approximated, e.g. using a Barron semi-norm that depends on the Fourier transform. The representation and bounds we present are an attempt to contribute alternative, simple measures of function difficulty, and we require no assumptions on the polynomials to obtain quantitative bounds for suitable, sub-Gaussian and mean 0 , \mathcal{D} . We focus on arbitrary polynomials f\in\mathcal{P}_{d} , the space of polynomials with domain \mathbb{R}^{d} . Polynomials have a finite degree, but we do not need to make explicit the degree in the sequel. We adopt multi-index notation: for \alpha\in\mathbb{N}_{0}^{d} , x^{\alpha}=\prod_{i=1}^{d}x_{i}^{\alpha_{i}} and \alpha!=\prod_{i=1}^{d}\alpha_{i}! . For a given smooth, non-polynomial, function of interest \varphi:\mathbb{R}^{d}\to\mathbb{R} , we could, e.g., define f via a truncated Taylor expansion of \varphi around 0 : f(x)=\sum_{\alpha:\left|\alpha\right|\leq m}\frac{\partial^{\alpha}\varphi(0)}{\alpha!}x^{\alpha}=\sum_{k=0}^{m}\sum_{\left|\alpha\right|=k}\frac{\partial^{\alpha}\varphi(0)}{\alpha!}x^{\alpha}=\sum_{k=0}^{m}f_{k}(x), where f_{k}\in\mathcal{P}_{d,k} , the space of homogeneous polynomials of degree k in \mathcal{P}_{d} . The natural way to measure the size of these polynomials, at least from the perspective of our quantitative bounds, is via the Fischer norm of its homogeneous components. For a homogeneous polynomial of degree k , g(x)=\sum_{\left|\alpha\right|=k}c_{\alpha}x^{\alpha} , the squared Fischer norm of g is \left\|g\right\|_{F}^{2}=\sum_{\left|\alpha\right|=k}c_{\alpha}^{2}\alpha! . In view of the Taylor expansion of \varphi , this means \left\|f_{k}\right\|_{F}^{2}=\sum_{\left|\alpha\right|=k}\frac{\left|\partial^{\alpha}\varphi(0)\right|^{2}}{\alpha!} . Of course, one may define a different polynomial approximation to \varphi if a Taylor approximation is not suitable, or \varphi is not smooth. The restriction to polynomials is not particularly strong, from the perspective that polynomials are dense in the space of continuous functions with compact domain and dense in L^{2}(\mathcal{D}) for many probability measures \mathcal{D} with suitable tail decay. In contrast to several other works, we do not begin by expressing f using a Fourier transform as in 3 , i.e., f(x)=\int_{\mathbb{R}^{d}}\exp\left(i\left\langle\omega,x\right\rangle\right)\tilde{f}(\omega){\rm d}\omega, where \tilde{f} is the complex-valued Fourier transform of f . This approach appears to be the main starting point for integral representations ( 8 ; 7 , see, e.g.,) . The Fourier approach, while very general as it can be used for different activation functions, tends to involve a sequence of approximations, which makes an explicit and interpretable integral representation difficult to obtain. Often, there are assumptions required to either represent the function of interest or provide quantitative bounds. In contrast, the representation we develop here is specifically for the ReLU network and we make no assumptions beyond the target function f being a polynomial: we will develop representations that are totally explicit for any given polynomial and the quantitative bounds are valid for any polynomial, accepting of course that they may not always be accurate. In particular, the bounds are consistent with the notion that a two-layer network is capable of approximating well functions that do not depend too much on too many components of their input, and that neither dimension nor degree are necessarily important. Our approach involves the explicit solution of an integral equation involving an appropriate ReLU integral operator from which a ReLU integral representation is immediately available (Theorem 5 ). Once a ReLU integral representation is obtained, it can be optimized to minimize the expected squared L^{2}(\mathcal{D}) error in an explicit way (Proposition 2 ). As we restrict our attention to polynomials, the analysis avoids many technicalities, and we find that there is a ReLU integral representation of a particular form for every polynomial (Theorem 9 ). It transpires that the most direct application of this result may produce complicated representations, but that by performing an appropriate harmonic extension, we are able to recover a representation for an extension of the function, leading ultimately to a simpler ReLU representation for the original function (Theorem 23 ), which involves sharpened versions of its homogeneous components: these are defined by running the heat equation backwards in time. Ultimately, the error bounds (Theorem 30 ) only involve the Fischer norms of the homogeneous components of f . The network and bounds seem to show some agreement with trained two-layer ReLU networks for some examples. A careful study would be required to determine whether these representations are a suitable model for trained networks. In particular, there are transformations that could be applied prior to constructing the network, and it is not clear which of these would minimize the error of the network. In simple examples, the trained network is consistent with having performed such a transformation. The bounds seem also seem to capture the sense that functions that depend only weakly on each individual input component, or that depend on only a few input components, are easier to represent. 2 ReLU networks, integral representations and operator 2.1 Two-layer ReLU network Let \varsigma=\max\{x,0\} , interpreted element-wise if x is a vector, denote the ReLU activation function. An L layer ReLU neural network, viewed as a function f_{{\rm NN}}:\mathbb{R}^{d}\to\mathbb{R} , may be written as f_{{\rm NN}}=f_{L}\circ\cdots\circ f_{1}, where for \ell\in\{1,\ldots,L-1\} , with W_{\ell} a matrix and b_{\ell} a vector, f_{\ell}(x)=\varsigma\left(W_{\ell}x+b_{\ell}\right), and f_{L}(x)=a+u^{T}x. Such networks, when trained appropriately, appear to have very impressive representational capacity and amenability to optimization. We focus on the case L=2 , the two-layer ReLU network, which can be written as f_{n}(x)=a+\frac{1}{n}\sum_{i=1}^{n}u_{i}\varsigma\left(\left\langle x,v_{i}\right\rangle+b_{i}\right),\qquad x\in\mathbb{R}^{d}, where the parameters of the function are (a,v_{1:n},b_{1:n},u_{1:n}) with a,b,u_{i}\in\mathbb{R} and v_{i}\in\mathbb{R}^{d} . One can think of f_{n} as mainly involving the average of parameterized functions \phi(\cdot;u,v,b)=u\varsigma\left(\left\langle\cdot,v\right\rangle+b\right), each of which cannot be an accurate representation of some target function f , but whose combination clearly can. Although considerably less powerful empirically than f_{{\rm NN}} , two-layer networks are a suitable target for mathematical research, and progress in their analysis may facilitate progress for deeper networks. 2.2 ReLU integral representations Since 3 , one useful perspective is to view the two-layer network f_{n} as an approximation of an integral representation of f . We

Read the complete text