Approximation Capabilities of Feedforward Neural Networks with GELU Activations

Konstantin Yakovlev, Nikita Puchkin · 2025

Read Approximation Capabilities of Feedforward Neural Networks with GELU Activations by Konstantin Yakovlev, Nikita Puchkin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21749.txt.

Introduction

Konstantin Yakovlev and Nikita Puchkin determine how well feedforward neural networks with GELU activations can approximate basic functions, together with all their derivatives at once. GELU is the smooth activation used in most modern transformers. Approximation theory usually bounds the error in function values alone; bounding derivatives matters when networks are used inside differential equations.

The authors derive error bounds that hold at once for a function and its derivatives up to any fixed order. The functions covered include polynomials in several variables, the exponential, and the reciprocal. They also report the needed network size, the size of the weights, and the behavior far from the origin. The engine of the proofs is a constructive multiplication scheme valid on growing domains. From it they build division and the exponential, with the approximators' higher derivatives bounded throughout. Such guarantees underpin certified uses in computing.

Catalog record

CREATORKonstantin Yakovlev, Nikita Puchkin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-21749

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Approximation Capabilities of Feedforward Neural Networks with GELU Activations Konstantin Yakovlev, Nikita Puchkin Abstract We derive an approximation error bound that holds simultaneously for a function and all its derivatives up to any prescribed order. The bounds apply to elementary functions, including multivariate polynomials, the exponential function, and the reciprocal function, and are obtained using feedforward neural networks with the Gaussian Error Linear Unit (GELU) activation. In addition, we report the network size, weight magnitudes, and behavior at infinity. Our analysis begins with a constructive approximation of multiplication, where we prove the simultaneous validity of error bounds over domains of increasing size for a given approximator. Leveraging this result, we obtain approximation guarantees for division and the exponential function, ensuring that all higher‑order derivatives of the resulting approximators remain globally bounded. 1 Introduction We investigate simultaneous approximation of multivariate functions and their higher-order derivatives using deep feedforward neural networks. Since approximating derivatives necessitates a smooth activation function, we employ networks with infinitely differentiable the Gaussian Error Linear Unit (GELU) ( Hendrycks and Gimpel 2016 ) . Our choice is motivated by the fact that higher‑order derivatives of the GELU activation can be expressed in terms of Hermite polynomials, which enables simple and tractable bounds on their absolute values. In addition, this activation function is widely adopted in state-of-the-art large language models ( Devlin et al. 2019 ; Raffel et al. 2020 ; Shoeybi et al. 2019 ) , which further underscores its practical relevance. The core of our constructive approach is the localized approximation of polynomials. While prior works have developed approximation theory for smooth functions and their derivatives on fixed compact sets ( Yarotsky 2017 ; De Ryck et al. 2021 ; Gühring and Raslan 2021 ; Belomestny et al. 2023 ) , they do not provide error bounds beyond the original domain nor offer simultaneous guarantees across a sequence of increasingly large domains. Despite recent advances in approximation of functions with noncompact domain presented in Schwab and Zech 2021 ; van Nuland 2024 ; Abdeljawad and Dittrich 2024 , the results either do not focus on the simultaneous approximation of derivatives or impose strong assumptions on the weight function of the underlying weighted L^{p} space ( Abdeljawad and Dittrich 2024 ) . We bridge this gap by providing explicit control on how approximation errors for fundamental operations (like multiplication) scale as the domain size grows. This allows us to construct an approximation of monomials with globally bounded higher‑order derivatives. This properties are relevant in approximation of functions with unbounded domains including generative modelling ( Oko et al. 2023 ; Tang and Yang 2024 ; Azangulov et al. 2024 ; Yakovlev and Puchkin 2025 ; Fukumizu et al. 2025 ) and physics-informed neural networks ( Abdo et al. 2024 ; Alejo et al. 2024 ) . Our key technical innovations are twofold. First, we systematically employ a clipping operation on the network input. By clipping the neural network input, we ensure that the derivatives are globally bounded, since they are bounded on a compact domain. Second, we establish approximation guarantees for partition-of-unity functions in Sobolev seminorms, constructing functions with globally bounded derivatives and light tails. As a consequence of these results, we provide approximation error bounds for the exponential function approximation and division approximation together with its derivatives, ensuring that their higher-order derivatives are globally bounded. Consequently, we extend the approximation results for elementary functions presented in Oko et al. 2023 ; Yakovlev and Puchkin 2025 to Sobolev seminorms on domains of increasing size. Paper structure. The remainder of the paper is structured as follows. Section 2 establishes necessary preliminaries and notations. In Section 3 , we present our main result on approximation error bounds. Proofs not included in the main text are provided in the Appendix. Notation. The set of non-negative integers is denoted by \mathbb{Z}_{+}=\{0,1,2,\dots\} . A multi-index {\mathbf{k}}\in\mathbb{Z}_{+}^{d} , where d\in\mathbb{N} , is denoted in bold. We also define |{\mathbf{k}}|=k_{1}+k_{2}+\ldots+k_{d} , {\mathbf{k}}!=k_{1}!\cdot k_{2}!\cdot\ldots\cdot k_{d}! For a vector v\in\mathbb{R}^{d} we define v^{\mathbf{k}}=v_{1}^{k_{1}}v_{2}^{k_{2}}\ldots v_{d}^{k_{d}} . For a function f of d variables, its weak derivative with respect to the multi-index {\mathbf{k}}\in\mathbb{Z}_{+}^{d} is denoted as \displaystyle\partial^{\mathbf{k}}f=\frac{\partial^{|{\mathbf{k}}|}f}{\partial x_{1}^{k_{1}}\partial x_{2}^{k_{2}}\ldots\partial x_{d}^{k_{d}}}. Throughout the paper, we employ the notation f\lesssim g to indicate that f={\mathcal{O}}(g) . If f\lesssim g and g\lesssim f , then we write f\asymp g . We frequently replace the expression for \min\{a,b\} and \max\{a,b\} with a\vee b and a\wedge b , respectively. For any x>0 , we define \log(x)=\ln(x\vee e) . 2 Preliminaries and notations Norms. We denote the Euclidean norm of a vector v as \|v\| , the maximal absolute value of its entries as \|v\|\infty , and the number of its non-zero entries as \|v\|_{0} . Similarly, \|A\|\infty and \|A\|_{0} represent the maximal absolute value of entries and the number of non-zero entries of matrix A , respectively. Finally, for a set \Omega\subseteq\mathbb{R}^{r} and a function f:\Omega\to\mathbb{R}^{d} , we define \displaystyle\|f\|_{L^{\infty}(\Omega)}=\operatornamewithlimits{esssup}_{x\in\Omega}\|f(x)\|. Smoothness spaces. We introduce the Sobolev space to characterize the regularity of functions in our analysis, and its definition is provided below. Definition 2.1 (Sobolev space) . Let \Omega\subseteq\mathbb{R}^{r} be an open set, and let k\in\mathbb{Z}_{+} . Then, the Sobolev space W^{k,\infty}(\Omega) is defined as follows: \displaystyle W^{k,\infty}(\Omega)=\{f\in L^{\infty}(\Omega):\partial^{\mathbf{k}}f\in L^{\infty}(\Omega)\quad\text{for every }{\mathbf{k}}\in\mathbb{Z}_{+}^{r}\text{ with }|{\mathbf{k}}|\leqslant k\}. Here, L^{\infty}(\Omega) is the Lebesgue space. We define the Sobolev seminorm on W^{k,\infty}(\Omega) as \displaystyle|f|_{W^{k,\infty}(\Omega)}=\max_{\begin{subarray}{c}{\mathbf{k}}\in\mathbb{Z}_{+}^{r},\;|{\mathbf{k}}|=k\end{subarray}}\|\partial^{\mathbf{k}}f\|_{L^{\infty}(\Omega)}. Finally, we define the Sobolev norm on W^{k,\infty}(\Omega) as \displaystyle\|f\|_{W^{k,\infty}(\Omega)}=\max_{0\leqslant m\leqslant k}|f|_{W^{m,\infty}(\Omega)}. Neural networks. In this paper, we focus on feed-forward neural networks employing the Gaussian Error Linear Unit (GELU) activation function: \displaystyle\mathrm{GELU}(x)=x\cdot\Phi(x),\quad\Phi(x)=\frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{x}e^{-t^{2}/2}{\mathrm{d}}t. The choice of GeLU is motivated by its infinite smoothness and bounded derivatives (see Lemma B.2 ). For a vector b=(b_{1},\dots,b_{r})\in\mathbb{R}^{r} , we define the shifted activation function \mathrm{GELU}_{b}:\mathbb{R}^{r}\to\mathbb{R}^{r} as \displaystyle\mathrm{GELU}_{b}(x)=(\mathrm{GELU}(x_{1}-b_{1}),\dots,\mathrm{GELU}(x_{r}-b_{r})),\quad x=(x_{1},\dots,x_{r})\in\mathbb{R}^{r}. Given a network depth L\in\mathbb{N} and a vector of layer sizes W=(W_{0},W_{1},\dots,W_{L})\in\mathbb{N}^{L+1} . Then, a neural network of depth L and architecture W is a function f:\mathbb{R}^{W_{0}}\to\mathbb{R}^{W_{L}} such that \displaystyle f(x)=-b_{L}+A_{L}\circ\mathrm{GELU}_{b_{L-1}}\circ A_{L-1}\circ\mathrm{GELU}_{b_{L-2}}\circ\dots\circ A_{2}\circ\mathrm{GELU}_{b_{1}}\circ A_{1}\circ x, (1) where A_{j}\in\mathbb{R}^{W_{j}\times W_{j-1}} is a weight matrix and b_{j}\in\mathbb{R}^{W_{j}} is a bias vector for all j\in{1,\dots,L} . The maximum number of neurons of each layer is given by \|W\|_{\infty} and is reffered to as the width of the neural network. We define the class \mathsf{NN}(L,W,S,B) of neural networks of the form ( 1 ) with at most S non-zero weights and the weight magnitude B as follows: \displaystyle\mathsf{NN}(L,W,S,B)=\left\{f\text{ of the form }\eqref{eq:feed_forward_nn_def}\,:\,\sum_{j=1}^{L}(\|A_{j}\|_{0}+\|b_{j}\|_{0})\leqslant S,\;\max_{1\leqslant j\leqslant L}\|A_{j}\|_{\infty}\vee\|b_{j}\|_{\infty}\leqslant B\right\}. 3 Main results This section presents our main results. Specifically, Subsection 3.1 details approximation error bounds for elementary operations, including the identity function, partition of unity, and square operation. Subsection 3.2 elaborates on the approximation of monomials. Finally, Subsection 3.3 provides approximation error bounds for the exponentiation and division operations. 3.1 Approximation of elementary operations Passing the output of one layer to a non-adjacent layer is frequently beneficial. Note that ReLU activation allows for an exact identity mapping ( Nakada and Imaizumi 2020a ) . However, in the case of GELU, an approximate mapping is guaranteed, as demonstrated by the following lemma, which provides an approximation error bound for a single-layer neural network. Lemma 3.1 (approximation of identity operation) . Let m\in\mathbb{N} and let \mathrm{id}(x)=x . Then, for any \varepsilon\in(0,1) there exists \varphi_{id}\in\mathsf{NN}(L,W,S,B) satisfying \displaystyle\|\varphi_{id}-\mathrm{id}\|_{W^{m,\infty}([-C,C])}\leqslant C^{2}\varepsilon,\quad\text{for all }C\geqslant

Read the complete text