A note on Cybenko’s Universal Approximation Theorem

Kun Wang · 2025

Read A note on Cybenko’s Universal Approximation Theorem by Kun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18893.txt.

Introduction

Kun Wang published a short note in 2025 on a famous result in neural network theory, Cybenko's universal approximation theorem. That theorem says a simple network can approximate any continuous function, and its proof has been cited for decades. Wang points out a mistake in Cybenko's proof of his version of the theorem.

The note warns that the mistake may not be easy to fix along the lines of the original argument. It also shows that the error leads to an interesting open question in measure theory. The piece is brief and aimed at specialists who rely on the classic proof.

Catalog record

CREATORKun Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A note on Cybenko’s Universal Approximation Theorem Kun Wang Abstract . In this short note, we point out a mistake in G.Cybenko’s proof of his version of the universal approximation theorem which has been widely cited. This mistake might not be easily fixable along the idea of his proof and it also leads to an interesting question in measure theory. 1. Introduction The universal approximation theorem serves as a mathematical foundation, ensuring that neural networks with at least one hidden layer and arbitrary width can be used to approximate any continuous functions to any degree of accuracy. In late 1980s and early 1990s, various versions of the universal approximation theorem have been proven by various authors. See [ 6 ] for a survey. Among these works, G.Cybenko’s [ 2 ] version of the universal approximation theorem is very widely cited (more than 19000 Goolge citations so far). His proof of the theorem makes nice use of the Hahn-Banach Theorem and the Riesz representation theorem. However, his proof contains a mistake which might not be easily fixable along the idea of his argument. As his work is very influential, we feel there is a need to point the mistake out. 2. The Mistake We first recall the definition of a sigmoidal function. Definition 2.1 . A function \sigma:\mathbb{R}\rightarrow\mathbb{R} is called sigmoidal if \sigma(t)\to 1 as t\to+\infty and \sigma(t)\to 0 as t\to-\infty . The following is Cybenko’s version of the universal approximation [ 2 , Theorem 2] : Theorem 2.2 . Let \sigma be any continuous sigmoidal function, I_{n}=[0,1]^{n} be the n -dimensional unit cube and C(I_{n}) be the space of all continuous functions on I_{n} . Then finite sums of the form G(x)=\sum_{j=1}^{N}\alpha_{j}\sigma(y_{j}^{T}x+\theta_{j}) where \alpha_{j},\theta_{j}\in\mathbb{R} are constants, y_{j}\in\mathbb{R}^{n} are constant vectors and y_{j}^{T}x is the inner product of y_{j} and x , are dense in C(I_{n}) . In other words, given any f\in C(I_{n}) and \epsilon>0 , there is a sum, G(x) , of the above form, for which |G(x)-f(x)|<\epsilon\ \ \ \ \ \ \ \text{for\ all}\ x\in I_{n}. Remark 2.3 . The above universal approximation theorem has been generalized by Kurt Hornik [ 4 ] to the case where \sigma can be taken to be any continuous, bounded and non-constant function and with I_{n} replaced by any compact subset of \mathbb{R}^{n} . Hornik’s result is further generalized by Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus and Shimon Schocken to its most general form where \sigma can be taken to be any nonpolynomial continuous function. Despite these generalizations, Cybenko’s paper is very influential and very widely cited. We now outline Cybenko’s original proof of the theorem and point out the mistake it contains. Let S\subseteq C(I_{n}) be the subspace of functions of the form G(x) , and R be its closure in C(I_{n}) . We want to show R=C(I_{n}) . Assume not, then by the Hahn-Banach theorem, there is a bounded linear functional L on C(I_{n}) so that L\neq 0 but L(R)=0 . By the Riesz representation theorem, there exists a finite signed regular Borel measure \mu on I_{n} so that L is of the form L(h)=\int_{I_{n}}h(x)d\mu(x),\ \forall h\in C(I_{n}). As \sigma(y^{T}x+\theta)\in R and L vanishes on R , we have \int_{I_{n}}\sigma(y^{T}x+\theta)d\mu(x)=0 for all y\in\mathbb{R}^{n} and \theta\in\mathbb{R} . Now for any y\in\mathbb{R}^{n},\lambda,\theta,\varphi\in\mathbb{R} , consider \sigma_{\lambda}(x)=\sigma(\lambda(y^{T}x+\theta)+\varphi). Then \sigma_{\lambda}(x)\in R and \displaystyle\sigma_{\lambda}(x)\begin{cases}\to 1\ \ \ \ \ \ \ \ \text{for}\ \ \ y^{T}x+\theta>0\ \ \ \text{as}\ \lambda\to+\infty\\ \to 0\ \ \ \ \ \ \ \ \text{for}\ \ \ y^{T}x+\theta<0\ \ \ \text{as}\ \lambda\to+\infty\\ =\sigma(\varphi)\ \ \ \ \text{for}\ \ \ y^{T}x+\theta=0\ \ \ \text{for\ all}\ \lambda\\ \end{cases} Thus \sigma_{\lambda}(x) converge pointwise to the function \displaystyle\gamma(x)=\begin{cases}1\ \ \ \ \ \ \ \ \text{for}\ \ \ y^{T}x+\theta>0\\ 0\ \ \ \ \ \ \ \ \text{for}\ \ \ y^{T}x+\theta<0\\ \sigma(\varphi)\ \ \ \ \text{for}\ \ \ y^{T}x+\theta=0\end{cases} Let \Pi_{y,\theta} be the hyperplane defined by \{x\in I_{n}\ |\ y^{T}x+\theta=0\} and let H_{y,\theta} be the open half-space defined by \{x\in I_{n}\ |\ y^{T}x+\theta>0\} , then as the functions \sigma_{\lambda}(x) are uniformly bounded, the Bounded Convergence Theorem gives \displaystyle 0 \displaystyle=\int_{I_{n}}\sigma_{\lambda}(x)d\mu(x) \displaystyle=\int_{I_{n}}\gamma(x)d\mu(x) \displaystyle=\sigma(\varphi)\mu(\Pi_{y,\theta})+\mu(H_{y,\theta}) for all \varphi,y,\theta . Therefore \mu(\Pi_{y,\theta})=0,\mu(H_{y,\theta})=0 for all y,\theta . Next, Cybenko proceeds to show that this implies \mu is the zero measure, which would imply the linear functional L is zero and thus obtain a contrdiction. To show \mu is the zero measure, Cybenko argues as follows where he makes a mistake. Fix y . Define a bounded linear functional F on L^{\infty}(\mathbb{R}) , the space of all essentially bounded measurable functions on \mathbb{R} , by F(h)=\int_{I_{n}}h(y^{T}x)d\mu(x),\ \ \forall h\in L^{\infty}(\mathbb{R}). Take h to be the indicator function of the interval [\theta,\infty) , we obtain F(h)=\int_{I_{n}}h(y^{T}x)d\mu(x)=\mu(\Pi_{y,-\theta})+\mu(H_{y,-\theta})=0. Similarly, F(h)=0 if h is the indicator function of the open interval (\theta,\infty) . By linearity, F(h)=0 for the indicator function of any interval. Cybenko then claims that F(h)=0 for any simple function h , and hence F=0 as simple functions are dense in L^{\infty}(\mathbb{R}) . Finaly, from F=0 , Cybenko deduces that \mu=0 by taking h(t)=\exp(it)=\cos t+i\sin t,t\in\mathbb{R} , as the Fourier transform of \mu is zero implies \mu itself must be zero. However, Cybenko’s argument for F=0 is wrong. First, there is a misuse of terminoloy. In Cybenko’s terminology, a simple function defined on \mathbb{R} is a finite linear combination of indicator functions of intervals. But this is not the usual definition of simple functions. Cybenko’s simple functions are really what we call step functions , and the usual definition of simple functions are those who are finite linear combinations of indicator functions of measurable subsets of \mathbb{R} , not merely indicator functions of intervals. Therefore, with the corrected terminologies, step functions are simple functions, but simple functions are not merely step functions. What Cybenko really proves is that F(h)=0 for all step functions h , not for all simple functions h . This does not imply F=0 , as step functions are not dense in L^{\infty}(\mathbb{R}) (while simple functions are do dense in L^{\infty}(\mathbb{R}) ). For example, consider the function f(x)=\sin x,\ x\in\mathbb{R} , then f\in L^{\infty}(\mathbb{R}) , but for any step function s defined on \mathbb{R} , we have ||f-s||_{\infty}\geq\frac{1}{2}. To see this, we write s=\sum_{i=1}^{k}a_{i}\chi_{I_{i}}, where a_{i}\in\mathbb{R} , I_{i} are intervals which are disjoint, and \chi_{I_{i}} is the indicator function of I_{i} . If one of the intervals, say I_{i} , is infinite, then \max_{x\in I_{i}}|f(x)-a_{i}\chi_{I_{i}}(x)|\geq\max\{|a_{i}-0|,|a_{i}-1|\}\geq\frac{1}{2}, which imply ||f-s||_{\infty}\geq\frac{1}{2}. If all of the intervals I_{i} are finite, then s=0 outside a finite interval, which imply ||f-s||_{\infty}\geq 1. In any case, we have shown that ||f-s||_{\infty}\geq\frac{1}{2} for any step function s , which imply step functions are not dense in L^{\infty}(\mathbb{R}) . The above argument also shows that the cosine and sine funcntions can not be approximated by step functions in L^{\infty}(\mathbb{R}) , hence the Fourier transform argument of Cybenko does not go through anyway. 3. Further Discussion Cybenko’s mistaken argument leaves the following interesting question open: Question 3.1 . Let \mu be a finite signed regular Borel measure on the unit cube I_{n}\subseteq\mathbb{R}^{n} satisfying \mu(\Pi_{y,\theta})=0,\mu(H_{y,\theta})=0,\ \forall y\in\mathbb{R}^{n},\theta\in\mathbb{R}, where \Pi_{y,\theta} and H_{y,\theta} are the hyperplanes and open half spaces respectively as defined previously. Is it true that \mu is the zero measure? An affirmative answer to the above question would

Read the complete text