A General Method for Proving Networks Universal Approximation Property
Wei Wang · 2025
Read A General Method for Proving Networks Universal Approximation Property by Wei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07857.txt.
Introduction
Wei Wang proves a universal fitting result once for many network designs. Deep designs keep multiplying, and each new one usually needs its own proof that it can approximate any sensible function. Split proofs waste effort and hide what the designs share.
A general method supplies a common foundation across fully connected networks, convolutional networks, Transformers, and more. New designs can inherit the result by checking shared conditions rather than starting over. Unified theory helps researchers see which parts grant expressive power. That clarity can steer design toward features that matter.
Catalog record
| CREATOR | Wei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07857
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A General Method for Proving Networks Universal Approximation Property Wei Wang Abstract Deep learning architectures are highly diverse. To prove their universal approximation properties, existing works typically rely on model-specific proofs. Generally, they construct a dedicated mathematical formulation for each architecture (e.g., fully connected networks, CNNs, or Transformers) and then prove their universal approximability. However, this approach suffers from two major limitations: first, every newly proposed architecture often requires a completely new proof from scratch; second, these proofs are largely isolated from one another, lacking a common analytical foundation. This not only incurs significant redundancy but also hinders unified theoretical understanding across different network families. To address these issues, this paper proposes a general and modular framework for proving universal approximation. We define a basic building block (comprising one or multiple layers) that possesses the universal approximation property as a Universal Approximation Module (UAM). Under this condition, we show that any deep network composed of such modules inherently retains the universal approximation property. Moreover, the overall approximation process can be interpreted as a progressive refinement across modules. This perspective not only unifies the analysis of diverse architectures but also enables a step-by-step understanding of how expressive power evolves through the network. 1 Introduction In recent years, deep learning has achieved remarkable success in fields such as computer vision and natural language processing. However, this success has largely relied on empirical validation and engineering optimization rather than solid theoretical foundations. The fundamental reason for this gap lies in the high complexity of deep neural network architectures, nonlinearity, high dimensionality, and hierarchical coupling make it extremely challenging to rigorously characterize their behavior using formal mathematics. From a theoretical perspective, the effectiveness of deep learning is widely attributed to its powerful function approximation capability. To further enhance model expressivity and generalization performance, researchers have proposed a variety of network architectures, including fully connected neural networks (FCNNs) 3 ; 7 , convolutional neural networks (CNNs) 9 ; 4 , and more recently, the widely adopted Transformer 10 ; 2 architecture. Although these models differ significantly in design motivation or application scenarios, they share a common goal: to effectively approximate complex functions through specific parameterized structures. Consequently, a central direction in the theoretical study of deep learning is the systematic analysis of the approximation capacity of different network architectures. Existing works 1 ; 5 ; 6 have established universal approximation theorem (UAT). Recently, model-specific UATs have been developed for architectures of CNNs 12 ; 8 and Transformers 11 , in order to account for their distinct structural properties. A typical research paradigm begins by analyzing single-hidden-layer architectures, proving that they can approximate any continuous function under certain conditions, and then gradually extends these results to multi-layer or deep settings. Such approaches usually treat the entire network as a single mathematical mapping and construct a formal approximation framework to demonstrate its universal approximation property. While these studies have made significant contributions to understanding the theoretical principles of deep learning, two key limitations remain. First, existing theories are highly architecture-specific and lack generality. When a new architecture emerges, researchers must rebuild the mathematical model from scratch and repeat complex proofs, making it difficult to develop a unified analytical framework. Second, these methods, although capable of proving approximation power by abstracting the entire network as a "black-box" composite function, fail to reveal the internal learning dynamics or information evolution mechanisms across layers or modules. Precisely, the insights are needed to understand model behavior and guide architectural design and optimization. To address these challenges, this paper proposes a more general theoretical framework. Our core idea is to move beyond the specifics of individual network architectures and instead take universal approximation property (UAP) as a foundational assumption. We abstract any substructure within a network, such as a single layer or a multi-layer block, that possesses this property into a Universal Approximation Module (UAM). Building upon this abstraction, we develop a modular theory of approximation that not only provides a unified proof of the overall universal approximation capability for diverse network architectures but also enables a step-by-step analysis of how information and representations evolve throughout the approximation process. The contribution of this paper as following: • This paper builds up a general approch for proving the universal approximation property of a neural network. It eliminates dependence on specific architectural details. As long as a component is proven to be universally approximating (regardless of its internal structure), it can be seamlessly integrated into our framework for compositional analysis. • This paper give the proof from the modular perspective. It allows us to trace the internal approximation pathway of the network, offering a theoretical lens for understanding the inner workings of deep models. This not only shedding light on how representations evolve across modules but also provides an extensible analytical paradigm for the design and theoretical validation of future neural network models. 2 Theoretical Analysis of Progressive Approximation in Module Networks 2.1 Problem Setup Let the input space \mathcal{X}\subseteq\mathbb{R}^{m} be a compact set, and the target function f:\mathcal{X}\to\mathbb{R}^{m} be continuous. Consider a neural network architecture composed of N cascaded learnable modules: \mathbf{x}_{i}=H_{i}(\mathbf{x}_{i-1}),\quad i=1,2,\dots,N, where the initial input is \mathbf{x}_{0}\in\mathcal{X} , each mapping H_{i}:\mathbb{R}^{m}\to\mathbb{R}^{m} belongs to some function class \mathcal{H}_{i} , and all intermediate representations satisfy \mathbf{x}_{i}\in\mathbb{R}^{m} . The overall prediction function is defined as the composition: \hat{f}:=H_{N}\circ H_{N-1}\circ\cdots\circ H_{1}, so that \hat{f}(\mathbf{x}_{0})=\mathbf{x}_{N} . The central question of this paper is how such a cascaded structure can approximate the continuous function f to arbitrary accuracy? More precisely, we aim to establish an approximation property where, for any \varepsilon>0 , there exists a positive integer N and a sequence of functions \{H_{i}^{*}\}_{i=1}^{N} (with H_{i}^{*}\in\mathcal{H}_{i} ) such that \sup_{\mathbf{x}_{0}\in\mathcal{X}}\|\hat{f}(\mathbf{x}_{0})-f(\mathbf{x}_{0})\|<\varepsilon. 2.2 Key Assumptions and Preliminaries We begin with a standard expressiveness assumption on each module. Assumption 1 (Universal Approximation Property, UAP). Each continuous function class \mathcal{H}_{i} is dense in C(\mathcal{K}_{i},\mathbb{R}^{m}) for any non-empty compact set \mathcal{K}_{i}\subset\mathbb{R}^{m} . That is, for any continuous g:\mathcal{K}_{i}\to\mathbb{R}^{m} and \varepsilon>0 , there exists H_{i}^{*}\in\mathcal{H}_{i} such that \sup_{\mathbf{x}_{i-1}\in\mathcal{K}_{i}}\|H_{i}^{*}(\mathbf{x}_{i-1})-g(\mathbf{x}_{i-1})\|<\varepsilon. This condition ensures that each module has sufficient expressive power and serves as the foundation for constructing the approximation path. Notably, many common model classes satisfy this property, such as sufficiently wide feedforward neural networks 1 ; 5 ; 6 . Property (Compactness Preservation). If a function H_{i}:\mathbb{R}^{m}\to\mathbb{R}^{m} is continuous and \mathcal{K}_{i-1}\subset\mathbb{R}^{m} is compact, then the image H_{i}(\mathcal{K}_{i-1}) is also compact. This property follows directly from the fact that continuous maps preserve compactness. Therefore, it is not an additional assumption but an inherent property of continuous functions. In this work, we repeatedly use this property to ensure the compactness of input and output sets at each module, thereby allowing the application of universal approximation theorems at every step. Condition (Target Recoverability, TR). At module i , we define the encoding map at step i as: \phi_{i}:=H_{i}\circ\cdots\circ H_{1},\quad\text{for }i=1\cdots n. For the final output \mathbf{x}_{N}=\hat{f}(\mathbf{x}_{0}) to approximate f(\mathbf{x}_{0}) , it is essential that the information about f(\mathbf{x}_{0}) is presered throughout the intermediate representations. To this end, we propose the following necessary condition: for any \mathbf{x}_{0}^{1},\mathbf{x}_{0}^{2}\in\mathcal{X} and any i=1\cdots N , if \phi_{i}(\mathbf{x}_{0}^{1})=\phi_{i}(\mathbf{x}_{0}^{2}) , then f(\mathbf{x}_{0}^{1})=f(\mathbf{x}_{0}^{2}) must hold. In other words, the equivalence relation induced by \phi_{i} must not be coarser than that induced by f .