Sinusoidal Approximation Theorem for Kolmogorov-Arnold Networks

Sergei Gleyzer, Hanh Nguyen, Dinesh P. Ramakrishnan, Eric A. F. Reinhardt · 2025

Read Sinusoidal Approximation Theorem for Kolmogorov-Arnold Networks by Sergei Gleyzer, Hanh Nguyen, Dinesh P. Ramakrishnan, Eric A. F. Reinhardt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00247.txt.

Introduction

Sergei Gleyzer and co-authors propose a new variant of Kolmogorov-Arnold Networks in work from 2025. These networks build complex functions from sums of single-variable functions, using learnable curves instead of fixed activation rules.

Their variant replaces both inner and outer functions with weighted sine waves whose frequencies can be learned. Phases are fixed at evenly spaced values, and the authors prove the design is valid in theory. In tests on multi-variable functions, the method beat fixed-frequency Fourier methods and matched standard perceptron networks. The work extends the search for better activation forms in this network family.

Catalog record

CREATORSergei Gleyzer, Hanh Nguyen, Dinesh P. Ramakrishnan, Eric A. F. Reinhardt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sinusoidal Approximation Theorem for Kolmogorov-Arnold Networks Sergei Gleyzer, Hanh Nguyen, Dinesh P. Ramakrishnan, Eric A. F. Reinhardt Abstract The Kolmogorov-Arnold representation theorem [ 15 ] states that any continuous multivariable function can be exactly represented as a finite superposition of continuous single variable functions. Subsequent simplifications of this representation involve expressing these functions as parameterized sums of a smaller number of unique monotonic functions. These developments led to the proof of the universal approximation capabilities of multilayer perceptron networks with sigmoidal activations, forming the alternative theoretical direction of most modern neural networks [ 26 ] . Kolmogorov-Arnold Networks (KANs) have been recently proposed as an alternative to multilayer perceptrons [ 20 ] . KANs feature learnable nonlinear activations applied directly to input values, modeled as weighted sums of basis spline functions. This approach replaces the linear transformations and sigmoidal post-activations used in traditional perceptrons. Subsequent works have explored alternatives to spline-based activations [ 2 , 34 , 24 , 4 , 1 , 33 ] . In this work, we propose a novel KAN variant by replacing both the inner and outer functions in the Kolmogorov-Arnold representation with weighted sinusoidal functions of learnable frequencies. Inspired by simplifications introduced by Lorentz [ 21 ] and Sprecher [ 31 ] , we fix the phases of the sinusoidal activations to linearly spaced constant values and provide a proof of its theoretical validity. We also conduct numerical experiments to evaluate its performance on a range of multivariable functions, comparing it with fixed-frequency Fourier transform methods and multilayer perceptrons (MLPs). We show that it outperforms the fixed-frequency Fourier transform and achieves comparable performance to MLPs. 1 Introduction The quest to represent complex mathematical functions in terms of simpler constituent parts is a central theme in analysis and applied mathematics. A pivotal moment in this endeavor arose from the list of 23 problems presented by David Hilbert in 1900, which profoundly shaped the mathematical landscape of the 20th century. Hilbert’s thirteenth problem, in particular, posed a fundamental question about the limits of function composition. The problem asked whether the solution to the general 7 th degree polynomial equation, considered as a function of its coefficients, a variant of which could be expressed as a finite superposition of continuous functions of only one variable each [ 11 ] . In the late 1950s, Andreĭ Kolmogorov and his graduate student Vladimir Arnold provided a powerful and affirmative answer to this problem. Their work culminated in the Kolmogorov-Arnold Representation Theorem (KART) [ 15 ] , which states that any multivariable continuous function f defined on a compact domain, such as the n -dimensional unit cube, can be represented exactly by a finite superposition of single variable functions and the single binary operation of addition. The canonical form of the theorem is expressed as: f(x_{1},\dots,x_{n})=\sum_{q=0}^{2n}\phi_{q}\left(\sum_{p=1}^{n}\psi_{p,q}(x_{p})\right), (1) where \psi_{p,q}:\mathbb{R}\to\mathbb{R} are the inner functions and the \phi_{q}:\mathbb{R}\to\mathbb{R} are the outer functions which are assumed to be continuous. Lorentz [ 21 ] and Sprecher [ 31 ] showed that the functions \phi_{q} can be replaced by only one function \phi and that the functions \psi_{p,q} by \lambda^{pq}\psi_{q} , where \lambda is a constant and \psi_{q} are monotonic and Lipschitz-continuous functions. Perceptrons were introduced around the same time by Frank Rosenblatt [ 25 ] for simple classification tasks. However, their limitations were famously highlighted in the 1969 book ”Perceptrons” by Marvin Minsky and Seymour Papert [ 22 ] . They proved that a single-layer perceptron could not solve problems that were not linearly separable, such as the simple XOR logical function. This critique led to a significant decline in neural network research for over a decade, often termed the first ”AI winter.” The revival came in the 1980s with the popularization of the Multilayer Perceptron (MLP), which were introduced in the late 1960s [ 3 ] . By adding one or more ”hidden” layers between the input and output layers, MLPs could overcome the limitations of the single-layer perceptron. The key breakthrough that made these deeper networks practical was the backpropagation algorithm. While developed earlier, its popularization by David Rumelhart, Geoffrey Hinton, and Ronald Williams in 1986 [ 27 ] provided an efficient method to train the weights in these new, multilayered networks, sparking a renewed wave of interest in the field. With MLPs demonstrating practical success, the next crucial step was to understand their theoretical capabilities. The Universal Approximation Theorem provides this foundation, showing that a standard MLP with just one hidden layer can, in principle, approximate any continuous function to any desired degree of accuracy. George Cybenko [ 6 ] used tools from functional analysis like Hahn-Banach theorem and Riesz Representation theorem and showed that given any multivariable continuous function in a compact domain, there exists values of the weights and biases of a multilayer perceptron with sigmoidal model whose loss with respect to the function is bound by any given positive value \epsilon . Almost concurrently, Kurt Hornik, Maxwell Stinchcombe, and Halbert White [ 12 ] provided a different, more general proof for the universal approximation property of a multilayer perceptron with any bounded, non-constant, and continuous activation function. The applicability of KART to continuous functions in a compact domain was first highlighted by Hecht-Nielsen [ 9 , 10 ] . However, the functions constructed in Kolmogorov’s proofs as well as in their later simplifications or improvements are highly complex and non-smooth, which were very different from the much simpler activation functions used in MLPs [ 8 ] . Soon, a series of papers by Věra Kůrková [ 16 , 17 ] showed that the one-dimensional inner and outer functions could be approximated using 2 layer MLPs and obtained a relationship between the number of hidden units and the loss based on the properties of the function being approximated. It was not until much later in the 2020s that KART had a direct practical application in the domain of neural networks and machine learning. Kolmogorov-Arnold networks (KANs) were first proposed by Liu et al. [ 19 , 20 ] as an alternative to MLPs which demonstrated interpretable hidden activations and higher regression accuracy. They used learnable basis splines to model the constituent hidden values similar to KART, and their stacked representation is simplified compared to the original form used in KART. Subsequent works retained the same representation but replaced splines with other series of functions such as radial basis functions and Chebyshev polynomials [ 18 , 23 , 32 , 35 ] which were computationally faster and demonstrated superior scaling properties with larger numbers of learnable parameters. However, issues of speed and numerical stability on smaller floating point types remain, especially compared to the well-established MLPs [ 28 , 36 ] . To address these challenges, we propose a variant of the representation theorems by Lorentz [ 21 ] and Sprecher [ 31 ] . We set the phases of the sinusoidal activations to linearly spaced constant values and establish its mathematical foundation to confirm its validity. Previous work has explored this approximation series compared to MLPs and basis-spline approximations and showed competitive performance on the inherently discontinuous domain task of labeling hand-written numerical characters [ 24 ] . We extend this work by providing a constructive proof of the

Read the complete text