Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability

Liangwei Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen · 2025

Read Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability by Liangwei Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09283.txt.

Introduction

A new kind of neural network builds its curves from spline pieces whose joints can move during training. Liangwei Nathan Zheng and co-authors analyze those moving joints and propose a leaner design in a 2025 paper.

The team first works out how many joints such networks need, setting firm lower and upper bounds. Their free-knots design then matches the parameter count of a classic network while keeping the spline advantage. A training trick keeps the learned curves smooth to a higher order than the original design, and widening ranges steadies training. Across eight data sets spanning images, text, and time series, the method performed well. The work makes the young architecture cheaper and calmer to train.

Catalog record

CREATORLiangwei Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability Liangwei Nathan Zheng, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen Abstract Kolmogorov-Arnold Neural Networks (KANs) have gained significant attention in the machine learning community. However, their implementation often suffers from poor training stability and heavy trainable parameter. Furthermore, there is limited understanding of the behavior of the learned activation functions derived from B-splines. In this work, we analyze the behavior of KANs through the lens of spline knots and derive the lower and upper bound for the number of knots in B-spline-based KANs. To address existing limitations, we propose a novel Free Knots KAN that enhances the performance of the original KAN while reducing the number of trainable parameters to match the trainable parameter scale of standard Multi-Layer Perceptrons (MLPs). Additionally, we introduce new a training strategy to ensure C 2 C^{2} continuity of the learnable spline, resulting in smoother activation compared to the original KAN and improve the training stability by range expansion. The proposed method is comprehensively evaluated on 8 datasets spanning various domains, including image, text, time series, multimodal, and function approximation tasks. The promising results demonstrates the feasibility of KAN-based network and the effectiveness of proposed method. Our code is openly available: https://github.com/IcurasLW/FR-KAN.git . † † affiliationnotice: Equal contribution 1 Introduction Multi-Layer Perception(MLP) is the dominated architecture in current deep network, which is a well-known as piecewise linear spline operator f_{\Theta}:\mathbb{R}^{q}\rightarrow\mathbb{R}^{p} that maps an input signal \mathbf{x}\in\mathbb{R}^{q} to an output \mathbf{y}\in\mathbb{R}^{p} ( Balestriero & Baraniuk, 2018 ; Balestriero et al., 2018 ; Telgarsky, 2016 ) . The learning process aligns with Universal Approximation Theorem(URT) ( Hornik et al., 1989 ) as each neuron contribute one knot to the final piecewise linear spline at a learnable position. This power enables MLP to approximate any continuous function by sufficient number of neurons. Recently, Kolmogorov–Arnold Networks(KANs) ( Liu et al., 2024 ) has attracted significant attention of the community due to high interpretability and the inspiring learning paradigm. Unlike MLP, KANs follow the Kolmogorov-Arnold Representation Theorem(KAT), which states that every multivariate continuous function f:[0,1]^{n}\rightarrow\mathbb{R} can be represented as a superposition of continuous single variable functions ( Sprecher & Draghici, 2002 ; Liu et al., 2024 ; Schmidt-Hieber, 2021 ; Leni et al., 2013 ) . The architecture with B-spline KAN can be considered viewed as fixed knots spline operator where the knots are not fully learnable, limiting the expressive power of KAN. In this paper, we aim to bring a new insight to reveal behavior of KAN from spline knots perspective and analyze the bound for the spline knots of KAN. Current community consistently believe that KAN is weaker than MLP in typical deep learning task by preliminary experiment. However, our analysis and comprehensive experiments show that multilayer and shallow KAN network actually outperform MLP under the same architecture setting, shedding light on why KANs often outperform traditional Multi-Layer Perceptrons (MLPs) across various tasks. KANs assume that the target function is the composition of smooth and uniform function, which can be expressed as typical basis function like e^{x} , log(x) and x^{n} . This makes KANs well-fit on scientific task such as function approximation and symbolic regression, significantly winning MLP on AI-for-sciencne tasks. Despite this, the original implementation is trivial and still infeasible for typical deep learning tasks such as image classification. We hypothesis that the uniform grid assumption is difficult to fit finite observation dataset from real world distribution and the scalability of KAN is also one of the obstacles for common machine learning task. There are still some challenges faced by original KANs: (1) Constrained Expressive Power : The expressive ability of KAN is constrained by the fixed grid segment. Neural networks are known as learnable spline operator from the approximation theory perspective ( Balestriero & Baraniuk, 2018 ; Balestriero et al., 2018 ; Telgarsky, 2016 ) . Original KAN employed learnable fixed grid B-spline function as the activation. This fixed knots setting can constraint the development of knots, resulting less expressive power of network. The fixed grid does not generate more spline knots even if more neurons are added. We show a bound of knots for the original KAN (Theorem 4.3 ) that the spline knots learned from the original KAN only correlated to grid size, spline order and layers of networks. This bound inspires us to unleash the power of KAN from fixed knots and bring new insights to understand the behavior of KAN. (2) Heavy Trainable Parameter : The original KANs learn a unique activation function for each neuron, which introduces additional parameters proportional to G+K for the spline combination compared to a standard MLP. This results in a parameter complexity of \mathcal{O}((d_{in}\times d_{out})(G+K+1)) , whereas a standard MLP has a parameter complexity of \mathcal{O}((d_{in}\times d_{out})+d_{out}) . (3) Oscillation and Unstable Training : The original KAN implementation constrains the activation function within a narrow range (e.g., [-1, 1]) and no discussion on the impacts of grid range, this constraint becomes problematic as the grid size increase, leading to densely packed grid points within the grid range. Consequently, this exacerbates oscillation and instability, particularly when stacking additional layers, often resulting in unstable training and NaN values in the output. Such oscillation limit the expressive power of the network by restricting the grid size and spline order. We demonstrate the underlying causes of activation oscillation through preliminary experiments shown in Figure 1 . To address the above problems, we propose FR ee-knots KAN ( FR-KAN ) motivated by the bound of spline knots of KAN. FR-KAN aims to release the constraint of fixed knots and significantly reduce the trainable parameter size to MLP level by disentangling the grid vector from number of neuron. We also release the constraints causing serious oscillation by regularizing the second derivatives of spline weight and enlarge the grid range. This makes KAN network to have more grid size than the original KAN for smoother fitting and stable training. Theoretically, we show the spline knot bound of KAN only correlated to grid size G , spline order K and network layer L , which provides theoretical insights for understanding behavior of KAN network with B-spline. This insight can also extend to other KAN variants. We summarize our main contributions as below: (C1) Knots Upper Bound: Derive a tight bound for spline knots in KAN to understand the behavior of KAN and to identify potential factors that may impact performance. (C2) Free Grid Spline: Extend KAN by replacing the fixed-grid constraint with a free-knots grid approach, enabling more adaptable spline configurations to enhance expressive power by the non-uniform grid. (C3) Stability Improvements: Address the severe oscillation issues inherent in the original KAN by implementing second derivative regularization, resulting in smoother training dynamics and improved overall performance. (C4) Comprehensive Evaluation: Implement fair experiments on 9 datasets including image, text, time series, multimodal, and function approximation, showcasing superior performance compared to MLP, KAN in terms of comparable parameter and performance. 2 Related Works Kolmogorov-Arnold Networks (KANs) is firstly proposed by ( Liu et al., 2024 ) for scientific tasks such as symbolic regression. The promising performance and interpretability of KANs arouse active discussion in the machine learning community. Many

Read the complete text