Catastrophic Forgetting in Kolmogorov-Arnold Networks

Mohammad Marufur Rahman, Guanchu Wang, Kaixiong Zhou, Minghan Chen, Fan Yang · 2025

Read Catastrophic Forgetting in Kolmogorov-Arnold Networks by Mohammad Marufur Rahman, Guanchu Wang, Kaixiong Zhou, Minghan Chen, Fan Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12828.txt.

Introduction

Mohammad Marufur Rahman and four co-authors examine whether Kolmogorov-Arnold Networks forget less than ordinary networks. These networks use learnable spline activations, localized by design, which led many to hope they would resist catastrophic forgetting, the tendency of models to erase old tasks while learning new ones. The authors build a theory linking forgetting to how much activation supports overlap and to the intrinsic dimension of the data. They then test the theory on synthetic and vision tasks.

The hope holds only in low-dimensional algorithmic settings. On image classification and language modeling, KANs forget much like other models. The team also introduces KAN-LoRA, an adapter for efficient continual fine-tuning of language models, and tests it on knowledge editing. Their conclusion is measured: localized activations are not a general cure. Designers get a clearer map of when this architecture helps.

Catalog record

CREATORMohammad Marufur Rahman, Guanchu Wang, Kaixiong Zhou, Minghan Chen, Fan Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Catastrophic Forgetting in Kolmogorov-Arnold Networks Mohammad Marufur Rahman, Guanchu Wang, Kaixiong Zhou, Minghan Chen, Fan Yang Abstract Catastrophic forgetting is a longstanding challenge in continual learning, where models lose knowledge from earlier tasks when learning new ones. While various mitigation strategies have been proposed for Multi-Layer Perceptrons (MLPs), recent architectural advances like Kolmogorov-Arnold Networks (KANs) have been suggested to offer intrinsic resistance to forgetting by leveraging localized spline-based activations. However, the practical behavior of KANs under continual learning remains unclear, and their limitations are not well understood. To address this, we present a comprehensive study of catastrophic forgetting in KANs and develop a theoretical framework that links forgetting to activation support overlap and intrinsic data dimension . We validate these analyses through systematic experiments on synthetic and vision tasks, measuring forgetting dynamics under varying model configurations and data complexity. Further, we introduce KAN-LoRA, a novel adapter design for parameter-efficient continual fine-tuning of language models, and evaluate its effectiveness in knowledge editing tasks. Our findings reveal that while KANs exhibit promising retention in low-dimensional algorithmic settings, they remain vulnerable to forgetting in high-dimensional domains such as image classification and language modeling. These results advance the understanding of KANs’ strengths and limitations, offering practical insights for continual learning system design. Code — https://github.com/marufur-cs/AAAI26 Introduction Catastrophic forgetting, also known as catastrophic interference ( McCloskey and Cohen 1989 ) , a fundamental challenge in machine learning, occurs when a neural network loses previously acquired information while learning from new data. This phenomenon is central to the field of continual learning, where models are trained incrementally on non-stationary data distributions ( Ven, Soures, and Kudithipudi 2024 ; Kemker et al. 2017 ) . Moreover, it is prevalent in a wide range of research fields such as meta-learning ( Spigler 2020 ) , domain adaptation ( Xu et al. 2020 ) , foundation models ( Luo et al. 2025 ) , and reinforcement learning ( Zhang et al. 2023 ) , where the retention of prior knowledge is critical for generalization and stability. Multi-Layer Perceptrons (MLPs) are inherently prone to catastrophic forgetting ( Zenke, Poole, and Ganguli 2017 ) . Several techniques have been proposed to overcome catastrophic forgetting in MLPs ( Wang et al. 2025 ; De Lange et al. 2022 ) . Regularization-based techniques ( Kirkpatrick et al. 2017 ; Kong et al. 2024 ) impose restrictions on the network’s weight adjustments, hence reducing the likelihood of interference with previously acquired knowledge. Architecture-based methods ( Yoon et al. 2018 ; Mirzadeh et al. 2022 ) mitigate forgetting by modifying the network’s architecture to accommodate new information. Rehearsal-based methods ( Buzzega et al. 2020 ; Riemer et al. 2019 ) aim to preserve prior information by including data samples from earlier learning sessions during the current session. Although catastrophic forgetting has been extensively studied in MLPs, it remains relatively underexplored in emerging fundamental neural architectures such as Kolmogorov-Arnold Networks (KANs) ( Liu et al. 2025 ) . KANs, inspired by the Kolmogorov-Arnold representation theorem ( Kolmogorov 1961 ) , have emerged as a promising alternative neural network architecture to traditional MLPs. KANs were introduced to address several fundamental limitations of MLPs. Unlike MLPs, which rely on fixed activation functions, KANs utilize learnable one-dimensional activation functions (spline) along the edges of the network. Splines can be easily adjusted locally and are accurate for low-dimensional functions, giving KANs the potential to avoid forgetting. As spline bases are local, a data sample affects only a few related spline coefficients, leaving other coefficients unaltered. This unique architecture enables KANs to learn non-linear relations more effectively and to be more robust against catastrophic forgetting in continual learning scenarios ( Lee et al. 2025 ) . KANs have been successfully applied in various domains ( Yang and Wang 2025 ; Abd Elaziz, Ahmed Fares, and Aseeri 2024 ) , yet studies around their effectiveness in mitigating catastrophic forgetting in continual learning are still quite limited. Only a few pioneer works have studied the catastrophic forgetting phenomenon in KANs under the continual learning settings. Lee et al. 2025 recently proposed a simple and heuristic strategy, WiseKAN, which allocates distinct parameter subspaces to different tasks to mitigate catastrophic forgetting in KANs. Liu et al. 2025 demonstrated robustness of KANs against catastrophic forgetting using synthetic data on regression tasks. Furthermore, some studies proposed modified KANs to achieve robust retention in specific domains, such as classification ( Hu et al. 2025 ) and face forgery detection ( Zhang et al. 2025 ) tasks. Despite these initial efforts, a comprehensive understanding of forgetting in KANs remains elusive, particularly in terms of theoretical characterization and empirical evaluation on practical real-world tasks. To bridge the gap, we first develop a theoretical framework for understanding catastrophic forgetting in KANs by formulating several key factors such as activation support overlap and intrinsic data dimension . Our analysis reveals that forgetting in KANs scales linearly with activation support overlap and grows exponentially with the intrinsic dimensionality of the task manifold, offering a principled explanation for KANs’ robustness in simple tasks and vulnerability in complex domains. Building on these insights, we then conduct extensive empirical experiments comparing KANs with MLPs across a spectrum of tasks, including the low-dimensional synthetic addition and the high-dimensional image classification. Furthermore, we design a novel LoRA ( Hu et al. 2022 ) adapter based on KAN, termed KAN-LoRA, to enable continual fine-tuning of language models (LMs) for sequential knowledge editing. Across all experimental settings, our results consistently corroborate the theoretical analysis, illustrating that while KANs achieve strong retention in structured and low-dimensional tasks, they remain susceptible to forgetting in high-dimensional domains, thereby highlighting both the strengths and limitations of KANs in practical continual learning scenarios. Our main contributions are summarized as below: • We develop a theoretical framework for catastrophic forgetting in KANs, deriving formal retention bounds based on activation support overlap and intrinsic data dimension, and characterizing how forgetting evolves; • We validate the theoretical analysis through empirical experiments on synthetic and image data, demonstrating strong alignment between the support overlap, task complexity, and the observed forgetting behavior; • We introduce KAN-LoRA, a novel KAN-based adapter for continual fine-tuning of LMs, and evaluate its performance in sequential knowledge editing, highlighting both the strength and limitations of KANs in practice. Preliminary Catastrophic Interference Neural networks learn the non-linear mapping between input and output spaces by finding a region in the parameter space where the network achieves expected behavior ( Bishop 1994 ) . When the neural network is trained on new data, the network’s parameter space shifts accordingly to capture the mapping between new input and output space. As a result, performance degrades on prior data. This phenomenon was termed as catastrophic interference by McCloskey and Cohen 1989 . It was observed in many machine learning models such as support vector machine ( Ayad 2014 ) , but is particularly pronounced in connectionist models (e.g., MLPs) due to their dense and globally updated parameterizations ( French 1999 ) . Standard neural training algorithms typically lack the capacity to progressively learn new tasks without overwriting previous knowledge ( Aleixo et al. 2023 ) , making them especially vulnerable to catastrophic interference. Such limitation has motivated continual learning studies

Read the complete text