LODAP: On-Device Incremental Learning Via Lightweight Operations and Data Pruning

Biqing Duan, Qing Wang, Di Liu, Wei Zhou, Zhenli He, Shengfa Miao · 2025

Read LODAP: On-Device Incremental Learning Via Lightweight Operations and Data Pruning by Biqing Duan, Qing Wang, Di Liu, Wei Zhou, Zhenli He, Shengfa Miao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19638.txt.

Introduction

Duan and co-authors present a 2025 paper on LODAP, a framework that lets edge devices learn new classes after deployment without phoning a server. Factories and field gadgets cannot always ship data away for retraining. The core is a slim module mixing plain convolutions with tiny adapter operations. Adapters pick up new-class features cheaply, holding accuracy while model size shrinks.

A pruning step then throws away much of the training data, cutting the learning bill again. On two image benchmarks, accuracy rose up to about four points over rivals, with half the model complexity. Real edge hardware trials close the paper. The authors pitch LODAP for devices that must keep learning in place.

Catalog record

CREATORBiqing Duan, Qing Wang, Di Liu, Wei Zhou, Zhenli He, Shengfa Miao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LODAP: On-Device Incremental Learning Via Lightweight Operations and Data Pruning Biqing Duan, Qing Wang, Di Liu, Wei Zhou, Zhenli He, Shengfa Miao Abstract Incremental learning that learns new classes over time after the model’s deployment is becoming increasingly crucial, particularly for industrial edge systems, where it is difficult to communicate with a remote server to conduct computation-intensive learning. As more classes are expected to learn after their execution for edge devices. In this paper, we propose LODAP , a new on-device incremental learning framework for edge systems. The key part of LODAP is a new module, namely Efficient Incremental Module (EIM). EIM is composed of normal convolutions and lightweight operations. During incremental learning, EIM exploits some lightweight operations, called adapters , to effectively and efficiently learn features for new classes so that it can improve the accuracy of incremental learning while reducing model complexity as well as training overhead. The efficiency of LODAP is further enhanced by a data pruning strategy that significantly reduces the training data, thereby lowering the training overhead. We conducted extensive experiments on the CIFAR-100 and Tiny-ImageNet datasets. Experimental results show that LODAP improves the accuracy by up to 4.32% over existing methods while reducing around 50% of model complexity. In addition, evaluations on real edge systems demonstrate its applicability for on-device machine learning. The code is available at https://github.com/duanbiqing/LODAP. [orcid=0000-0001-0000-0000] [orcid=0000-0001-0000-0000] [orcid=0000-0002-4365-2768] [orcid=0000-0002-5881-9436] [orcid=0000-0002-7986-2222] [orcid=0000-0003-1210-1135] † † corresponding: Equal contributors † † note: This work is supported in part by NSFC under Grant 62362068, the National Key Research Development Program of China under Grant No.2024YFC3014300, and the Yunnan Province Major Science and Technology Project under Grant No.202302AD080006. 1 Introduction Deep neural networks (DNNs), particularly convolution neural networks (CNNs), have demonstrated remarkable performance across various fields, e . g ., image classification and object detection [ 1 ] . Since the training of CNNs is computation-intensive and CNNs themselves have been becoming increasingly complex to pursue higher accuracy, CNN models are typically trained on high-performance servers. Then, the well-trained model is deployed on a target platform, another server or an edge platform. Such conventional training-deployment paradigm may suffer from the inferior prediction accuracy when applying the well-trained model to a new context, in which some new data or categories are introduced after deployment. Thus, methods are proposed to address this issue, such as transfer learning or model adaption. In some cases, new training data becomes available or is introduced to the model incrementally , necessitating incremental learning (IL) capability [ 2 ] . An increasing number of CNNs have been gradually implemented on edge systems to enable local processing of sensitive and private data and ensure on-time inference, even with unreliable networks [ 3 ] . The intersection of edge computing and artificial intelligence is commonly referred to as EdgeAI or Edge Intelligence . EdgeAI is deemed as the key element of ubiquitous AI, efficiently and effectively processing local data without expensive data communication. Some EdgeAI systems are required to learn new classes or tasks after their model deployment. However, data collected from edge systems are confidential or sensitive, and users are reluctant or forbidden to share these data with the model provider. As a result, on-device machine learning is emerging as a promising solution [ 4 ] and has been deployed on mobile devices 1 1 1 https://machinelearning.apple.com/research/recognizing-people-photos . On-device incremental learning is one of on-device machine learning methods which learns new classes on top of the deployed model [ 5 ] [ 6 ] . On-device incremental learning (ODIL) poses challenges from both the perspectives of IL and edge systems. IL typically suffers from catastrophic forgetting [ 6 ] – as the model learns new tasks, its performance on previous tasks rapidly deteriorates. The catastrophic forgetting is due to that the features learned for new classes overwrite the features of old classes [ 6 ] . Hence, increasing the model capacity to learn new classes becomes a natural solution, but results in a more complex model [ 7 ] , which impedes its applicability on resource-constrained edge systems. Knowledge distillation is a widely used technique to mitigate catastrophic forgetting in IL [ 8 , 9 ] . However, their performance is inferior in terms of achievable accuracy. Parameter freezing is proposed in [ 5 ] to conduct ODIL. Parameter freezing utilizes redundant features/weights in the old model to learn new features. It can reduce the training overhead and mitigate catastrophic forgetting. However, as the number of incremental classes increases, its performance degrades significantly. From the perspective of on-device ML, two factors contribute to the challenges of implementing ODIL on resource-constrained edge platforms: the high model complexity and the high training overhead . Model complexity : Although new and effective model architectures have been proposed, CNN model design still adheres to the scaling law, aiming to increase parameters and capacities for higher accuracy. Model training : Model training is computationally intensive and memory-demanding, thus requiring substantial resources. Although IL only learns a few classes once, it still requires a large amount of diverse and high-quality data to train the model, which leads to high training overhead. However, many edge platforms, such as those used in mobile edge computing, are subject to limited resources, memory, and battery power. The substantial training overhead can rapidly deplete an edge system’s battery. In this paper, we propose a new ODIL framework for edge systems, dubbed LODAP , which deploys a lightweight module to address model complexity and data pruning to reduce training overhead. In LODAP , we propose a new lightweight operation to increase the capacity of the model which is used to learn features of the new classes without affecting the old classes. By doing so, we preserve the features of old classes while reducing both model size and training overhead. Additionally, we employ a fusion method after IL to integrate the newly learned features with the old ones, ensuring that the model size remains constant throughout the IL process. Then, inspired by the data pruning method [ 10 ] , we propose a simple yet effective data pruning method to prune the data during the IL phases. In particular, LODAP makes the following novel contributions: 1. We propose a new module for efficient IL, namely Efficient Incremental Module ( EIM ), which consists of normal convolutional kernels and lightweight operations. EIM only uses some lightweight operations, called adapters , to learn new features for new tasks. By means of EIM , we can reduce the model complexity and improve the effectiveness of IL. To further reduce the increased complexity over the course of IL, we exploit structural reparameterization to fuse the newly learned adapters into the main structure. 2. To further reduce the training cost, we propose a progressive data pruning method that is based on calculating the Error L2-Norm(EL2N) scores. Our data pruning method prunes data with low EL2N scores. This method not only significantly reduces the training overhead, but also slightly improves the overall accuracy. This may provide a new lens for IL. 3. Experimental results on CIFAR-100 and Tiny-ImageNet datasets show that LODAP can improve the accuracy by up to 4.32% over the state-of-the-art methods while requiring only 50% of model parameters. In addition,

Read the complete text