MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices

Sijia Li*, Young D. Kwon*, Lik-Hang Lee, Pan Hui · 2025

Read MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices by Sijia Li*, Young D. Kwon*, Lik-Hang Lee, Pan Hui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00174.txt.

Introduction

Meta-continual learning lets models learn new classes from few examples, a boon for devices that cannot phone home for labels. Sijia Li and co-authors built in 2025 MetaCLBench, the first benchmark that judges such methods on real edge hardware, not just accuracy.

Six methods were run across three architectures and five datasets on devices with half to four gigabytes of memory. Up to half the methods ran out of memory on the smallest devices. One method, LifeLearner, matched the oracle's accuracy while using several times less energy. Bigger models did not reliably win. The authors provide deployment guidelines for choosing methods that actually fit the device.

Catalog record

CREATORSijia Li*, Young D. Kwon*, Lik-Hang Lee, Pan Hui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devices Sijia Li*, Young D. Kwon*, Lik-Hang Lee, Pan Hui Abstract Meta-Continual Learning (Meta-CL) enables models to learn new classes from limited labelled samples, making it promising for IoT applications where manual labelling is costly. However, existing studies focus on accuracy while ignoring deployment viability on resource-constrained hardware. Thus, we present MetaCLBench , a benchmark framework that evaluates Meta-CL methods for both accuracy and deployment-critical metrics (memory footprint, latency, and energy consumption) on real IoT devices with RAM sizes ranging from 512 MB to 4 GB. We evaluate six Meta-CL methods across three architectures (CNN, YAMNet, ViT) and five datasets spanning image and audio modalities. Our evaluation reveals that, depending on the dataset, up to three of six methods cause out-of-memory failures on sub-1 GB devices, significantly narrowing viable deployment options. LifeLearner achieves near-oracle accuracy while consuming 2.54-7.43 × \times less energy than the Oracle method. Notably, larger or more sophisticated architectures such as ViT and YAMNet do not necessarily yield better Meta-CL performance, with results varying across datasets and modalities, challenging conventional assumptions about model complexity. Finally, we provide practical deployment guidelines and will release our framework upon publication to enable fair evaluation across both accuracy and system-level metrics. I Introduction IoT sensing applications [ 1 , 2 , 3 , 4 ] increasingly require models that adapt post-deployment [ 5 ] . Smart speakers must recognise new voice commands [ 6 ] as users’ vocabularies evolve [ 7 ] ; industrial acoustic monitors must identify novel machinery sounds and/or anomalies [ 8 ] indicating emerging faults [ 9 ] ; object detectors deployed in the wild must accommodate new object categories as environments change [ 10 ] . These scenarios share two constraints: models operate on resource-limited hardware (typically under 1 GB RAM) [ 11 ] , and labelled training data for new classes is scarce [ 12 , 13 ] . A fundamental obstacle arises when models learn new classes: Catastrophic Forgetting (CF) [ 14 ] , where acquiring new knowledge degrades performance on previously learned classes. Also, addressing CF on edge devices is challenging as solutions must fit within tight memory and energy budgets. Continual Learning (CL) [ 15 , 16 ] addresses CF by enabling models to learn sequential tasks while retaining prior knowledge. Existing approaches are grouped into three categories: (1) rehearsal-based methods that store and replay past samples [ 17 , 18 , 19 ] , (2) regularisation-based methods that constrain weight updates to preserve important parameters [ 20 , 21 ] , and (3) parameter isolation methods that allocate separate model capacity for each task [ 22 , 23 , 24 ] . Rehearsal-based approaches achieve the strongest accuracy but require substantial labelled data [ 25 ] and impose significant memory and computational overheads [ 26 ] , often exceeding what typical IoT devices can accommodate. To address these limitations, Meta-Continual Learning (Meta-CL) has been introduced by integrating few-shot learning with CL [ 27 , 28 ] . Through meta-training on source tasks, Meta-CL methods learn representations that enable rapid adaptation to new classes from only a few labelled samples to minimize the burden of manual labeling of conventional CL methods [ 27 , 29 , 30 ] and optimize resource utilization through algorithm-system co-design [ 12 ] . Despite the benefits of Meta-CL, there are still notable limitations that warrant further investigation. First , many existing studies [ 27 , 29 , 30 ] primarily focus on accuracy, neglecting deployment viability on resource-constrained hardware (e.g., A method achieving high accuracy is unusable if it requires 2 GB RAM on a device with only 512 MB available.) Second , the effectiveness of the existing methods, which are typically used for tasks involving images, has not been investigated fully with sequential time series data provided by auditory sensor systems, where the modality of the data is significantly different from images [ 31 ] . Lastly , the datasets tested have been relatively simple and limited in their variety [ 12 ] , consisting of a few image/audio or specially crafted datasets that lack generalizability. In this paper, we address these gaps through a systematic benchmark study of six representative Meta-CL methods across three network architectures and five datasets on real IoT hardware. The employed methods include regularisation-based approaches ((1) OML [ 27 ] , (2) ANML [ 29 ] , (3) OML+AIM, and (4) ANML+AIM [ 30 ] ) and rehearsal-based approaches ((5) Latent OML and (6) LifeLearner [ 12 ] ). Furthermore, prior works of Meta-CL focus on image classification, leaving audio sensing applications unexplored despite their importance for IoT (to name a few applications: voice interfaces, acoustic monitoring). We therefore include audio datasets for keyword spotting (GSCv2 [ 32 ] ) and environmental sound classification (UrbanSound8K [ 33 ] , ESC-50 [ 34 ] ), alongside image datasets (CIFAR-100 [ 35 ] , MiniImageNet [ 36 ] ). In addition, prior studies [ 29 , 30 , 12 ] evaluate only three-layer CNN architectures. We extend the evaluation to YAMNet [ 37 ] , a MobileNet-based architecture for audio, and Vision Transformers (ViT) [ 38 ] for image to conduct a comprehensive benchmark study. Unlike prior benchmarks reporting only accuracy on server-class machines, we evaluate on real IoT devices, including Jetson Nano (4 GB), Raspberry Pi 3B+ (1 GB), and Pi Zero 2 (512 MB), measuring peak memory footprint, end-to-end latency, and energy consumption alongside accuracy. This enables practitioners to assess deployment viability, not just model quality. Overall, our key contributions include: • Edge-Deployable Benchmark Framework (Section III ): We develop MetaCLBench for evaluating on-device Meta-CL across IoT sensing modalities. The framework incorporates audio datasets for smart home and industrial applications (GSCv2, UrbanSound8K, ESC-50) and visual datasets (CIFAR-100, MiniImageNet), measuring deployment-critical metrics (memory footprint, latency, and energy) on real IoT hardware from 512 MB to 4 GB device classes. • Deployment-Oriented Findings (Section IV ): We reveal that, depending on the dataset, up to three of six methods cause out-of-memory (OOM) issues on sub-1 GB devices (see Table I for ESC-50 results), substantially narrowing viable IoT deployment options. Our analysis quantifies the computational and memory demands on edge devices (Section IV-A ) We quantify energy-accuracy trade-offs: LifeLearner achieves near-oracle accuracy at 2.54-7.43 \times lower energy consumption than the Oracle method. Notably, we find that well-tuned basic 3-layer CNN architecture with proper pre-training can outperform more sophisticated models such as ViT and YAMNet (Tables I and II ) under difficult Meta-CL setups, challenging conventional assumptions about model complexity. • Open Tools and Practical Guidelines (Section V ): By implementing MetaCLBench on IoT devices, we provide realistic insights and concrete deployment guidelines specifically tailored for resource-constrained devices. Furthermore, we plan to release our benchmark framework and evaluation tools to enable fair and comprehensive assessment of both accuracy and system-level performance metrics. This will facilitate reproducible research and accelerate the development of practical Meta-CL solutions for real-world IoT applications. II Related Work Continual Learning: CL aims to learn new knowledge from dynamically evolving input data distributions [ 10 , 39 ] . When algorithms are trained sequentially, the new learning will interfere with previous established weights,

Read the complete text