CLANE: Continual Learning of Actions on Neuromorphic Hardware from Event Cameras
Elvin Hajizada, Michael Neumeier, Edward Paxon Frady, Yulia Sandamirskaya, Axel von Arnim, Bing Li, Eyke Hüllermeier · 2026
Read CLANE: Continual Learning of Actions on Neuromorphic Hardware from Event Cameras by Elvin Hajizada, Michael Neumeier, Edward Paxon Frady, Yulia Sandamirskaya, Axel von Arnim, Bing Li, Eyke Hüllermeier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28387.txt.
Introduction
Elvin Hajizada and six colleagues present CLANE, a system that learns new human actions on the device itself. It watches through event cameras and keeps earlier actions in memory as it adds new ones. The system runs on Intel Loihi 2 hardware, pairing a spiking network with an on-chip learning head. On a 50-class real-world set it scored about 70 percent, using over a hundred times less energy and running much faster than a GPU baseline.
Catalog record
| CREATOR | Elvin Hajizada, Michael Neumeier, Edward Paxon Frady, Yulia Sandamirskaya, Axel von Arnim, Bing Li, Eyke Hüllermeier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29615 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28387
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLANE: Continual Learning of Actions on Neuromorphic Hardware from Event Cameras Elvin Hajizada, Michael Neumeier, Edward Paxon Frady, Yulia Sandamirskaya, Axel von Arnim, Bing Li, Eyke Hüllermeier Abstract Recognizing and continuously learning novel human actions without forgetting prior classes is a requirement for emerging AR/VR and robotics applications. For these applications, both on-device processing and learning are essential for privacy and low-latency adaptation. Event cameras address the efficiency of visual sensing with sparse, asynchronous output that is naturally compatible with neuromorphic processing. Yet no prior system has deployed a continual on-device learning pipeline for event-based action recognition using neuromorphic hardware. We present CLANE – C ontinual L earning of A ctions on N euromorphic hardware from E vent cameras – deployed end-to-end on Intel Loihi 2. CLANE combines a spiking 2D CNN for spatiotemporal feature extraction with CLP-SNN as its on-chip learning head, extended to action clips via a Temporal Aggregation Layer and a fixed-point Normalization Layer – both novel Loihi 2 modules. On THU E-ACT -50 dataset (50 classes, real-world conditions), CLANE achieves 70.4% accuracy at a continual learning task while delivering >100× energy reduction and 16× lower latency over a sequential CNN+GRU+CLP edge GPU baseline, validated through iso-algorithm cross-platform benchmarking across three evaluation levels. 1 Introduction Continual learning from experience – acquiring new knowledge sequentially without forgetting what came before – is a routine capability of the brain, yet engineering it into always-on, power-constrained edge devices remains a significant open challenge [ 4 ] . A key reason is architectural: standard deep learning requires batch retraining via global error propagation, large activation buffers, and access to past data or large replay buffers, all incompatible with the power and memory budgets of edge devices [ 7 ] . Biological nervous systems, on the other hand, update synaptic weights through local three-factor plasticity rules, depending only on pre- and post-synaptic activity and diffuse modulatory signals available at the synapse [ 6 ] . Global error propagation, by contrast, requires transmitting loss gradients across all layers and storing intermediate activations – operations with no plausible biological substrate [ 17 ] that stretch both compute and memory of on-device computers. Neuromorphic processors draw inspiration from the brain and implement event-driven spike routing, near-memory computing, and sparse parallel computation [ 28 ] . Intel Loihi 2 neuromorphic chip also provides on-chip learning acceleration, computing weight updates close to where the weights reside, avoiding DRAM transfers that dominate GPU energy budgets [ 2 , 24 ] . This makes neuromorphic hardware a natural candidate for energy-efficient continual learning – provided the learning algorithm itself respects locality constraints. Real-world deployment scenarios for such a system include AR/VR and robotics, where action or gesture recognition must operate in real time at the edge under strict power budgets. The challenges of visual action perception, such as variable lighting and temporal granularity, leading to redundant data in static scenes and motion blur during fast movement, can be addressed by event-based cameras – retina-inspired sensors that asynchronously report per-pixel brightness changes as sparse spike trains with microsecond precision and high dynamic range [ 8 ] . These sensors communicate only pixels with intensity changes, producing representations that are inherently sparse and architecturally compatible with the event-driven computation of neuromorphic processors, enabling low-latency, low-energy action recognition systems. A further challenge of on-device action recognition is that the space of possible human actions is large, unstructured, and evolving [ 29 ] . Users introduce novel gestures, environmental conditions shift, and recognition systems must adapt continuously without forgetting previously learned actions. This demands online, class-incremental CL directly on the deployed device, rather than periodic retraining from shuffled batches [ 13 ] . The on-chip local learning acceleration properties of Loihi 2 make it a natural execution substrate for exactly this regime. Figure 1: Our CLANE pipeline with Spiking CNN, Temporal Aggregation, Normalization, and CLP-SNN including sparse binary or graded spike activity maps and vectors. Schematic visualization of pretraining in A and continual learning in B. Spiking neural networks (SNNs) complete the brain-inspired stack outlined above. Their stateful neurons temporally integrate incoming spike streams and communicate via asynchronous spikes [ 32 ] , making them a natural computational fit for the sparse, asynchronous data produced by event cameras [ 23 , 31 ] . Together, event cameras, SNNs, and neuromorphic hardware form a stack where sparsity generated at the sensor propagates through computation—a three-way architectural alignment that serves as the co-design principle [ 28 ] , central to our work. This alignment extends naturally to learning that is event-driven, sparse, and local. Yet despite on-chip learning mechanisms being available on chips like Loihi 2, deploying this full stack with a principled continual learning (CL) algorithm for large-scale spatiotemporal tasks remains unrealized. Previous work has taken steps toward this goal by demonstrating few-shot gesture learning on Loihi 1, though limited to a small benchmark and without the CL and forgetting analysis [ 30 ] . More recently, Hajizada et al. proposed CLP-SNN, a neuromorphic CL method implemented on Loihi 2 for object recognition [ 10 ] . This work was limited to static spatial inputs. More broadly, no prior neuromorphic CL work has been evaluated on a large-scale, real-world dataset approaching the complexity of full-body action recognition. In this regard, extending CLP-SNN to large-scale spatiotemporal recognition tasks poses three unresolved challenges: (1) CLP-SNN processes fixed-size static feature vectors and cannot directly handle temporal spike sequences produced by a deep SNN over an action clip duration; (2) CLP-SNN requires L2-normalized input vectors, yet Loihi 2 provides no native division instruction, making on-chip normalization a non-trivial hardware design problem; (3) the neuromorphic CL literature lacks a benchmarking methodology that simultaneously controls for model capacity, learning algorithm, and hardware – without which efficiency claims cannot be cleanly attributed to hardware and algorithmic differences. We present CLANE (Continual Learning of Actions on Neuromorphic Hardware from Event Cameras), an end-to-end SNN with CL capability deployed on Intel Loihi 2 addressing all the above ( fig. 1 ). CLANE integrates a 2D spiking CNN for spatiotemporal feature extraction with CLP-SNN [ 10 ] as its on-chip continual learning head, learning novel action classes online and class-incrementally from THU E-ACT -50 dataset [ 9 ] in real time. Our contributions are: • CLP-Loihi : Extension of CLP-SNN to spatiotemporal action recognition with Temporal Aggregation and Normalization Layers, implemented on Loihi 2 via custom neuron models. • Full neuromorphic deployment : Co-deployment of a deep spiking CNN feature extractor and the CLP-Loihi on a single Loihi 2 chip, directly processing event-camera-derived spike inputs without a GPU in either the inference or learning loop, evaluated on THU E-ACT-50 under real-world conditions including lighting variation, viewpoint changes, and occlusions. • Rigorous iso-algorithm cross-platform benchmarking against Nvidia Jetson Orin Nano across three levels: full pipeline, learning algorithm, and feature extractor. This addresses challenge (3) by isolating the hardware contribution from algorithmic and model-capacity confounds. CLANE achieves more than 100\times energy reduction and 16\times latency improvement over a 2D CNN+GRU baseline with the same CLP algorithm on the GPU, at an accuracy cost of only 2.6%. 2 Related Work 2.1 Event-based Action Recognition Event-based cameras have been widely applied to vision tasks such as optical flow