Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data

Douwe den Blanken, Charlotte Frenkel · 2025

Read Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data by Douwe den Blanken, Charlotte Frenkel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24852.txt.

Introduction

Douwe den Blanken and Charlotte Frenkel present Chameleon, a 2025 chip that learns on the device from streams of raw data, without multipliers in its math. Edge devices want local learning for privacy and low delay, yet backpropagation hardware wastes power and simple learners miss accuracy targets. Their silicon, built in 40-nanometer technology, unifies training and inference in one design.

Temporal convolutional networks give the chip its grip on long sequences, and a dual-mode array either sips power like top keyword-spotting chips or runs 4.3 times faster at peak. It set records for on-chip few-shot learning on Omniglot and kept learning across 250 classes continually. All this ran on live 16-kilohertz audio. The authors frame it as proof that end-to-end learning belongs at the edge.

Catalog record

CREATORDouwe den Blanken, Charlotte Frenkel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data Douwe den Blanken, Charlotte Frenkel Abstract On-device learning at the edge enables low-latency, private personalization with improved long-term robustness and reduced maintenance costs. Yet, achieving scalable, low-power end-to-end on-chip learning, especially from real-world sequential data with a limited number of examples, is an open challenge. Indeed, accelerators supporting error backpropagation optimize for learning performance at the expense of inference efficiency, while simplified learning algorithms often fail to reach acceptable accuracy targets. In this work, we present Chameleon, leveraging three key contributions to solve these challenges. (i) A unified learning and inference architecture supports few-shot learning (FSL), continual learning (CL) and inference at only 0.5% area overhead to the inference logic. (ii) Long temporal dependencies are efficiently captured with temporal convolutional networks (TCNs), enabling the first demonstration of end-to-end on-chip FSL and CL on sequential data and inference on 16-kHz raw audio. (iii) A dual-mode, multiplier-free compute array allows either matching the power consumption of state-of-the-art inference-only keyword spotting (KWS) accelerators or enabling 4.3 × 4.3\times higher peak GOPS. Fabricated in 40-nm CMOS, Chameleon sets new accuracy records on Omniglot for end-to-end on-chip FSL (96.8%, 5-way 1-shot, 98.8%, 5-way 5-shot) and CL (82.2% final accuracy for learning 250 classes with 10 shots), while maintaining an inference accuracy of 93.3% on the 12-class Google Speech Commands dataset at an extreme-edge power budget of 3.1 µ ​ W 3.1\text{\,}\mathrm{\SIUnitSymbolMicro W} . \AtNextBibliography I Introduction The rise of edge computing has driven demand for deploying deep learning models on resource-constrained devices [ 1 ] , forming the backbone of the Internet-of-Things (IoT) ecosystem [ 2 ] . However, most edge artificial intelligence (AI) devices focus on inference [ 2 ] , relying on pre-trained models that cannot be adapted post-deployment. This lack of adaptability limits robustness to emerging features and data distribution shifts [ 2 ] , making smart sensors unreliable over time [ 3 ] . Fig. 1 (a) shows that while cloud-hosted retraining offers a workaround where new labeled data is uploaded from an edge device to a central server for model updates, this approach is cloud-connectivity-dependent and does not fit latency-constrained scenarios, while leading to privacy concerns [ 4 ] . On the other hand, training a model from scratch on-device incurs high energy penalties and still requires the availability of a large labeled dataset, which is not realistic at the edge. Therefore, enabling efficient on-device learning at the edge requires the ability to continually learn from limited data. Fig. 1 : (a) Overview of three approaches to endow edge AI devices with the ability to adapt, where on-device few-shot learning (FSL) and continual learning (CL) enable offline, low-latency, and sample-efficient learning. (b) Comparison of an inference and three learning architectures, including ours; we minimize the footprint of learning by integrating it into the inference pipeline rather than treating it separately. (c) Our architecture supports end-to-end on-chip inference, FSL, and CL at a low overhead. However, over a device’s lifetime, far fewer tasks are learned than inferences performed. Hence, it is key for an edge AI device to support learning without degrading inference performance and efficiency ( Fig. 1 (b)). Yet, current efforts to extend inference accelerators with few-shot learning (FSL) capabilities incur significant area, energy and accuracy penalties. For example, using quantized gradient descent (GD) to implement on-chip FSL [ 5 , 1 , 6 ] , requires maintaining all intermediate activations in memory, transposing weight matrices, and using costly data types such as floating-point (FP) representations [ 1 ] . Alternatively, gradient-free FSL approaches using compute-in-memory (CIM) macros [ 7 , 8 ] have been proposed to accelerate distance computation between input and stored embedding vectors for FSL. However, they fail to provide end-to-end learning support as they rely on off-chip FP32 deep neural networks (DNNs) to compute high-quality embeddings. To the best of our knowledge, only one prior work computes the DNN embeddings on chip for gradient-free FSL [ 9 ] . This design adds dedicated learning hardware alongside inference modules, yielding a 2\times increase in peak power, a 25-% area overhead, and requires external memory to store all model parameters for learning, undermining its suitability for extreme edge scenarios. Another challenge is that state-of-the-art (SotA) on-chip FSL architectures lack the ability to handle temporal information in two key ways. First, they cannot process sequential data . All existing FSL accelerators target image classification tasks [ 5 , 1 , 6 , 1 , 7 , 8 , 9 ] , whereas real-world applications often involve sequential data, such as keyword spotting (KWS) where audio is typically sampled at frequencies ranging from 16\text{\,}\mathrm{kHz}48\text{\,}\mathrm{kHz} . However, due to the limited input context length of current end-to-end KWS accelerators (60 [ 10 , 11 , 12 ] to 101 [ 13 ] ), feature extraction (FE) such as mel-frequency cepstral coefficients (MFCC) is typically performed for two-order-of-magnitude reductions in input sequence length, at the expense of a 50–75% area overhead [ 10 , 14 ] . Hence, capturing temporal dependencies that span multiple orders of magnitude is not only key for complex temporal tasks, it also allows streamlining system design. Second, current on-chip FSL designs cannot perform sequential learning of tasks with data distributions that shift over time . This continual learning (CL) scenario is typical of edge applications, where users add features, such as new keywords, over time. While high-quality embeddings produced by off-chip embedders have recently been shown to also support CL over up to 100 classes [ 15 ] , end-to-end on-chip CL is still an open challenge. Overall, due to limited on-chip memory and strict power, area, and energy budgets, no extreme-edge solution has been proposed that effectively balances efficiency and versatility for end-to-end learning on sequential data. In this work, we present Chameleon, the first work to demonstrate scalable on-chip FSL and CL on sequential data, in a fully end-to-end fashion, and without degrading the inference performance or efficiency ( Fig. 1 (c)). We achieve this through three key contributions: 1. To minimize the footprint of on-device learning, we build on prototypical networks (PNs) [ 16 ] to reframe FSL using distance computation as a forward pass through an equivalent fully connected (FC) layer, thereby leading to a unified learning and inference architecture . By integrating learning within the inference process, we enable gradient-free, end-to-end FSL and CL at only 0.5% area overhead and 0.02% added latency while setting new FSL accuracy records (96.8%, 5-way 1-shot) and CL accuracy (82.2% final, 89.0% average for learning 250 classes with 10 shots) on the Omniglot [ 17 ] dataset. 2. To enable accurate FSL and CL with temporal information, a DNN that captures long-range relationships to produce high-quality embeddings is crucial. Hence, we propose to use temporal convolutional networks (TCNs), whose parameter efficiency scales better than recurrent neural networks [ 18 ] , as efficient on-chip PN embedders for sequential data at the edge. Compared to SotA TCN accelerators [ 19 , 11 , 13 ] , we expand the receptive field by 160\times while cutting activation memory by 4\times . 3. To enable inference at an extreme-edge power budget while

Read the complete text