Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression
Bibin Wilson · 2026
Read Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression by Bibin Wilson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00138.txt.
Introduction
Wilson presents Latent Replay Detection, the first system for continual object detection on microcontrollers under tight memory limits. It compresses past examples with task-adaptive FiLM conditioning, so each task's embeddings steer compression toward its own discriminative features. It also picks replay examples to vary bounding boxes, using farthest-point sampling in IoU space, curbing localisation bias. Storing 150 bytes per sample against over 10KB for images, a 64KB buffer holds over 400 examples; tests on CORe50 show strong results across tasks on three microcontroller chips.
Catalog record
| CREATOR | Bibin Wilson |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35099 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00138
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Replay Detection: Memory-Efficient Continual Object Detection on Microcontrollers via Task-Adaptive Compression Bibin Wilson Abstract Deploying object detection on microcontrollers (MCUs) enables intelligent edge devices but current models cannot learn new object categories after deployment. Existing continual learning methods require storing raw images—far exceeding MCU memory budgets of tens of kilobytes. We present Latent Replay Detection (LRD) , the first framework for continual object detection under MCU memory constraints. Our key contributions are: (1) Task-Adaptive Compression : Unlike fixed PCA, we propose learnable compression with FiLM (Feature-wise Linear Modulation) conditioning, where task-specific embeddings modulate the compression to preserve discriminative features for each task’s distribution; (2) Spatial-Diverse Exemplar Selection : Traditional sampling ignores spatial information critical for detection—we select exemplars maximizing bounding box diversity via farthest-point sampling in IoU space, preventing localization bias in replay; (3) MCU-Deployable System : Our latent replay stores ∼ \sim 150 bytes per sample versus > > 10KB for images, enabling a 64KB buffer to hold 400+ exemplars. Experiments on CORe50 (50 classes, 5 tasks) demonstrate that LRD achieves XX.X% mAP@50 on the initial task and maintains strong performance across subsequent tasks—a significant improvement over naive fine-tuning while operating within strict MCU constraints. Our task-adaptive FiLM compression and spatial-diverse exemplar selection work synergistically to preserve detection capabilities. Deployed on STM32H753ZI, ESP32-S3, and MAX78000 MCUs, LRD achieves 4.9–97.5ms latency and 49–2930 μ \mu J energy per inference within a 64KB memory budget—enabling practical continual detection on edge devices for the first time. 1 Introduction The deployment of object detection on microcontrollers (MCUs) has enabled a new generation of intelligent edge devices—from smart home sensors to industrial robots and wearable cameras. These devices must operate under strict constraints: power budgets measured in milliwatts, memory limited to hundreds of kilobytes, and latency requirements of tens of milliseconds. Recent advances in TinyML have demonstrated impressive detection capabilities within these constraints [ 37 ; 31 ; 3 ] , achieving real-time inference with models under 500KB. However, a critical limitation remains: these models cannot learn new object categories after deployment . Consider a warehouse robot equipped with an MCU-based detector trained to recognize packages and pallets. When the warehouse begins handling new product types, the only options are (1) retraining from scratch on a central server and re-deploying, which is expensive and requires collecting new data, or (2) fine-tuning on the device, which leads to catastrophic forgetting of previously learned categories [ 36 ] . Neither option is practical for real-world edge deployments. Continual learning (CL) offers a solution by enabling models to learn incrementally without forgetting [ 30 ; 28 ; 41 ] . Yet, existing CL methods are designed for cloud-scale models with abundant memory. The dominant approach—experience replay—stores exemplars from previous tasks to rehearse during training [ 40 ; 9 ] . For image classification, this typically requires storing hundreds to thousands of images (tens to hundreds of megabytes)—far exceeding MCU memory budgets. While continual classification has been studied extensively [ 15 ] , continual object detection remains underexplored [ 32 ; 7 ] , and continual detection on MCUs is entirely unaddressed. MCU Detector Latent Buffer 64KB 450KB Task 1 Task 2 Task 3 Detections Figure 1: LRD enables continual object detection on MCUs. Our method stores compressed latent features instead of raw images, fitting a replay buffer within 64KB while supporting incremental learning of new object categories. In this paper, we bridge this gap with Latent Replay Detection (LRD) , the first framework for continual object detection under MCU memory constraints. Our key insight is that replay does not require storing raw images or even full feature maps. Instead, we can store highly compressed latent representations extracted from intermediate network layers (specifically, FPN outputs) and replay these directly during training. This latent replay is > 60 \times more memory-efficient than image replay while preserving the information necessary for detection supervision. However, simply applying existing compression schemes (e.g., fixed PCA) to detection features is suboptimal. Different tasks exhibit different feature distributions, and a single compression matrix cannot optimally preserve discriminative information across all tasks. Furthermore, standard exemplar selection methods (random, herding) ignore spatial information critical for object detection—leading to localization bias when replayed exemplars cluster in specific image regions. Our framework introduces three novel technical contributions: 1. Task-Adaptive Compression : Unlike fixed PCA which uses a single projection matrix for all tasks, we propose learnable, task-conditioned compression using FiLM (Feature-wise Linear Modulation) [ 39 ] . Task-specific embeddings modulate the compression network, adapting to each task’s feature distribution and better preserving discriminative features for continual learning. 2. Spatial-Diverse Exemplar Selection : We propose spatial-aware sampling that maximizes bounding box diversity in the stored exemplars. Using farthest-point sampling in IoU space, we ensure exemplars cover diverse spatial locations (corners, center, various scales), preventing localization bias in replay—a novel consideration specific to detection. 3. MCU-Deployable System : We design our latent replay to fit within strict MCU memory budgets ( < 64KB), storing \sim 150 bytes per sample (compressed features + bbox + class + task ID) compared to > 10KB for a 128 \times 128 image. This enables > 400 exemplars within the memory budget. We evaluate LRD extensively on the CORe50 [ 33 ] benchmark, demonstrating state-of-the-art continual detection performance under memory constraints. Deployed on STM32H753ZI, ESP32-S3, and MAX78000 MCUs, LRD achieves 4.9–97.5ms latency and 49–2930 \mu J energy per inference while supporting incremental learning of 10 object categories across 5 tasks. Contributions. Our contributions are: • We formalize the problem of continual object detection for MCUs and analyze the memory constraints that make existing methods infeasible. • We propose Task-Adaptive Compression with FiLM conditioning—the first task-conditioned feature compression for continual learning that adapts to each task’s feature distribution. • We propose Spatial-Diverse Sampling —the first exemplar selection method for detection that maximizes bounding box diversity to prevent localization bias. • We demonstrate LRD on CORe50 achieving 76.64% mAP@50 on initial tasks with comprehensive ablations showing both novel components contribute independently, achieving state-of-the-art under memory constraints. • We will release code, trained models, and MCU deployment scripts to facilitate future research. 2 Related Work We review continual learning for object detection, compression techniques, meta-learning approaches, and theoretical foundations. 2.1 Continual Object Detection Regularization-based Methods. Early approaches prevent forgetting through regularization. EWC [ 28 ] penalizes changes to important parameters via Fisher information. LwF [ 30 ] distills knowledge from previous models. PackNet [ 35 ] allocates separate parameters per task. However, these methods struggle with long task sequences and provide no theoretical guarantees. Replay-based Methods. Storing exemplars improves stability. iCaRL [ 41 ] combines exemplar replay with nearest-mean classification. PODNet [ 16 ] distills pooled outputs across spatial dimensions. FOSTER [ 44 ] uses feature boosting with dynamic expansion. REMIND [ 25 ] employs product quantization for feature compression. Recent works like CL-DETR [ 32 ] and RODEO [ 8 ] adapt transformers for continual detection but require GB-scale memory. Gap: Existing replay methods either store raw images (memory-intensive) or use fixed compression (suboptimal). None provide theoretical analysis or edge deployment capabilities. 2.2 Compression for Continual Learning Fixed Compression. Traditional approaches use predetermined compression. PCA [ 4 ] projects features to lower dimensions. VQ [ 43 ]