Data-Efficient Stream-Based Active Distillation For Scalable Edge Model Deployment

Dani Manjah*, Tim Bary*, Benoît Gérin, Benoît Macq and Christophe de Vleeschouwer · 2025

Read Data-Efficient Stream-Based Active Distillation For Scalable Edge Model Deployment by Dani Manjah*, Tim Bary*, Benoît Gérin, Benoît Macq and Christophe de Vleeschouwer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20484.txt.

Introduction

Cameras at the edge of a network cannot stream everything to the cloud for labeling. Dani Manjah and four co-authors study how to pick the few images worth sending in a 2025 arXiv paper. In their setup, a large teacher model on a server labels selected images. Small models on the cameras then train on those labels and must be refreshed as scenes change.

The selection strategy mixes two instincts. Confident stream items are favored as they arrive, one by one. A diversity rule stops the chosen set from repeating the same scene over and over. For the same training effort, the paired strategy produced a high-quality edge model. It did so while asking the server for very few labels. The result matters for large camera fleets where bandwidth and labeling budgets are tight.

Catalog record

CREATORDani Manjah*, Tim Bary*, Benoît Gérin, Benoît Macq and Christophe de Vleeschouwer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20484

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data-Efficient Stream-Based Active Distillation For Scalable Edge Model Deployment Dani Manjah*, Tim Bary*, Benoît Gérin, Benoît Macq and Christophe de Vleeschouwer Abstract Edge camera-based systems are continuously expanding, facing ever-evolving environments that require regular model updates. In practice, complex teacher models are run on a central server to annotate data, which is then used to train smaller models tailored to the edge devices with limited computational power. This work explores how to select the most useful images for training to maximize model quality while keeping transmission costs low. Our work shows that, for a similar training load ( i.e. , iterations), a high-confidence stream-based strategy coupled with a diversity-based approach produces a high-quality model with minimal dataset queries. † † address: ICTEAM, UCLouvain, 1348 Louvain-la-Neuve, Belgium 1 Introduction Edge camera systems are a cost-effective and privacy-preserv-ing solution for visual analytics tasks such as object detection and classification. As these networks scale to improve coverage and resilience [ 1 ] , each added sensor introduces a new visual domain, making it infeasible to rely on a single, universal Deep Neural Network. Additionally, existing sensors are subject to distribution shifts over time [ 2 ] , requiring frequent model updates. However, the limited compute and memory of edge devices constrain Deep Neural Network size and performance [ 3 , 4 ] . Scalable deployment pipelines are essential for reducing retraining costs and maintaining system reliability. Stream-Based Active Distillation addresses these challenges by training lightweight Student models locally, using data pseudo-labeled by a powerful Teacher model [ 5 ] . Its on-the-fly confidence-based sampling selects only images the Student is most confident about, reducing noisy labels and enabling continuous adaptation. While effective, confidence alone can result in redundant datasets and unnecessary data transfer. To improve efficiency, we introduce D-Stream-Based Active Distillation, which adds an on-device filtering stage to Stream-Based Active Distillation. As shown in Fig. 1 , candidate frames are buffered and filtered using a lightweight embedding model that selects a diverse subset of samples for training. Our experiments on 15 WALT cameras [ 6 ] show that this two-stage selection significantly reduces bandwidth and training costs while preserving model performance. By maximizing embedding diversity with minimal overhead, D-Stream-Based Active Distillation enhances the scalability of edge learning systems. Figure 1: Candidate frames extracted from the device stream are stored in a buffer, where a pool-based filtering module selects high-quality samples to reduce server-bound training data while maintaining dataset quality. The selected frames are then pseudo-annotated by a teacher model to train the next iteration of the edge model. 2 Related Work 2.1 Online Knowledge Distillation Knowledge Distillation encompasses techniques where a Student model learns from another Teacher model [ 7 ] . This approach can be used to create compact models that are based on the capabilities of a more complex model, and is especially useful when deploying on edge devices, where storage and computational capacity become critical factors [ 8 ] . Through regular updates, the Teacher ’s knowledge ensures that the Student model continues to perform effectively even when operating on changing data distributions thanks to its greater robustness [ 9 ] . Recently, KD has supported innovative applications in video analytics, particularly through Online Distillation [ 2 , 10 ] , where a compact model’s weights are updated in real-time to mimic the output of a larger, pre-existing model. 2.2 Active Distillation Typical Active Learning assumes the presence of a perfect annotator, an assumption that does not hold in a distillation framework where pseudo-labeling can lead to the accumulation of errors due to incorrect predictions ( i.e. , confirmation bias ) in semi-supervised or unsupervised learning [ 11 ] . To mitigate noisy labels in KD, the Active Distillation framework has been proposed by [ 12 , 5 , 13 ] as a query-efficient Active Learning method that is robust to imperfections in the Teacher model. Specifically, in Stream-Based Active Distillation [ 5 ] , the authors demonstrated that high-confidence automatic labeling of training samples plays a crucial role in the Active Distillation selection process. By selecting samples where the Student model exhibits high confidence, the framework reduces labeling errors and enhances the reliability of the distillation process. In this work, we retain the Stream-Based Active Distillation and its top-confidence selection rule due to its effectiveness in handling large volumes of unlabeled images and its ability to continuously adapt to incoming cameras without requiring labeled data at test time [ 14 ] . However, we propose a two-stage approach that balances exploration and exploitation, similar to PPAL [ 15 ] . In that sense, we extend Stream-Based Active Distillation with a filtering module designed to identify a subset that is representative of the camera’s domain after exploration. 2.3 Pool-Based Active Learning Pool-based methods aim to select an informative subset from the pool of data available in the buffer. Uncertainty-based methods select samples based on predictive confidence scores or error rates. The Minmax strategy [ 16 ] selects the least confident samples from the unlabeled pool, assuming that low-confidence predictions are more informative for training. Error-based strategies refine this process by selecting samples where predicted and actual errors diverge. Wu et al. [ 17 ] proposed Entropy-Diversity Sampling , combining entropy-based NMS (Non-Maximum Suppression) with a diversity-based prototype selection phase to maximize intra- and inter-class variation. Density-based methods select samples that best represent the underlying data distribution. Yang et al. [ 15 ] proposed PPAL , a two-stage strategy using uncertainty sampling followed by category-conditioned similarity matching to ensure diversity in the selected set. 2.4 Dataset Pruning Dataset pruning belongs to coreset techniques with the focus of selecting a representative subset from a larger dataset to maintain or enhance model performance while reducing training data size [ 18 ] . Entropy [ 19 ] is a classic approach but primarily limited to classification tasks. Many state-of-the-art methods, such as CSOD [ 20 ] or AUM [ 21 ] , rely on ground truth labels. CSOD trains a model to identify informative samples, while AUM removes mislabeled or ambiguous data using ground truth information. In edge computing environments, utilizing the edge model in a self-supervised manner can introduce confirmation bias, particularly given the model’s limited capacity. Alternatively, querying a larger, general-purpose Teacher model, or an Oracle , conflicts with scalability objectives due to increased bandwidth usage and annotation costs associated with transmitting images. Self-supervised methods that compute per-sample importance scores, such as distance to other samples [ 22 ] , or that utilize clustering heuristics to identify representative samples [ 15 ] , offer alternatives. Finally, MODERATE [ 23 ] employs redundancy and representativeness strategies for dynamic and moderate pruning, while TFDP introduces a training-free dataset pruning approach for instance segmentation tasks [ 24 ] . In this work, we benchmark MODERATE and TFDP as they focus on reducing redundancy and selecting representative samples without any labels. 3 D-SBAD Our framework follows a client-server dynamic, where a lightweight, pre-trained Student model \theta , deployed on an edge device ( e.g. , a camera), processes a video stream \mathbf{X} . To ensure the Student remains up-to-date, it is periodically fine-tuned on a server using a curated set \mathcal{T} of training samples selected from

Read the complete text