Teach YOLO to Remember: A Self-Distillation Approach for Continual Object Detection

Riccardo De Monte, Davide Dalle Pezze, Gian Antonio Susto · 2025

Read Teach YOLO to Remember: A Self-Distillation Approach for Continual Object Detection by Riccardo De Monte, Davide Dalle Pezze, Gian Antonio Susto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04688.txt.

Introduction

Riccardo De Monte, Davide Dalle Pezze, and Gian Antonio Susto teach a fast object detector to learn new classes without forgetting old ones. Their 2025 paper deals with continual learning, where data arrive in stages. Detectors like YOLO excel when trained once on large data sets. In stages, they suffer catastrophic forgetting of earlier classes. Most past fixes target slower two stage detectors.

A popular method, learning without forgetting, was thought unfit for YOLO. Its noisy outputs risk passing on corrupted knowledge, critics said. The team's variant uses self distillation, tuned to this detector family. Paired with a replay memory of old examples, it cut forgetting sharply. Scores rose by 2.1 points on one standard test set and 2.9 on another. The result set a new best for this detector type, the authors report.

Catalog record

CREATORRiccardo De Monte, Davide Dalle Pezze, Gian Antonio Susto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Teach YOLO to Remember: A Self-Distillation Approach for Continual Object Detection Riccardo De Monte, Davide Dalle Pezze, Gian Antonio Susto Abstract Real-time object detectors like YOLO achieve exceptional performance when trained on large datasets for multiple epochs. However, in real-world scenarios where data arrives incrementally, neural networks suffer from catastrophic forgetting, leading to a loss of previously learned knowledge. To address this, prior research has explored strategies for Class Incremental Learning (CIL) in Continual Learning for Object Detection (CLOD), with most approaches focusing on two-stage object detectors. However, existing work suggests that Learning without Forgetting (LwF) may be ineffective for one-stage anchor-free detectors like YOLO due to noisy regression outputs, which risk transferring corrupted knowledge. In this work, we introduce YOLO LwF, a self-distillation approach tailored for YOLO-based continual object detection. We demonstrate that when coupled with a replay memory, YOLO LwF significantly mitigates forgetting. Compared to previous approaches, it achieves state-of-the-art performance, improving mAP by +2.1% and +2.9% on the VOC and COCO benchmarks, respectively. K eywords Continual Learning \cdot Object Detection \cdot Deep Learning 1 Introduction Object detection is one of the most important research areas concerning computer vision, with many applications, from autonomous driving [ 1 ] to medical applications [ 2 ] . Thanks to Deep Learning, several object detectors reach remarkable performance [ 3 , 2 , 4 ] . Moreover, architectures such as RT-DETR [ 2 ] and YOLO [ 5 , 6 , 4 ] reach a good trade-off between accuracy and speed, allowing real-time object detection. In particular, YOLO is nowadays a widely used CNN-based object detector in most applications. Figure 1: Artificial example for the missing annotation problem in Continual Learning for Object Detection. Objects of classes for previous tasks might be considered as background in future tasks (e.g., class "dog" in Task 2 and Task 3). In a real scenario, the three images might be different, but the same issue might arise. On the other hand, these object detectors are trained in the ideal scenario where all the data are given at once. Such a situation doesn’t reflect real world scenarios in which models are required to learn incrementally, acquiring knowledge over time. For this purpose, Continual Learning (CL) methods aim to promote new knowledge acquisition and avoid the so-called catastrophic forgetting (CF) [ 7 ] , namely the tendency of neural networks to lose previously learned knowledge when exposed to new data. When considering Continual Learning for Object Detection (CLOD), the most common scenario is Class Incremental Learning (CIL), where an object detector learns from a stream of tasks, where each new task presents unseen object classes. However, contrary to single-label image classification, in CLOD, previous instances could reappear in future tasks with a different ground truth (new classes). This causes additional issues, such as the missing annotation problem , for which new images miss annotations for old classes, as in Fig. 1 . For this reason, consolidated methods like Experience Replay show poor performance, requiring adaptation to make it effective in the CLOD context [ 8 ] . CL literature for single-label image classification suggests that Learning without Forgetting (LwF) [ 9 ] , a kind of self-distillation approach can be a valid approach, especially if coupled with a replay buffer as in Dark Experience Replay (DER) [ 10 ] . However, previous works [ 11 , 8 ] show that distillation-based approaches might not be effective for one-stage and anchor-free object detectors as YOLO. Therefore, in our work, we rethink LwF in the context of the YOLO object detector, achieving the best performance for all CLOD benchmarks proposed in the literature and therefore showing the effectiveness of self-distillation also in the case of YOLO, especially if coupled with a replay buffer. Therefore, we make the following contributions: • We propose an adaptation of LwF for YOLO-based architectures, and we call it YOLO LwF. • We conduct extensive experiments on YOLOv8 architecture for VOC and COCO benchmarks, demonstrating how YOLO LwF can be a valid CL strategy. • By coupling YOLO LwF with a replay memory, our approach outperforms all competing methods, demonstrating superior performance and improving mAP by +2.1% and +2.9% on the VOC and COCO benchmarks, respectively. 2 Related Work Most of the CLOD literature focuses on two-stage object detectors such as Faster R-CNN [ 12 , 13 , 14 , 15 , 16 ] . All these works propose some solutions to prevent forgetting by leveraging the different parts of the Faster R-CNN architecture. Faster ILOD [ 12 ] proposes a distillation loss for each of the three components of the network, namely the feature extractor, the Region Proposal Network (RPN), and the class-level classification and bounding box regression network (RCN). [ 13 ] revisits Faster ILOD by modifying both the distillation loss for RCN and the usual loss of Faster R-CNN in order to solve the missing annotation problem. Instead, ABR [ 15 ] solves the missing annotation problem by saving box images from previous tasks and adding them to current images with both MixUp and Mosaic augmentation. Additionally, they replace the RPN distillation loss introduced in [ 12 ] with the so-called Attentive RoI Distillation (ARD). [ 16 ] proposes to train an additional model on the current task to then jointly distill the past knowledge of the old teacher and the new knowledge from the new teacher. However, nowadays, the trend in object detection is to design fast object detectors by allowing real-time detection. For this reason, other works focus on one-stage object detectors. In [ 11 ] , the authors highlight the problem of the noisy regression output for one-stage object detectors. They propose a distillation-based method, called SID, for the FCOS architecture [ 17 ] . In particular, they propose to distill the classification output and from intermediate features. Additionally, they propose an Inter-Related Distillation loss to maintain the inter-relation between features from different training instances. [ 18 ] introduce Elastic Response Distillation (ERD) for GFLV1 [ 19 ] and FCOS [ 17 ] . ERD performs an L2 distillation for the classification head and a KL divergence for the regression one. To overcome the noisy output problem highlighted in SID, ERD selects the candidate bounding boxes both at the classification head and regression one. This is done by applying the so-called Elastic Response Selection (ERS). [ 8 ] proposes a method called RCLPOD for YOLOv8 [ 6 ] architecture. RCLPOD doesn’t rely on output distillation, instead, it exploits Label Propagation combined with a balanced replay buffer to mitigate the forgetting. Additionally, they rely on intermediate feature distillation to improve stability. Following RCLPOD, in this work, we focus on the YOLO object detector, but contrary to previous works on one-stage object detectors, we propose a LwF-based approach exploiting fully the network output, namely also the regression one. 3 Methodology 3.1 Preliminaries Here, we provide a brief description of the main futures of modern YOLO detectors, justifying the need to adapt LwF in this context. In particular, we describe how the YOLO output is processed during both training and inference to highlight the reason why LwF should not be applied in a naive manner. Since in our experiments we test YOLOv8,

Read the complete text