Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection

Duc Thanh Pham, Hong Dang Nguyen, Nhat Minh Nguyen Quoc, Linh Ngo Van, Sang Dinh Viet, Duc Anh Nguyen · 2025

Read Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection by Duc Thanh Pham, Hong Dang Nguyen, Nhat Minh Nguyen Quoc, Linh Ngo Van, Sang Dinh Viet, Duc Anh Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08562.txt.

Introduction

Duc Thanh Pham and co-authors published a computing paper in 2025 on teaching object detectors to learn new classes step by step. New objects appear in the real world all the time, and detectors must learn them without forgetting old ones. Past step-by-step detectors were slow or weak in practice. The team built a transformer-based detector to fix both faults.

Their system, called Hier-DETR, draws on a pattern called neural collapse to handle uneven class counts. It also uses the ranked links between class labels. The authors say the design keeps speed up while matching strong rivals in accuracy. The work aims to make step-by-step detection usable outside the lab.

Catalog record

CREATORDuc Thanh Pham, Hong Dang Nguyen, Nhat Minh Nguyen Quoc, Linh Ngo Van, Sang Dinh Viet, Duc Anh Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08562

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection Duc Thanh Pham, Hong Dang Nguyen, Nhat Minh Nguyen Quoc, Linh Ngo Van, Sang Dinh Viet, Duc Anh Nguyen Abstract Recently, object detection models have witnessed notable performance improvements, particularly with transformer-based models. However, new objects frequently appear in the real world, requiring detection models to continually learn without suffering from catastrophic forgetting. Although Incremental Object Detection (IOD) has emerged to address this challenge, these existing models are still not practical due to their limited performance and prolonged inference time. In this paper, we introduce a novel framework for IOD, called Hier-DETR : Hier archical Neural Collapse De tection Tr ansformer, ensuring both efficiency and competitive performance. First, we propose a Hierarchical Neural Collapse Tree, inspired by the general orthogonal frame characteristics of the convergence of the class means, and the semantic hierarchical relationship between classes. After each query is labeled using the Hungarian Matching Algorithm, we pre-allocate new class classifiers of queries for each decoder’s output layer as Hierarchical Neural Collapse Tree and fix this architecture as knowledge distillation while guiding each query to its class and super-class prototypes by Proxy-NCA. We conduct extensive experiments on COCO 2017, MTSD datasets and demonstrate that Hierarchical Neural Collapse helps model’s queries more ”stable” in Hungarian Matching, Hier-DETR achieves state-of-the-art performance in IOD settings: 48.4 AP (70+10) in COCO and 45.46 AP (150+71) in MSTD, outperforming with remarkable + 8 AP and + 7.5 AP improvement compared to the previous best method. 1 Introduction In recent years, there exist numerous significant advancements in object detection (OD) Wang et al. 2024 , Zong et al. 2023 . However, these models are designed to train from a fixed dataset based on the assumption that all objects’ class data are available at the training phase. Existing studies demonstrate that object detection frequently encounters continual learning scenarios in real-world implementations, especially in self-driving automobiles where novel traffic signs are regularly introduced. In light of this, incremental object detection such as: Liu et al. 2023 , Kim et al. 2024 has been proposed to tackle this problem. Currently, state-of-the-art detection models are the family of transformer-based detection models: DETR Carion et al. 2020 , Deformable DETR Zhu et al. 2020 , DN-DETR Li et al. 2022 , DINO Zhang et al. 2022 , Co-DETR Zong et al. 2023 , RT-DETR Wang et al. 2024 . Despite its outstanding results in several detection benchmarks, DETR has 2 main problems: its poor convergence and the ”black box” meaning of queries. Many methods have been proposed, such as alternating attention mechanism Zhu et al. 2020 , introducing new query structure (positional query and content query) Liu et al. 2022 , reducing instability in Hungarian matching by: denoising queries Li et al. 2022 , constrastive queries Zhang et al. 2022 , one-to-many labels positive queries Zong et al. 2023 . Besides, in continual learning, there are 2 most common strategies to reduce catastrophic forgetting, which are Knowledge Distillation (KD): Li and Hoiem 2018 ; Douillard et al. 2020 ; Zhao et al. 2020 and Exemplar Replay (ER): Rebuffi et al. 2017 ; Liu et al. 2020 ; Wang et al. 2022 ; Castro et al. 2018 . To be specific, Knowledge Distillation applies regularization to maintain prior knowledge while training the model in new tasks, this helps preserve feature maps of the model in new tasks resembling its upstream tasks. In addition, Exemplar Replay is a way to store old data from previous tasks in buffers, that models can use in current tasks to prevent forgetting. In comparison with traditional continual learning, incremental object detection requires models not only to classify objects into their corresponding classes but also to detect their positions in images (their bounding boxes). Furthermore, it also differs from standard detection, in that models need to be able to train incrementally while avoiding catastrophic forgetting upstream tasks’ knowledge. Due to its importance and difficulties, incremental object detection has attracted considerable research interest. CL-DETR Liu et al. 2023 has shown that directly applying ER and KD methods of continual learning to detection models is not effective. Therefore, they proposed new methods of KD in object detection by pseudo labeling: using a model in the first task (task 0) to generate the most confident foreground predictions as pseudo labels and merge them with the ground-truth labels of the current task. Besides, there exists a notable method of ER in continual object detection such as: SDDGR Kim et al. 2024 used diffusion models as a new way of data-replaying. Such methods suffer from both poor performance and low computational efficiency in inference time, which is impractical in real-life implementations. In addition, despite the semantic relationships between classes often forming a hierarchical structure, recent studies have not effectively utilized this inherent characteristic, particularly in the context of object detection. In contrast, assuming that we already have the hierarchical information of all classes’ labels, leveraging it to enhance performance while maintaining efficiency in the continual learning settings is more challenging. To address the above problems, we propose Hier-DETR, a model for incremental object detection that achieves a good balance between high performance and efficiency during inference. Firstly, we use a new backbone: RT DETR Wang et al. 2024 , and adapt it to CL settings. Though, to our best knowledge, Co-DERT Zong et al. 2023 is the SOTA of the object detection model but is not efficient for real-time running, RT-DETR is still the first real-time end-to-end object detector, that can maintain equilibrium between speed and accuracy. Secondly, we introduce a novel technique namely Hierarchical Neural Collapse (HNC) for boosting both the performance and efficiency of OD. Using neural collapse (NC) to improve efficiency Yang et al. 2023 ; Yang et al. 2022 is an emerged technique, that pre-assigns fixed prototypes as the structure of simplex Equiangular Tight Frame (ETFs) for each class and preserves them as consistent prior throughout incremental learning. Nevertheless, the implementation of neural collapse within a hierarchical structure has not been investigated. We propose a strategy to initialize Hierarchical Neural Collapse in incremental object detection. Our approach is proven to alleviate catastrophic forgetting by retaining information from upstream tasks while simultaneously balancing with downstream knowledge thanks to the characteristics of NC. Additionally, our experiments also indicate that integrating HNC into various DETR-based models reduces the instability of queries during the Hungarian matching process, thereby contributing to faster convergence. We empirically demonstrate the advantage of Hier-DETR in MTSD (Mapillary Traffic Sign Dataset) Ertler et al. 2020 and COCO Lin et al. 2014 datasets with various settings. The results show that Hier-DETR outperforms 48.4 in COCO and 45.46 in MSTD datasets in comparison with recent SOTAs. The ablation study also indicates that Hierarchical NC boosts models’ stability in Hungarian Matching. To summarize, our main contributions are as follows: 1. We proposed Hier-DETR, an innovative continual learning strategy for object detection based on NC and a hierarchical structure of classes. The method can be easily applied to any DETR-based model to achieve outstanding performance and faster convergence. 2. To the best of our knowledge, this paper is the first to apply the hierarchical semantic meaning of classes’ labels with NC. We expect

Read the complete text