YOLO-UniOW: Efficient Universal Open-World Object Detection

Lihao Liu, Juexiao Feng, Hui Chen, Ao Wang, Lin Song, Jungong Han, Guiguang Ding · 2024

Read YOLO-UniOW: Efficient Universal Open-World Object Detection by Lihao Liu, Juexiao Feng, Hui Chen, Ao Wang, Lin Song, Jungong Han, Guiguang Ding (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20645.txt.

Description

Abstract Traditional object detection models are constrained by the limitations of closed-set datasets, detecting only categories encountered during training. While multimodal models have extended category recognition by aligning text and image modalities, they introduce significant inference overhead due to cross-modality fusion and still remain restricted by predefined vocabulary, leaving them ineffective at handling unknown objects in open-world scenarios.

Catalog record

CREATORLihao Liu, Juexiao Feng, Hui Chen, Ao Wang, Lin Song, Jungong Han, Guiguang Ding
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20645

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

YOLO-UniOW: Efficient Universal Open-World Object Detection Lihao Liu, Juexiao Feng, Hui Chen, Ao Wang, Lin Song, Jungong Han, Guiguang Ding Abstract Traditional object detection models are constrained by the limitations of closed-set datasets, detecting only categories encountered during training. While multimodal models have extended category recognition by aligning text and image modalities, they introduce significant inference overhead due to cross-modality fusion and still remain restricted by predefined vocabulary, leaving them ineffective at handling unknown objects in open-world scenarios. In this work, we introduce Universal Open-World Object Detection (Uni-OWD), a new paradigm that unifies open-vocabulary and open-world object detection tasks. To address the challenges of this setting, we propose YOLO-UniOW, a novel model that advances the boundaries of efficiency, versatility, and performance. YOLO-UniOW incorporates Adaptive Decision Learning to replace computationally expensive cross-modality fusion with lightweight alignment in the CLIP latent space, achieving efficient detection without compromising generalization. Additionally, we design a Wildcard Learning strategy that detects out-of-distribution objects as “unknown” while enabling dynamic vocabulary expansion without the need for incremental learning. This design empowers YOLO-UniOW to seamlessly adapt to new categories in open-world environments. Extensive experiments validate the superiority of YOLO-UniOW, achieving achieving 34.6 AP and 30.0 AP r \text{AP}_{r} on LVIS with an inference speed of 69.6 FPS. The model also sets benchmarks on M-OWODB, S-OWODB, and nuScenes datasets, showcasing its unmatched performance in open-world object detection. Code and models are available at https://github.com/THU-MIG/YOLO-UniOW . 1 Introduction Object detection has long been one of the most fundamental and widely applied techniques in the field of computer vision, with extensive applications in security [ 46 ] , autonomous driving [ 57 ] , and medical imaging [ 13 ] . Many remarkable works have achieved breakthroughs for object detection, such as Faster R-CNN [ 41 ] , SSD [ 30 ] , RetinaNet [ 26 ] , etc. Figure 1 : Speed-Accuracy Trade-off Curve . Comparison of YOLO-UniOW and recent methods in speed and accuracy on the LVIS minival dataset. Inference speed is measured on a single NVIDIA V100 GPU without TensorRT. Circle size indicates model size. Figure 2 : Comparisons of Detection Framework. (a) Open-vocabulary detector with cross-modality fusion. (b) Our efficient open-vocabulary detector with Adaptive Decision Learning. (c) Open-world and open-vocabulary detectors. (d) Our Uni-OWD detector for both open-vocabulary and open-world tasks. In recent years, the YOLO (You Only Look Once) [ 40 , 1 , 20 , 51 ] series of models has gained widespread attention for its outstanding detection performance and real-time efficiency. The recent YOLOv10 [ 51 ] establishes a new standard for object detection by employing a consistent dual assignment strategy, achieving efficient NMS-free training and inference. However, traditional YOLO-based object detection models are often confined to a closed set definition, where objects of interest belong to a predefined set of categories. In practical open-world scenarios, when encountering unknown categories that have not been encountered in the training datasets, these objects are often misclassified as background. This inability of models to recognize novel objects can also negatively impact the accuracy of known categories, limiting their robust application in real-world scenarios. Thanks to the development of vision-language models, such as [ 39 , 19 , 47 , 3 ] , combining their open-vocabulary capabilities with the efficient object detection of YOLO presents an appealing and promising approach for real-time open-world object detection. YOLO-World [ 4 ] is a pioneering attempt, where YOLOv8 [ 20 ] is used as the object detector, and CLIP’s text encoder is integrated as an open-vocabulary classifier for region proposals (i.e., anchors in YOLOv8). The decision boundary for object recognition is derived from representations of class names generated by CLIP’s text encoder. Additionally, a vision-language path aggregation network (RepVL-PAN) using re-parameterization [ 6 , 50 ] is introduced to comprehensively aggregate text and image features for better cross-modality fusion. Although YOLO-World is effective for open-vocabulary object detection (OVD), it still relies on a predefined vocabulary of class names, which must include all categories that are expected to be detected. This reliance significantly limits its ability to dynamically adapt to newly emerging categories, as determining unseen class names in advance is inherently challenging, preventing it being truly open-world. Moreover, the inclusion of RepVL-PAN introduces additional computational costs, especially with large vocabulary sizes, making it less efficient for real-world applications. In this work, we first advocate a new setting of Universal Open-World Object Detection (Uni-OWD) , in which we encourage realizing open-world object detection (OWOD) and open-vocabulary object detection (OVD) with one unified model. Specifically, it emphasizes that the model can not only recognize categories unseen during training but also effectively classify unknown objects as “unknown”. Additionally, we call for a efficient solution following YOLO-World to meet the efficiency requirement in real-world applications. To achieve these, we propose a YOLO-UniOW model to achieve effective universal open-world detection but also enjoying greater efficiency. Our YOLO-UniOW emphasizes several insights for efficient Uni-OWD. (1) Efficiency . Except using recent YOLOv10 [ 51 ] for the more efficient object detector, we introduce a novel adaptive decision learning strategy, dubbed AdaDL, to wipe out the expensive cross-modality vision-language aggregation in RepVL-PAN, as illustrated in Fig. 2 (b). The goal of AdaDL is to adaptively capture task-related decision representations for object detection without sacrificing the generalization ability of CLIP. Therefore, we can well align the image feature and class feature directly in the latent CLIP space with no any heavy cross-modality fusion operations, achieving efficient and outstanding detection performance (see Fig. 1 ). (2) Versatility . The challenge of open-world object detection (OWOD) lies in differentiating all unseen objects with only one “unknown” category without any supervision about unknown objects . To solve this issue, we design a wildcard learning method that use a wildcard embedding to unlock generic power of open-vocabulary model. This wildcard embedding is optimized through a simple self-supervised learning, which seamlessly adapts to dynamic real-world scenarios. As shown in Fig. 2 (d), our YOLO-UniOW can not only benefit from the dynamic expansion of the known category set like YOLO-World, i.e . , open-vocabulary detection, but also can highlight any out-of-distribution objects with “unknown” category for open-world detection. (3) High performance . We evaluated our zero-shot open-vocabulary capability in LVIS [ 14 ] , and the open-world approach in benchmarks such as M-OWODB [ 44 ] , S-OWODB [ 16 ] , and nuScenes [ 2 ] . Experimental results show that our method can significantly outperform existing state-of-the-art methods for efficient OVD, achieving 34.6 AP , 30.0 \text{AP}_{r} on the LVIS dataset with the speed of 69.6 FPS. Besides, YOLO-UniOW can also perform well in both zero-shot and task-incremental learning for open-world evaluation. These well demonstrate the effectiveness of the proposed YOLO-UniOW. The contributions of this work are as follows: • We advocate a new setting of Universal Open-World Object Detection, dubbed Uni-OWD to solve the challenges of dynamic object categories and unknown target recognition with one unified model. We provide an efficient solution based on YOLO detector, ending up with our YOLO-UniOW. • We design a novel adaptive decision learning (AdaDL) strategy to adapt the representation of decision boundary into the task of Uni-OWD without sacrificing the generalization ability of CLIP. Thanks to AdaDL, we

Read the complete text