Reference-based Category Discovery: Unsupervised Object Detection with Category Awareness

Yichen Li, Ying Fu, Qiankun Liu · 2026

Read Reference-based Category Discovery: Unsupervised Object Detection with Category Awareness by Yichen Li, Ying Fu, Qiankun Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04606.txt.

Introduction

Yichen Li, Qiankun Liu, and Ying Fu propose an unsupervised object detector that knows what categories it is finding, without using any hand-labelled training data. One-shot detectors need labelled examples, while general unsupervised methods propose boxes but cannot say what kind of object each box contains.

Their method uses unlabelled reference images. A similarity score between a predicted object and the references guides learning. It pushes the model toward features tied to each class, through a loss designed for that purpose. The same framework can also run in a category-blind mode when no references are available. Number-based and visual tests on both modes show the approach works. It narrows the gap between label-free detection and labelled-data accuracy.

Catalog record

CREATORYichen Li, Ying Fu, Qiankun Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reference-based Category Discovery: Unsupervised Object Detection with Category Awareness Yichen Li, Ying Fu, Qiankun Liu Abstract Existing unsupervised methods fail to generate category labels or learn category-aware features. In contrast, one-shot detectors depend on manual category labels to perform sample assignment and classification loss calculation. As a result, these detectors are not suitable for unsupervised scenarios where no category labels are available. To overcome these limitations, we propose Ref erence-based C ategory D iscovery (RefCD), an unsupervised detector that enables category-aware 1 1 1 Unlike traditional methods that detect objects with pre-defined category labels, this work detects objects that share the same category with the unlabeled reference image provided on-the-fly. detection without any manually annotated labels. Specifically, It leverages feature similarity between predicted objects and unlabeled reference images. Unlike previous unsupervised methods that lack category awareness and one-shot methods which require labeled data, RefCD introduces a carefully designed Feature Similarity (FS) loss, which explicitly guides the learning of potential category-aware features, enabling RefCD to acquire category-aware information in an unsupervised paradigm without category labels. Additionally, RefCD supports category-agnostic detection without reference images, serving as a unified framework. Comprehensive quantitative and qualitative analysis of category-aware and category-agnostic detection results demonstrates its effectiveness. 1 Introduction Object detection aims to localize and categorize objects within images. Traditional methods heavily rely on manually annotated datasets 26 ; 38 , which are not only time-consuming and labor-intensive but also suffer from poor generalization to unseen categories. Moreover, these methods are confined to predefined categories, inherently limiting their scalability. To reduce the reliance of object detection on manual annotations, unsupervised object detection has been proposed. Non-deep learning object detection method 42 uses graph-based algorithms to detect objects. Early deep learning based studies 43 ; 3 adopt complex architectures for detector training and generate coarse pseudo-boxes. Recent methods 45 ; 44 explore purely unsupervised learning for object detection, where self-supervised Vision Transformers (ViT) are used to generate pseudo-boxes for detector training. Specifically, TokenCut 45 and CutLER 44 utilize attention maps from DINO 6 to perform normalized cuts for pseudo-box generation. Despite these advances, all of these unsupervised methods are confined to category-agnostic detection and lack the capability of object classification, as illustrated in Figure 1 (b). Figure 1: Comparison of different detection paradigms. (a) Previous one-shot detection methods 47 ; 19 ; 28 ; 27 rely on manual annotations for supervised training and one-shot fine-tuning to enable detectors to learn novel categories. (b) Previous unsupervised methods 45 ; 44 employ unsupervised models to generate pseudo boxes for detector training. They localize all foreground objects without category awareness. (c) The proposed unsupervised method introduces reference images and feature similarity loss, constraining the model to identify similar objects through feature similarity without any category label. (Cls. and Reg. denote Classification and Regression, respectively.) To improve generalization of unseen categories, one-shot detection 47 ; 19 ; 27 ; 28 and open-vocabulary detection 48 ; 18 ; 30 ; 53 have been proposed to address the limitations of closed-set by identifying objects beyond predefined categories. Open-vocabulary methods leverage pre-trained vision-language models 11 ; 36 , they are constrained by a large number of model parameters and fine-grained annotations, resulting in substantial labor and computational costs. However, one-shot detection methods also require fine-grained annotated data and even one-shot fine-tuning (Figure 1 (a)). They neither reduce annotation costs nor avoid the requirement for high annotation accuracy. In contrast, our work focuses on unsupervised object detection, aiming to completely eliminate reliance on manual annotations, which is a key drawback that existing one-shot methods fail to overcome. To bridge this gap, this paper proposes an unsupervised Ref erence-based C ategory D iscovery (RefCD) detector. As shown in Figure 1 (c), RefCD enables category-aware object detection without any manual annotations. Unlike one-shot methods that require labeled data or unsupervised methods that ignore category information, RefCD leverages feature similarity between predicted objects and unlabeled reference images to guide the learning of category-aware features. Specifically, it uses reference images to introduce contextual semantics and proposes a Feature Similarity (FS) loss function. This loss is pivotal as it differs fundamentally from supervised loss functions, which rely on explicit category labels. In contrast, FS loss operates in an unsupervised manner, guiding the detector to learn potential category-specific features by maximizing similarity between objects of the same implicit category and minimizing it for others. The proposed RefCD aims to locate all objects via category-agnostic detection and then identify specific category objects through category-aware object matching. In the category-aware object matching, we match the category-agnostic detection results with the reference image based on their feature similarities. During training, an image patch cropped from the target image with a randomly selected pseudo box is used as a reference image to guide the detector to locate the object within the selected pseudo box. Other objects that share high feature similarities with the reference image are also identified. In the absence of category labels, the novel FS loss explicitly guides the detector to learn potential category-specific features. As a result, RefCD can infer category information from features without relying on explicit category labels. Additionally, RefCD can perform category-agnostic object detection, a task enhanced by the category-aware feature learning. Our main contributions are as follows: • We propose an unsupervised reference-based detection method that enables open-set object detection with reference images without requiring any annotations. • We introduce a feature similarity loss that encourages the detector to mine the potential category information of different objects during unsupervised training. • Experimental results show that RefCD achieves state-of-the-art performance in unsupervised object detection, even competitive with some supervised detectors. 2 Related Work This section briefly reviews related works for object detection from different aspects, including unsupervised object detection and one-shot object detection. 2.1 One-shot Object Detection One-shot object detection methods are proposed to reduce the annotation cost of novel categories and break the limitations of pre-defined closed-set categories. Traditional training-based methods 47 ; 49 ; 19 pre-train on base categories and then fine-tune with one-shot samples from novel categories. SimTrans 7 applies similarity transfer to the learning of novel categories. SimFormer 8 proposes a complementary loss to explore the positions of novel categories based on the pixels of base categories. But the lack of strong supervision for novel classes affects the performance of detectors when handling novel categories. Recent methods have introduced reference-based object detection 34 ; 48 ; 24 ; 27 ; 28 , where flexible reference images serve as prompts during inference, replacing additional one-shot fine-tuning. Early methods 22 leverage single object tracking methods to detect all objects in the image that share the same category as the reference image. More recently, Siamese-DETR 27 integrates reference image features as queries into a Transformer-based framework to detect objects of the same category, while SINE 28 introduces an in-context interaction module and ID queries to enhance model accuracy and flexibility. However, the aforementioned methods also rely on labor-intensive and fine-grained category annotations to help the model accurately associate references with target objects. In contrast, the proposed method eliminates expensive annotation costs and effectively detects objects of interest categories through unsupervised training. 2.2 Unsupervised Object Detection Unsupervised object detection 46 ; 3 ; 40 ; 43 ; 44 ;

Read the complete text