Solving Instance Detection from an Open-World Perspective
Qianqian Shen, Yunhan Zhao, Nahyun Kwon, Jeeeun Kim, Yanan Li, Shu Kong · 2025
Read Solving Instance Detection from an Open-World Perspective by Qianqian Shen, Yunhan Zhao, Nahyun Kwon, Jeeeun Kim, Yanan Li, Shu Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00359.txt.
Introduction
Qianqian Shen and co-authors tackle instance detection: finding a named object in a fresh scene photo, given sample images of that object. The task serves robots and mixed reality tools, but test scenes are new and sample images can look quite unlike the scene.
Their method, IDOW, starts from a large pretrained model that finds most objects but matches them poorly. They adapt it with a form of metric learning and fresh training tricks. Fake views of the target enrich the samples, and decoy objects serve as bad matches to learn from. Tests show clear gains over past methods on this open world task.
Catalog record
| CREATOR | Qianqian Shen, Yunhan Zhao, Nahyun Kwon, Jeeeun Kim, Yanan Li, Shu Kong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Solving Instance Detection from an Open-World Perspective Qianqian Shen, Yunhan Zhao, Nahyun Kwon, Jeeeun Kim, Yanan Li, Shu Kong Abstract Instance detection (InsDet) aims to localize specific object instances within a novel scene imagery based on given visual references. Technically, it requires proposal detection to identify all possible object instances, followed by instance-level matching to pinpoint the ones of interest. Its open-world nature supports its broad applications from robotics to AR/VR but also presents significant challenges: methods must generalize to unknown testing data distributions because (1) the testing scene imagery is unseen during training, and (2) there are domain gaps between visual references and detected proposals. Existing methods tackle these challenges by synthesizing diverse training examples or utilizing off-the-shelf foundation models (FMs). However, they only partially capitalize the available open-world information. In contrast, we approach InsDet from an Open-World perspective, introducing our method IDOW. We find that, while pretrained FMs yield high recall in instance detection, they are not specifically optimized for instance-level feature matching. Therefore, we adapt pretrained FMs for improved instance-level matching using open-world data. Our approach incorporates metric learning along with novel data augmentations, which sample distractors as negative examples and synthesize novel-view instances to enrich the visual references. Extensive experiments demonstrate that our method significantly outperforms prior works, achieving > > 10 AP over previous results on two recently released challenging benchmark datasets in both conventional and novel instance detection settings. 1 Introduction Instance detection (InsDet) aims to localize object instances of interest in novel scene imagery (Fig. 1 ), where these objects are specified by some visual references (as known as support templates) [ 53 , 34 , 3 , 48 , 26 ] . Its open-world nature supports its wide-ranging applications in robotics and AR/VR. For example, a robot can be commanded to search for a customer’s suitcase at an airport [ 26 ] , or my-keys in my cluttered bedroom [ 62 ] . Its open-world nature also presents significant challenges: during training, one has no knowledge of testing data distribution, which can be instantiated by the unknown scene imagery [ 9 ] , novel object instances encountered only in testing [ 26 ] , and domain gaps between visual references and detected proposals [ 48 ] . Figure 1 : Instance Detection (InsDet) aims to localize specific object instances in novel scene imagery based on given visual references ( aka support templates). It is a challenging problem due to its open-world nature: the testing scene images are unseen during training and thus are unknown to InsDet models, and visual references and detected proposals have domain gaps (e.g., due to occlusions and arbitrary lighting conditions in the latter). (a) background imagery [ 11 , 9 ] (b) object images [ 26 ] (c) Foundation Models [ 48 ] (d) Solving Instance Detection in the Open World ( ours ) Figure 2: Existing InsDet methods leverage the open-world information in different aspects, such as (a) background image sampling (from the open world) to synthesize training data [ 11 , 9 ] , (b) object image sampling (from the open world) to learn feature representations [ 26 ] , and (c) foundation model utilization (pretrained in the open world) for proposal detection and instance-level feature matching [ 48 ] . (d) As FMs are not specifically designed for instance-level feature matching required by InsDet, we propose to adapt them by leveraging rich data sampled from the open world. We gather data from multiple sources: (1) any available visual references of instances in the CID setting, (2) abundant multi-view object images sampled in the open world similar to (b) , (3) synthetic data by training NeRF [ 36 , 1 ] to generate novel-view images based on the given instances, (4) distractors by running FMs (esp. SAM [ 22 ] ) on random open-world imagery to generate random object-like proposals (Fig. 5 ). We use the data above to adapt FM through metric learning. The technical novelty of our work lies in (3) and (4), as well as the design choice of metric learning to adapt FMs for InsDet. Status quo. The literature of InsDet has two settings and existing InsDet methods attempt to address the open world in different aspects (Fig. 2 ). The conventional instance detection (CID) setting provides visual references of object instances for training. Yet, the testing scene is still unknown in training time. Therefore, early method Cut Paste Learn (CPL) collects random background images to model diverse testing scenes (Fig. 2 a), cuts instances from the visual references, and pastes them on these background images to construct a synthetic training set [ 11 , 9 ] . It is worth noting that background image sampling is utilizing open-world information in a way of data collection in the open world. On the other hand, the novel instance detection (NID) setting, introduced recently [ 26 ] , studies detecting novel object instances that are specified only in testing time where a trained model is not allowed to be tuned further. To approach NID, VoxDet [ 26 ] uses large-scale synthetic data (Fig. 2 b) to learn 3D voxel representations from visual references and a matching function between 2D proposals and 3D voxels. Recent work OTS-FM [ 48 ] uses off-the-shelf pretrained foundation models (Fig. 2 c) for InsDet. Specifically, it uses SAM [ 22 ] to detect object proposals in scene imagery, and DINOv2 [ 38 ] to represent visual references and proposals for instance-level feature matching. OTS-FM is a non-learned method and hence is applicable in both CID and NID settings. Importantly, it achieves the state-of-the-art performance on the recently released benchmark HR-InsDet [ 48 ] . The open-world perspective. As illustrated in Fig. 1 , the challenges of InsDet lie in its open-world nature: the unknown testing data distribution, and domain gaps between visual references and detected proposals. To address the open-world challenges, existing methods take different strategies to exploit the open-world information (Fig. 2 a-c). For example, to address the unknown testing scene imagery, some methods sample diverse background images [ 11 , 9 ] , demonstrating a practice of sampling data in the open world. To mitigate the domain gap between detected proposals and visual references, [ 26 ] attempts to learn features for instance-level matching by utilizing external datasets (ShapeNet [ 5 ] and ABO [ 8 ] ). This data utilization can be considered as data collection in the open world. For better proposal detection and more discriminative feature representations, [ 48 ] adopts off-the-shelf pretrained foundation models (FMs) SAM [ 22 ] and DINOv2 [ 38 ] , respectively. One can think of FMs as being pretrained in the open world. Different from existing methods which have not fully capitalized the open-world information, we take the open-world perspective as the first principle and solve InsDet by exploiting the Open World (Fig. 2 d). We call our approach IDOW . Insights and novelties. Our insights are rooted in the understanding of InsDet’s open-world nature and the practice of existing methods. Existing methods address open-world issues to some extent (Fig. 2 ), particularly by sampling data in the open world [ 9 , 23 , 26 ] and