MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation
Sungmin Cho, Sungbum Park, Insoo Oh · 2025
Read MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation by Sungmin Cho, Sungbum Park, Insoo Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17866.txt.
Introduction
Sungmin Cho, with two co-authors, presents MUSE, a method that finds and outlines objects in images without any training on those objects. The system builds flat views from three-dimensional models of unseen items. It then matches those views against regions proposed from a test photo.
Matching joins class and patch features, and a final step adjusts each score by how reliable the region looks. The authors report that MUSE ranked first on three tracks of the BOP Challenge in 2025. Readers get a clear case that careful matching, rather than more training, can carry detection and outlining of new objects.
Catalog record
| CREATOR | Sungmin Cho, Sungbum Park, Insoo Oh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17866
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation Sungmin Cho, Sungbum Park, Insoo Oh Abstract In this work, we present MUSE ( M odel-based U ncertainty-aware S imilarity E stimation), a training-free framework for model-based zero-shot 2D object detection and segmentation. First, MUSE incorporates 2D multi-view templates from 3D unseen objects and 2D object proposals from the input query image, respectively. In the embedding stage, we propose a new feature embedding scheme which integrates class and patch embeddings. Specifically, the patch embeddings are normalized using the generalized mean pooling (GeM). In the matching stage, a joint similarity score is introduced, which integrates an absolute score and a relative score. Finally, we update the similarity score using an uncertainty-aware object prior. MUSE achieves state-of-the-art performance on the BOP Challenge 2025, ranking first in the Classic Core, H3, and Industrial tracks—without any additional training or fine-tuning. Therefore, we believe that MUSE is a promising framework for zero-shot 2D object detection and segmentation. Figure 1 : Comparison of detection performance on the BOP Classic Core benchmark. Each axis corresponds to a dataset, and values are mAP scores. It is demonstrated that MUSE (dotted red) consistently outperforms other methods in all datasets. 1 Introduction Recent advances in vision-language models [ 25 , 20 , 36 ] and large-scale pretraining models [ 6 , 32 , 37 ] have enabled impressive generalization to unseen categories. However, recognizing and localizing object instances is still challenging in real-world applications such as robotics [ 17 , 24 , 43 ] , augmented reality [ 29 , 28 ] , and industrial inspection [ 33 , 26 ] . To address this, the BOP Challenge [ 12 , 39 , 13 , 14 ] has introduced rigorous benchmarks for model-based zero-shot 2D object detection and segmentation. Usually, in model-based zero-shot 2D object detection and segmentation, there are four stages: Onboarding, proposal, embedding, and matching, respectively. The onboarding stage provides 2D templates from 3D model objects through predefined multi-view camera points. In the proposal stage, candidate proposals are generated from a query image. Specifically, Grounding DINO [ 25 ] and SAM [ 20 ] generate candidate object proposals for matching. In the embedding stage, a vision foundation model such as DINOv2 [ 32 ] converts the templates into template features. Using the same model, the proposals are also converted into query features for feature matching. In the matching stage, finally, template features and query features are matched using a similarity metric like the cosine similarity metric. The resulting similarity scores are aggregated across all camera views using simple heuristics such as top-K averaging, providing final predictions for all object candidates. As 3D object models and query images are not used in training, model-based zero-shot object recognition is very challenging. Previous methods [ 31 , 22 , 27 ] typically rely solely on cosine similarity between each proposal and all templates to identify the true class in the zero-shot setting. However, these approaches compute raw (absolute) similarity scores for each class independently, without considering the relative differences among classes like other end-to-end classification [ 38 , 21 ] . When the scores for different classes are close to each other, the prediction often fails to identify the true class. Because each proposal belongs to exactly one class, it is therefore reasonable to consider relative differences among classes in order to obtain more accurate results. Also, in the matching stage, the methods in [ 31 , 22 , 27 ] do not utilize to the proposal confidence which is provided by Grounding DINO [ 25 ] and SAM [ 20 ] . As the confidence indicates the reliability of the object proposal, it could be an object prior. Specifically, the proposal confidence is helpful in scenarios with cluttered backgrounds. Therefore, in this work, we improve the similarity score by weighting it with the proposal confidence. To overcome challenges in model-based zero-shot object recognition, we propose MUSE ( M odel-based U ncertainty-aware S imilarity E stimation), a model-based zero-shot detection framework that leverages pretrained vision foundation models without any additional training or fine-tuning. First, we introduce a novel feature integration scheme to handle class embeddings and patch embeddings together. Specifically, patch embeddings are aggregated using the generalized mean pooling (GeM). Then, they are combined with class embeddings, yielding new feature embeddings. Second, a novel joint similarity score metric is presented. The joint similarity score combines absolute and relative similarities together. The joint similarity score enables explicit reasoning over competing candidates [ 47 ] . Also, it improves the recognition of hard negative cases. Finally, we propose an uncertainty-aware similarity score metric by weighting the proposal confidence within a principled Bayesian framework. Through extensive evaluation, MUSE achieves robust and accurate zero-shot detection and segmentation, even in cluttered and visually ambiguous scenarios. In summary, the main contributions of this work are fourfold: • We propose a novel feature integration scheme between class embeddings and patch embeddings using the generalized mean pooling (GeM). • We introduce a joint similarity metric that combines absolute and relative similarities, enabling more effective discrimination even in hard negative candidates. • We propose an uncertainty-aware objectness prior using a principled Bayesian framework. • We achieve state-of-the-art results on the BOP Challenge 2025 at the time of submission, specifically in the Classic-Core, H3, and Industrial tracks. Figure 2 : Overview of the proposed framework. Templates and proposals are obtained in the onboarding stage and the proposal stage, respectively. Then, DINOv2 [ 32 ] yields class and patch embeddings from both templates and proposals. To integrate class and patch embedding into similarity metric, we introduce a generalized mean pooling (GeM). Finally, we further modify the similarity metric using both a joint similarity score scheme and an uncertainty-aware objectness prior, which yields state-of-the-art object detection results. 2 Related Work 2.1 Vision Foundation Model Vision foundation models have demonstrated strong generalization across diverse vision tasks [ 35 , 20 , 25 , 32 , 42 , 37 , 1 ] . Specifically, Segment Anything (SAM) [ 20 ] , Segment Anything 2 (SAM 2) [ 36 ] , Grounding DINO [ 25 ] , and DINOv2 [ 32 ] have been widely used in model-based recognition tasks due to their robust segmentation and representation capabilities. SAM [ 20 ] enables a prompt-based segmentation across a wide range of visual concepts. DINOv2 [ 32 ] provides strong class-level embeddings without task-specific supervision. Grounding DINO [ 25 ] further extends DINO [ 25 ] with open-vocabulary grounding capabilities. Recent models also have emerged to improve performance or efficiency [ 19 , 48 , 46 , 45 ] . For example, SAM-HQ [ 19 ] introduces high-quality masks. FastSAM [ 48 ] tries to optimize the original network for fast CNN-based segmentation. Florence models [ 46 , 45 ] also propose modified multi-modal networks for large-scale vision-language understanding. As our framework mainly focuses on a similarity metric between feature embeddings, MUSE is easily applicable to vision foundation models. 2.2 Model-based Zero-Shot Object Detection and Segmentation The BOP Challenge [ 13 ] introduces several benchmarks regarding detection and segmentation of unseen objects. Specifically, the target objects are not shown in the training. In inference, they