State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detection

Jiaying Zhou, Qingchao Chen · 2025

Read State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detection by Jiaying Zhou, Qingchao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18012.txt.

Introduction

Jiaying Zhou and Qingchao Chen improve weakly supervised open vocabulary object detection. The model must find objects from new categories using only box labels for some classes and image level labels for others. Fixed text prototypes miss how an object's look changes with its state, and proposed boxes carry scene context that text labels lack.

Their method enriches prototypes with state aware descriptions, such as a sleeping cat, and with scene aware descriptions, such as a cat on a sofa. A soft alignment step then matches regions and text more consistently. Together the two prototype upgrades sharpen discrimination and improve detection results.

Catalog record

CREATORJiaying Zhou, Qingchao Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detection Jiaying Zhou, Qingchao Chen Abstract Open-Vocabulary Object Detection (OVOD) aims to generalize object recognition to novel categories, while Weakly Supervised OVOD (WS-OVOD) extends this by combining box-level annotations with image-level labels. Despite recent progress, two critical challenges persist in this setting. First, existing semantic prototypes, even when enriched by LLMs, are static and limited, failing to capture the rich intra-class visual variations induced by different object states (e.g., a cat’s pose). Second, the standard pseudo-box generation introduces a semantic mismatch between visual region proposals (which contain context) and object-centric text embeddings. To tackle these issues, we introduce two complementary prototype enhancement strategies. To capture intra-class variations in appearance and state, we propose the State-Enhanced Semantic Prototypes (SESP), which generates state-aware textual descriptions (e.g., ”a sleeping cat”) to capture diverse object appearances, yielding more discriminative prototypes. Building on this, we further introduce Scene-Augmented Pseudo Prototypes (SAPP) to address the semantic mismatch. SAPP incorporates contextual semantics (e.g., ”cat lying on sofa”) and utilizes a soft alignment mechanism to promote contextually consistent visual-textual representations. By integrating SESP and SAPP, our method effectively enhances both the richness of semantic prototypes and the visual-textual alignment, achieving notable improvements. 1 Introduction Figure 1 : Semantic Prototype Augmentation for Weakly Supervised Open-Vocabulary Object Detection. (a) State-Enhanced Semantic Prototypes (SESP). Standard category names (e.g., ”cat”) inadequately capture intra-class visual diversity across various object states (e.g., ”sleeping cat,” ”sitting cat”). SESP addresses this by augmenting prototypes with state-aware textual descriptions for more discriminative representations. (b) Scene-Augmented Pseudo Prototypes (SAPP). Standard pseudo-boxes (max-size proposals used in image classification data) often include substantial contextual information, leading to misalignment with class-centric prototypes. SAPP resolves this by incorporating contextual semantics to generate pseudo prototypes and using a soft alignment mechanism to bridge the visual-textual gap. Object detection aims to localize and classify all objects of interest within an image. Conventional detectors are trained under closed-set assumption, meaning they perform well only on a predefined set of categories and struggle to generalize to unseen classes, particularly in terms of classification accuracy 50 . This inherent limitation has driven the development of open-vocabulary object detection (OVOD), which aims to localize and recognize novel objects beyond the pre-defined categories. OVOD commonly exploits pre-trained vision-language models (VLMs) for recognition by substituting the learnable classification head with text embeddings of category names (e.g., “cat”, “car”). Detection is then performed by aligning region proposals with these semantic embeddings through cross-modal similarity, enabling the recognition of categories unseen during training. To improve generalization to novel classes, Weakly Supervised Open-Vocabulary Object Detection (WS-OVOD) leverages image-level labels, typically sourced from large-scale image classification datasets containing unseen categories, as a labeling-efficient alternative to bounding box annotations. In this setting, max-size proposals derived from classification data are retained and aligned with semantic prototypes to bridge the gap between image-level supervision and object-level recognition. Despite growing interest in this paradigm, two critical challenges remain unresolved. First, category names often convey only coarse or underspecified semantic information. Previous methods have attempted to enrich semantic prototypes by prompting large language models (LLMs) to generate generic textual descriptions. However, such descriptions remain static and fail to reflect the rich intra-class visual variations induced by object states and contextual factors. For example, as illustrated in Fig. 1 (a), a cat may appear in various poses—such as sitting, sleeping, or grooming—each corresponding to distinct visual configurations. To overcome this limitation, we propose the State-Enhanced Semantic Prototypes (SESP), in which category-specific textual descriptions are generated by prompting a language model with diverse object states (e.g., “a sleeping cat is curled or stretched out, typically appearing relaxed and content”, or analogous prompts for states such as sitting, running, etc.). These state-aware descriptions are incorporated into the semantic prototypes alongside standard (state-agnostic) category descriptions. By explicitly modeling intra-class variations induced by object states, SESP better reflects the visual diversity of object instances, leading to more discriminative and informative semantic prototypes. Second, the common pseudo-box strategy in WS-OVOD, which selects the max-size region proposal as the pseudo-box, introduces a persistent semantic mismatch between visual and textual representations. As shown in Fig. 1 (b), although these max-size proposals typically cover the target object, they also encompass surrounding context that is absent from the object-centric semantic prototypes. Typical prototypes, even when enhanced, focus solely on the object itself, neglecting the implicit contextual information present in the visual features. To mitigate this issue, we extend the original semantic prototypes by introducing a group of scene-augmented pseudo prototypes (SAPP). These pseudo prototypes incorporate contextual semantics while remaining anchored around the target object(e.g., “cat lying on sofa”). During training, a soft alignment mechanism is applied to associate the scene-augmented pseudo prototypes with corresponding region proposals, promoting more contextually consistent visual-textual representations and thereby reducing the semantic gap. By integrating state-enhanced semantic prototypes (SESP) with scene-augmented pseudo prototypes (SAPP), our method effectively addresses two key challenges in weakly supervised open-vocabulary object detection: intra-class variation due to object states and semantic mismatch from contextual discrepancy, leading to consistent performance gains on both base and novel categories. The main contributions of this work are summarized as follows: • We design state-enhanced semantic prototypes that explicitly incorporate descriptions of different object states, enabling richer and more flexible category representations. • We propose scene-augmented pseudo prototypes to utilize contextual scene information, enhancing the semantic expressiveness of detected objects. • Comprehensive experiments on WS-OVOD benchmarks demonstrate that our method surpasses existing approaches, validating its effectiveness. 2 Related Work 2.1 Open-Vocabulary Object Detection Open-vocabulary object detection aims to enable the model to generalize to arbitrary categories through cross-modal alignment, detecting objects unseen during training. This task was first proposed by OVR-CNN 54 , which adapted Faster R-CNN into an open-vocabulary framework by leveraging image-caption datasets. Existing solutions broadly fall into four categories: (1) fine-tuning vision-language models (VLMs), (2) knowledge distillation, (3) prompt learning, and (4) pseudo-labeling under weak supervision. Fine-tuning-based approaches adapt pretrained VLMs by constructing region-level data pairs and fine-tuning VLMs using contrastive objectives to achieve region-level alignment 23 ; 30 ; 59 ; 33 . Although these methods achieve competitive performance, they require extensive labeled data and substantial computational resources. Moreover, the domain gap between pretraining corpora and detection scenarios can hinder effective transfer. Knowledge distillation paradigms exploit the zero-shot generalization ability of large VLMs to transfer knowledge to object detectors. Techniques such as ViLD 16 align detector region features with CLIP embeddings, while methods like BARON 52 aggregate contextually related proposals to enhance alignment. Hierarchical distillation frameworks 37 ; 51 ; 13 further integrate multi-level knowledge transfer, encompassing instance, region, and global semantics. Other distillation variants, e.g., DK-DETR 26 and CAKE 36 , refine semantic and relational knowledge transfer, emphasizing category-specific nuances. Despite their efficiency, these methods often rely on fixed VLM features and may struggle to adapt to fine-grained visual distinctions in complex scenes. Prompt-learning-based methods aim to enrich semantic category representations beyond simple name embeddings by generating descriptive textual features. For instance, MM-OVOD 22 leverages large language models (LLMs) to create multiple descriptive prompts per category, while DVDet 21 produces fine-grained descriptors to integrate into embeddings. Hierarchical semantics are exploited by SHiNe 32 to provide structured prompts. Additionally, learnable prompts

Read the complete text