SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection
Xin LinChong ShiZuopeng Yang, Haojin TangZhili ZhouGuangzhou University [email protected], [email protected], [email protected], [email protected], [email protected] · 2025
Read SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection by Xin LinChong ShiZuopeng Yang, Haojin TangZhili ZhouGuangzhou University [email protected], [email protected], [email protected], [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00414.txt.
Introduction
Xin Lin and four co-authors teach a vision system to spot human object interactions it was never trained on, in this 2025 paper. Open vocabulary detection leans on big vision language models, but two flaws bite: only the last layer's features get used, wasting fine detail, and generated descriptions blur similar action classes together.
Their network first senses features at several depths, blending global meaning with local detail before matching them to text. A second module uses a language model to group and compare classes hierarchically, writing sharper descriptions that separate near twins. On two standard benchmarks the system set state of the art results for open vocabulary interaction detection. Code is promised for release, and the authors frame granularity plus discrimination as the twin keys.
Catalog record
| CREATOR | Xin LinChong ShiZuopeng Yang, Haojin TangZhili ZhouGuangzhou University [email protected], [email protected], [email protected], [email protected], [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00414
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection Xin LinChong ShiZuopeng Yang, Haojin TangZhili ZhouGuangzhou University [email protected], [email protected], [email protected], [email protected], [email protected] Abstract Recent open-vocabulary human-object interaction (OV-HOI) detection methods primarily rely on large language model (LLM) for generating auxiliary descriptions and leverage knowledge distilled from CLIP to detect unseen interaction categories. Despite their effectiveness, these methods face two challenges: (1) feature granularity deficiency, due to reliance on last layer visual features for text alignment, leading to the neglect of crucial object-level details from intermediate layers; (2) semantic similarity confusion, resulting from CLIP’s inherent biases toward certain classes, while LLM-generated descriptions based solely on labels fail to adequately capture inter-class similarities. To address these challenges, we propose a stratified granular comparison network. First, we introduce a granularity sensing alignment module that aggregates global semantic features with local details, refining interaction representations and ensuring robust alignment between intermediate visual features and text embeddings. Second, we develop a hierarchical group comparison module that recursively compares and groups classes using LLMs, generating fine-grained and discriminative descriptions for each interaction category. Experimental results on two widely-used benchmark datasets, SWIG-HOI and HICO-DET, demonstrate that our method achieves state-of-the-art results in OV-HOI detection. Codes will be released on GitHub . 1 Introduction Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions, providing an efficient way for human-centric scene understanding. It plays a crucial role in various computer vision tasks, such as assistive robotics Mascaro et al. 2023 ; Massardi et al. 2020 and video analysis Xi et al. 2023 ; Li et al. 2024a . Recently, the emerging field of open-vocabulary HOI (OV-HOI) detection, which broadens the scope of HOI detection by recognizing and associating objects beyond predefined categories, has gained increasing attention. Figure 1 : (a) The last layer capture high-level global semantics but contain fewer low-level local details compared to intermediate layers. The red dot marks the selected patch. (b) Category-level and part-level descriptions overlook inter-class similarity, leading to difficulty in distinguishing semantically similar classes. OV-HOI detection has made notable progress, largely due to the success of vision-language models (VLMs) like CLIP Radford et al. 2021 . Unlike zero-shot learning, which prohibits the access of any unannotated unseen classes during training, open-vocabulary learning leverages class name embeddings from VLMs for auxiliary supervision Wu et al. 2024 ; Zhu and Chen 2024 . Therefore, existing CLIP-based HOI detection methods naturally fall under OV-HOI detection. However, these methods face challenges in open-world scenarios because they rely on pretrained object detectors, and using category names as classifiers struggles to capture the variability of HOIs. To overcome these challenges, recent works Wang et al. 2022b ; Lei et al. 2024 have removed the need for pretrained detectors by extracting HOI features directly through image encoders while incorporating language priors from LLMs via text encoders. Despite these advancements, current OV-HOI detection methods without pretrained object detectors still suffer from feature granularity deficiency. This issue arises because CLIP, trained with image-level alignment, produces globally aligned image-text features lacking the local detail needed for effective OV-HOI detection. As shown in Figure 1 (a), the last layer focuses on high-level semantic features but neglects low-level local details like arm and face postures, compared to intermediate layers. Additionally, feature dissimilarity between shallow and deep layers increases with network depth, posing significant challenges in aligning intermediate visual features with text embeddings. Moreover, OV-HOI methods that rely on category-level or part-level text classifiers are prone to semantic confusion due to several inherent limitations. First, CLIP’s training on large-scale, long-tail datasets tends to introduce biases towards certain classes Wang et al. 2022c . For example, as shown on the left side of Figure 1 (b), CLIP often confuses “hug cat” with “hold cat.” Second, descriptions generated by LLMs based solely on labels may fail to distinguish between semantically similar classes. As illustrated on the right side of Figure 1 (b), LLMs produce similar part-level descriptions like “arm extension” for both “hold cat” and “chase cat”. To address the aforementioned challenges, we propose an end-to-end OV-HOI detection network named SGC-Net. First, we design the granularity sensing alignment (GSA) module, leveraging multi-granularity visual features from CLIP to improve the transferability for novel classes. The primary challenge in this process is effectively aggregating global semantic features with local details to refine coarse interaction representations while ensuring alignment between intermediate visual features and text embeddings. Accordingly, we introduce a block partitioning strategy that groups layers based on their relative distances. Specifically, we divide the visual encoder of CLIP into several blocks, ensuring a little variation in the features within each block. To align local details more precisely, we perform a fusion of multi-granularity features within each block, assigning distinct and trainable Gaussian weights to each layer. Once these intra-block features are fused, we proceed to aggregate the fused features across all blocks to maintain global semantic consistency. Furthermore, to preserve the pre-trained alignment between text and visual features in CLIP, we employ visual prompt tuning Li et al. 2024c ; Zhou et al. 2023 ; Ding et al. 2022 by adding learnable tokens to the visual features of each layer. Secondly, we propose the hierarchical group comparison (HGC) module to effectively differentiate semantically similar categories by recursively grouping and comparing classes via LLMs. Specifically, we utilize clustering algorithms ( e.g . , K -means Macqueen 1967 ) for grouping. Even with extensive prior knowledge of LLMs, comparing numerous categories for OV-HOI detection remains challenging due to the quadratic growth of the description matrix with the number of categories. To address this challenge, we adopt group-specific comparison strategies to improve efficiency. For smaller groups, we directly query LLMs for comparative descriptions, whereas for larger groups, we summarize the group’s characteristics using LLMs and incorporate these summaries into new queries to highlight distinctive features of each class within the group. By recursively construct a class hierarchy, we can classify HOI by descending from the top to the bottom of the hierarchy, comparing HOI and text embeddings at each level. In summary, the innovation of the proposed SGC-Net is three-fold: (1) The GSA module effectively aggregates multi-granular features while aligning the local details and global semantic features of OV-HOI; (2) The HGC module iteratively generates discriminative descriptions to refine the classification boundaries of labels; (3) The efficacy of the proposed SGC-Net is evaluated on two widely-used OV-HOI detection benchmark datasets, i.e ., HICO-DET Chao et al. 2018 and SWIG-HOI Wang et al. 2021 . Experimental results show that our SGC-Net consistently outperforms state-of-the-art methods. 2 Related Work 2.1 HOI Detection Existing HOI detection works can be divided into two categories, two-stage methods Cao et al. 2023 ; Gao et al. 2018 ; Gao et al. 2020 ; Lei et al. 2023 ; Li et al. 2019 ; Park et al. 2023 ; Tamura et al. 2021 ; Zhang et al. 2022 ; Zhang et al. 2023 and one-stage methods Chen et al. 2021 ; Gkioxari et al. 2018 ; Kim et al. 2020 ; Kim et al. 2022 ; Liao et al. 2020 ; Tu et al. 2023 ; Zhong et al. 2021 . Two-stage methods typically use an object detector to recognize humans and objects, followed by specialized