Locality-Aware Zero-Shot Human-Object Interaction Detection

Sanghyun Kim Deunsol Jung Minsu Cho · 2025

Read Locality-Aware Zero-Shot Human-Object Interaction Detection by Sanghyun Kim Deunsol Jung Minsu Cho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19503.txt.

Introduction

Sanghyun Kim, Deunsol Jung, and Minsu Cho detect unseen human-object interactions in a 2025 paper. Vision-language models deal well with new categories. Their features blur the fine detail interactions require, though. Knowing a person touches an object differs from naming both.

Their framework sharpens the big model's output in two ways. Pooling data from nearby patches brings back local detail and layout. Spatial priors from nearby patches guide that pooling. Patterns of how a person and object act on each other are caught in full. The two fuse into the pair's features. Benchmarks across several zero-shot settings followed. The method beat past approaches throughout. Local detail cues and interaction cues each proved needed. Reach to unseen classes is the headline. Surveillance, robotics, and image search all need this skill. The work lifts the base features rather than swapping them out.

Catalog record

CREATORSanghyun Kim Deunsol Jung Minsu Cho
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19503

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Locality-Aware Zero-Shot Human-Object Interaction Detection Sanghyun Kim Deunsol Jung Minsu Cho Abstract Recent methods for zero-shot Human-Object Interaction (HOI) detection typically leverage the generalization ability of large Vision-Language Model (VLM), i.e., CLIP, on unseen categories, showing impressive results on various zero-shot settings. However, existing methods struggle to adapt CLIP representations for human-object pairs, as CLIP tends to overlook fine-grained information necessary for distinguishing interactions. To address this issue, we devise, LAIN, a novel zero-shot HOI detection framework designed to enhance the locality and interaction awareness of CLIP representations. The locality awareness, which involves capturing fine-grained details and the spatial structure of individual objects, is achieved by aggregating the information and spatial priors of adjacent neighborhood patches. The interaction awareness, which involves identifying whether and how a human is interacting with an object, is achieved by capturing the interaction pattern between the human and the object. By infusing locality and interaction awareness into CLIP representations, LAIN captures detailed information about the human-object pairs. Our extensive experiments on existing benchmarks show that LAIN outperforms previous methods in various zero-shot settings, demonstrating the importance of locality and interaction awareness for effective zero-shot HOI detection. 1 Introduction The task of Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize the interactions between them in a given image, i.e., identifying a set of HOI instances (human, object, interaction). HOI detection is useful for a wide range of computer vision applications, including image retrieval Wu et al. 2022a ; Yoon et al. 2021 ; Gordo and Larlus 2017 and image captioning Wu et al. 2022b ; Yao et al. 2018 ; Herdade et al. 2019 , where a comprehensive understanding of human-object relationships is essential. Although significant advances have been made recently, conventional HOI methods Li et al. 2024 ; Lei et al. 2023 ; Kim et al. 2023 have primarily been relying on fully supervised learning, limited to identifying predefined HOI categories. Given that humans interact with objects in a compositional way, it is costly and impractical to collect annotations for all possible HOI categories, limiting their ability to identify novel HOI categories not present in the training set. Figure 1 : (a)-(b): Since CLIP primarily encodes global information, it struggles to capture the fine-grained details required to accurately identify interactions within human-object pairs. (c): When existing methods adapt CLIP representations to zero-shot HOI detection, this limitation hinders CLIP’s generalization, and results in degraded performance which is even lower than CLIP’s original zero-shot performance in UC-RF and UV settings. Recently, strong generalization ability of CLIP on unseen categories, which stems from contrastive image-level pre-training on large-scale data, has inspired the development of a zero-shot HOI model that leverages this capability to recognize unseen HOI categories. While existing methods Ning et al. 2023 ; Wu et al. 2023a ; Mao et al. 2024 for zero-shot HOI detection have achieved strong performance by leveraging CLIP representations, the domain gap between the image-level pre-training task and the region-level task poses the challenges in adapting CLIP representations to zero-shot HOI detection. Since CLIP predominantly encodes global information Zhong et al. 2022 ; Zhou et al. 2023 , it often fails to extract fine-grained information about individual objects. This hinders the HOI model from capturing whether and how the person interacts with the object. As shown in Figure 1 (a), CLIP assigns a high confidence score to the interaction ‘ride-bike’ even though the human-object pair (1,2) does not focus on the bike region, indicating that CLIP has limited capacity to capture fine-grained object details. As a result, the human-object pair (1,4) has a similar confidence score for the unseen HOI category ‘hold-frisbee’ compared to others, i.e., (2,3) and (3,4) , struggling to distinguish the interactive pair in Figure 1 (b). These issues weaken the generalization ability of CLIP when existing methods attempt to adapt its representations for zero-shot HOI detection, resulting in lower zero-shot performance compared to CLIP’s original results as shown in Figure 1 (c). To address the challenges mentioned above, we introduce a novel zero-shot HOI detection framework, dubbed L ocality- A ware I nteraction N etwork (LAIN), that learns locality-aware interaction via Locality Adapter (LA) and Interaction Adapter (IA). The LA extracts locality-aware features from image patch tokens by considering visual context of neighboring regions and spatial priors, and then infuses them back into the image patch tokens. The IA leverages the locality-aware patch tokens to update human-object tokens by performing interaction reasoning between human and object regions, resulting in interaction-aware human-object tokens. In this manner, the LA provides fine-grained details and spatial structure for individual objects, enabling the IA to perform effective contextual reasoning. The IA complements CLIP representations by incorporating fine-grained details, providing a relational context that cannot be captured by locality awareness alone. By incorporating locality and interaction awareness, which play complementary roles, each layer of LAIN effectively captures detailed information for the human-object pair, facilitating the adaptation of CLIP representations to zero-shot HOI detection. To demonstrate the effectiveness of our proposed method, we conducted extensive evaluations on two public benchmarks, HICO-DET Chao et al. 2018 and V-COCO Gupta and Malik 2015 . The experimental results show that LAIN outperforms the previous methods for zero-shot detection across all zero-shot settings, demonstrating the robust generalization ability of our approach in zero-shot scenarios. Our ablation studies demonstrate the importance of locality and interaction awareness for zero-shot HOI detection. Our contribution can be summarized as follows: • We propose the Locality-Aware Interaction Network (LAIN), which incorporates a Locality Adapter (LA) and an Interaction Adapter (IA). • By enriching CLIP representations with locality and interaction awareness, LAIN effectively captures the fine-grained details about human-object pairs. • Extensive experiments demonstrate that LAIN achieves outstanding zero-shot performance, achieving a new state-of-the-art. 2 Related work 2.1 Human-Object Interaction (HOI) Detection Conventional HOI detection methods can be roughly divided into two categories: two-stage and one-stage methods. Two-stage methods Zhang et al. 2022a ; Zhang et al. 2022b ; Gao et al. 2020 ; Gao et al. 2018 ; Li et al. 2020 ; Qi et al. 2018 ; Ulutan et al. 2020 ; Wang et al. 2020 ; Zhang et al. 2021b ; Hou et al. 2020 first detect humans and objects using a pre-trained detector Ren et al. 2015 ; Carion et al. 2020 . After constructing all possible human-object pairs based on the detection results, these pairs are fed into an interaction classifier. To generate discriminative features for classifying the interaction of a human-object pair, they incorporate additional information Li et al. 2019 ; Liu et al. 2020 ; Gupta et al. 2019 and perform relational reasoning on a graph structure Wang et al. 2020 ; Ulutan et al. 2020 ; Zhang et al. 2021b . In contrast to two-stage methods that follow a sequential cascade to determine the interaction between the human-object pairs, one-stage methods Fang et al. 2021 ; Liao et al. 2020 ; Kim et al. 2020 concurrently detect individual instances, pair the human-object instances, and classify interactions. Inspired by the transformer-based detector, i.e . , DETR Carion et al. 2020 , where each query learns to detect an object, recent one-stage methods Tamura et al. 2021 ; Zhou et al. 2022 ; Kim et

Read the complete text