UniHOI: Unified Human-Object Interaction Understanding via Unified Token Space

Panqi Yang, Haodong Jing, Nanning Zheng, Yongqiang Ma · 2025

Read UniHOI: Unified Human-Object Interaction Understanding via Unified Token Space by Panqi Yang, Haodong Jing, Nanning Zheng, Yongqiang Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15046.txt.

Introduction

Panqi Yang and three co-authors present UniHOI, a unified model for understanding human object interactions in images. Finding asks where an interaction occurs, while generation asks to create it. The two dual tasks were long handled separately, limiting shared learning. UniHOI models both in one unified token space.

A symmetric interaction aware attention module links image and interaction semantics in both directions. A semi supervised scheme supports learning with limited labels. Experiments show top performance on both finding and generation. Long tail finding accuracy improves notably. Open vocabulary generation metrics rise sharply as well. Sharing one token space helps the model generalize to rare and unseen interactions.

Catalog record

CREATORPanqi Yang, Haodong Jing, Nanning Zheng, Yongqiang Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniHOI: Unified Human-Object Interaction Understanding via Unified Token Space Panqi Yang, Haodong Jing, Nanning Zheng, Yongqiang Ma Abstract In the field of human-object interaction (HOI), detection and generation are two dual tasks that have traditionally been addressed separately, hindering the development of comprehensive interaction understanding. To address this, we propose UniHOI, which jointly models HOI detection and generation via a unified token space, thereby effectively promoting knowledge sharing and enhancing generalization. Specifically, we introduce a symmetric interaction-aware attention module and a unified semi-supervised learning paradigm, enabling effective bidirectional mapping between images and interaction semantics even under limited annotations. Extensive experiments demonstrate that UniHOI achieves state-of-the-art performance in both HOI detection and generation. Specifically, UniHOI improves accuracy by 4.9% on long-tailed HOI detection and boosts interaction metrics by 42.0% on open-vocabulary generation tasks. 1 State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center of Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiao Tong University {yangpq,jinghd}@stu.xjtu.edu.cn, {nnzheng,musayq}@xjtu.edu.cn Figure 1: UniHOI is the first to achieve unified modeling for the two inverse tasks of HOI detection and generation. Through a unified token space, our method enables generalizable interaction semantics understanding and cross-task knowledge sharing. UniHOI achieves state-of-the-art results on most metrics for both HOI detection and generation. Here, HICO-D refers to the Rare metric of the Default split in the HICO-DET ( Chao et al. 2017 ) , other abbreviations follow similarly. Introduction The Human-Object Interaction (HOI) understanding encompasses both HOI detection (identifying \langle\text{human},\text{action},\text{object}\rangle triplets in images) and HOI generation (synthesizing images conditioned on specified interactions). These two tasks are essentially inverse to each other, sharing highly related underlying semantic representations and reasoning processes. However, existing approaches largely treat detection and generation as isolated problems: most HOI generation methods ( Li et al. 2023b ; Hoe et al. 2024 ; Gao et al. 2020 ) rely on explicit spatial constraints ( e.g. , bounding boxes), limiting their ability to generalize to novel or complex interactions. Meanwhile, state-of-the-art HOI detection models ( Wang et al. 2024 ; Luo et al. 2024 ; Hui et al. 2025 ) focus solely on recognition and remain decoupled from generation, which hinders knowledge sharing and demands extensive fine-grained annotations for each task. This significantly limits the scalability of HOI understanding in open-world scenarios. This motivates a key question: Can we unify HOI detection and generation within a single framework to fully exploit their shared semantic representations? We contend that such unification would promote richer interaction understanding, improve data and knowledge efficiency, and enable cross-task and open-world generalization, thus paving the way for more holistic HOI reasoning. The recent emergence of Multimodal Large Language Models (MLLMs) offers a promising approach for unifying perception and generation within a single, expressive framework. Inspired by works like MMaDA ( Yang et al. 2025 ) and particularly Liquid ( Wu et al. 2024b ) , which demonstrated the potential of unified token spaces for joint vision-language modeling, we argue that a unified representation for interaction semantics holds the key to bridging the HOI detection-generation gap. To this end, our approach is grounded in two key insights: First, Unified Token Space : the semantic essence of a HOI triplet \langle\text{human},\text{action},\text{object}\rangle can be effectively represented within the same discrete vocabulary as visual tokens derived from images ( e.g. , VQGAN ( Esser 2020 ) ). This enables bidirectional mapping within a MLLM, where a textual prompt like “person feeding cat” can evoke spatial relationships (“hand-bowl-cat”) during generation, while visual tokens can be decoded back into semantic triplets during detection. Second, Dual Complementarity : HOI detection and generation are inherently complementary; detection provides explicit interaction priors ( e.g. , typical spatial distributions and fine-grained annotations) to guide generation, while generation encourages the model to acquire richer, more compositional interaction representations. This reciprocal relationship improves both the expressiveness and generalization of the learned HOI representations, allowing the model to effectively acquire unified HOI semantic representations. Therefore, we propose UniHOI , the first unified semi-supervised MLLM for HOI detection and generation. UniHOI operates on a massively expanded vocabulary merging visual codebook tokens and text tokens, accepting either images (for detection) or text prompts (including structured HOI triplets for generation) to produce corresponding outputs. Specifically, our approach incorporates: (1) an Interaction-Aware Attention (IAA) module that injects HOI triplet embeddings into cross-attention layers to focus on interaction-relevant regions; (2) a semi-supervised learning framework grounded in cycle consistency within the discrete token space, which enables effective joint training of HOI detection and generation even under limited or heterogeneous supervision. By integrating strong supervision (HICO-DET ( Chao et al. 2017 ) ), weak supervision (image-text pairs from LAION-SG ( Li et al. 2024 ) ), and unlabeled images, this framework mitigates reliance on exhaustive HOI annotations and facilitates the learning of more generalizable interaction representations for both tasks. And our contributions are as follows: • We demonstrate that interaction-aware semantic representations can be jointly encoded and reasoned about in a unified discrete token space. Our modality-aware unified token space enables bidirectional mapping and compositional reasoning between HOI detection and image generation within a single model, surpassing traditional embedding-level alignment approaches. • We propose an Interaction-Aware Attention (IAA) module with parameter-shared, symmetric cross-attention that explicitly encodes structured HOI semantics as relational priors for both detection and generation. This unified mechanism enables interpretable, context-aware cross-modal reasoning. • We present a unified semi-supervised learning strategy based on cycle consistency in the shared token space, allowing effective training with mixed supervision and reducing annotation cost for open-world HOI recognition and generation. • Extensive experiments demonstrate that UniHOI achieves highly competitive performance on standard HOI detection and generation benchmarks. Specifically, UniHOI improves accuracy by 4.9% on long-tailed HOI detection and boosts interaction metrics by 42% on open-vocabulary generation tasks. Related Work Text-to-Image Models Diffusion-based T2I models ( Nichol et al. 2021 ; Saharia et al. 2022 ; Rombach et al. 2022 ; Ramesh et al. 2022 ) iteratively denoise latent text-conditioned embeddings ( Radford et al. 2021 ; Raffel et al. 2020 ) , generating high-resolution images ( Rombach et al. 2022 ; Podell et al. 2024 ; Saharia et al. 2022 ) . Recent studies ( Kim et al. 2023 ; Mo et al. 2024 ; Mokady et al. 2022 ; Park et al. 2025 ) reveal that their intermediate representations encode rich semantics, supporting advanced image editing via feature-text interactions. Meanwhile, MLLM-based T2I models ( Wu et al. 2024b ; Yang et al. 2025 ; Chen et al. 2025 ; Xie et al. 2024 ) leverage large language models for enhanced multimodal understanding and unified vision-language reasoning and generation. For instance, MMaDA ( Yang et al. 2025 ) adopts a unified diffusion framework for joint inference and generation, while Liquid ( Wu et al. 2024b ) shows mutual benefits between understanding and generation within MLLM architectures. Based on these works, we explore how interaction-aware cross attention in MLLMs perceives and generates interaction semantics. Figure 2: An overview of the UniHOI pipeline. The bottom-right shows the details of the IAA module, illustrating the bidirectional transformation between text tokens and visual tokens. HOI Understanding HOI understanding encompasses two core tasks: HOI detection and HOI image generation. HOI detection (

Read the complete text