No More Sibling Rivalry: Debiasing Human-Object Interaction Detection

Bin Yang Yulin Zhang Hong-Yu Zhou Sibei Yang · 2025

Read No More Sibling Rivalry: Debiasing Human-Object Interaction Detection by Bin Yang Yulin Zhang Hong-Yu Zhou Sibei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00760.txt.

Introduction

Bin Yang, with three co-authors, finds a bias in human-object interaction detection. This task finds human-action-object triplets in images. Detection transformers have improved the task a lot. But many triplets look alike yet differ in meaning. Those rivals interfere with each other during learning. The authors call this problem Toxic Siblings bias.

When sibling classes look alike, one triplet's gain can cost its sibling. Precision falls as a result. The team proposes two debiasing objectives to fix this. One samples wrong sibling-like triplets and rebuilds them as correct ones. It uses strong positional priors as a guide. The other first learns shared features within a sibling group. It then refines the traits that set siblings apart. Tests beat both baseline and leading methods on HICO-Det.

Catalog record

CREATORBin Yang Yulin Zhang Hong-Yu Zhou Sibei Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

No More Sibling Rivalry: Debiasing Human-Object Interaction Detection Bin Yang Yulin Zhang Hong-Yu Zhou Sibei Yang Abstract Detection transformers have been applied to human-object interaction (HOI) detection, enhancing the localization and recognition of human-action-object triplets in images. Despite remarkable progress, this study identifies a critical issue—“Toxic Siblings” bias—which hinders the interaction decoder’s learning, as numerous similar yet distinct HOI triplets interfere with and even compete against each other both input side and output side to the interaction decoder. This bias arises from high confusion among sibling triplets/categories, where increased similarity paradoxically reduces precision, as one’s gain comes at the expense of its toxic sibling’s decline. To address this, we propose two novel debiasing learning objectives—“contrastive-then-calibration” and “merge-then-split”—targeting the input and output perspectives, respectively. The former samples sibling-like incorrect HOI triplets and reconstructs them into correct ones, guided by strong positional priors. The latter first learns shared features among sibling categories to distinguish them from other groups, then explicitly refines intra-group differentiation to preserve uniqueness. Experiments show that we significantly outperform both the baseline (+9.18% mAP on HICO-Det) and the state-of-the-art (+3.59% mAP) across various settings. 1 Introduction Figure 1 : Illustration of “Toxic Siblings” biases in input and output level between highly similar HOI triplets and classes. (a)-(b): Input level biases mislead HOI recognition by introducing similar HOIs within images. For example, in (a), the interaction between the marked person and the stool is misclassified from standing to sitting, influenced by other nearby sibling HOIs humans who are all seated with the stool. (c) Real confusion matrix in some HOI categories: Output level bias misclassifies rare interactions, such as “releasing a bird” and “eating a pizza”, by confusing them to sibling common interactions. Human-object interaction (HOI) detection [ 29 , 40 , 83 , 41 ] aims to detect humans and objects in an image and, importantly, to recognize their interactions as human-action-object triplets. Its capability to understand objects and relationships within scenes [ 11 , 64 , 42 , 77 , 22 ] potentially enables diverse applications in image captioning [ 34 , 66 ] , visual question answering [ 1 , 56 ] , robotics [ 14 ] , and human-computer interaction [ 51 , 25 ] . Similar to object detection, HOI detection involves multiple target triplets per image. To handle targets as direct set predictions, recent approaches [ 62 , 41 , 4 ] often employ the Detection Transformer (DETR) [ 5 , 63 , 61 , 88 ] architecture, with sequential detectors: an object detector for identifying humans and objects, followed by an interaction detector for recognizing interactions. In turn, compared to object detection [ 61 , 59 , 18 , 57 ] , action-object categories in HOI increase quadratically. Meanwhile, a multitude of distinct human-action-object triplets (HOIs) may share identical human, action, or object components. Consequently, accurately recognizing subtle variations among the vast array of similar HOIs poses a significant challenge, often leading to considerable category confusion. To enhance the differentiation of HOIs, some approaches [ 39 , 78 , 79 ] utilize external datasets for explicit data augmentation such as those for object detection [ 43 ] , scene graphs [ 32 , 8 ] , or action recognition [ 6 ] to pretrain object and interaction detectors. Others employ external models [ 28 , 78 , 70 , 53 , 15 , 41 ] , such as VLMs [ 48 , 34 ] , LLMs [ 3 , 84 ] , and diffusion models [ 55 , 23 , 26 , 27 ] , to augment data or extract interaction knowledge. Alternatively, to enhance interaction recognition and support relational reasoning, recent methods [ 74 , 2 , 37 ] integrate human-object spatial priors [ 2 ] , human body pose priors [ 37 ] , or fine-grained interaction descriptions alongside object features [ 4 , 46 ] . Despite advancements and notable performance improvements, we empirically observe that many HOIs remain difficult to distinguish due to the presence of “Toxic Siblings Bias”, which hinder existing learning mechanisms from effectively helping the detector comprehend the vast and highly similar HOI categories. In the following two paragraphs, we progressively define the “Toxic Siblings” biases and analyze their impact. Figure 2 : Illustrates the training patterns and effects of Toxic Siblings Bias in HOI Detection. (a) & (b) : The training mAP curves for sibling HOI pairs demonstrate a mutually detrimental learning dynamic (red lines) at both the input-level (a) and output-level (b). (c) presents a statistical analysis of the impact of input-level ”Toxic Siblings” bias, which significantly increases the error rate. (d) demonstrates that at the output level, the more Toxic Siblings a category has (i.e., the higher the similarity), the worse its performance. (See Appendix A.1 for more details.) What is “Toxic Siblings Bias”? (1) Input bias within single images : Some HOIs frequently co-occur within the same image, sharing the same object or interaction, as shown in Fig. 1 (a),(b) . In such cases, the image features input to the interaction decoder contain multiple similar, mutually interfering features for different HOIs, making it harder for the interaction decoder to learn differentiation. (2) Output bias across different images : Even when appearing in different images, some categories exhibit semantic similarity—for example, “watching a bird”, “feeding a bird”, “holdinging a bird”, as they all involve “releasing a bird”, as shown in Fig. 1 (c) . Due to this similarity, the interaction decoder’s output classification heads for sibling categories become highly alike, causing confusion and competition during learning. How profound is the impact of “Toxic Sibling” bias on learning? Based on the co-occurrence frequency in the same input image and the similarity of class features, we can identify ”siblings” for each HOI (details in Appendix A ). The learning curves and statistical results for different groups of siblings reveal their impact on model learning and recognition. (1) Competition hinders simultaneous accuracy improvement among sibling categories, as illustrated by the two red lines in Fig. 2 (a) and Fig. 2 (b) . Meanwhile, the severe fluctuations and inverted performance curves indicate that toxic siblings impede further learning. (2) Then, for the input-level bias, we found that when siblings are present in the input image, the probability of a sample being completely incorrect (mAP=0) increases by an average of 27.59 (see Fig. 2 (c) ). (3) Finally, for non-head classes, a 0.1 increase in the average cosine similarity of the classification head initialization (e.g., using CLIP [ 54 ] ) with other classes results in a 5.51 drop in AP, as shown in Fig. 2 (d) . This indicates that for these HOIs, the more “Toxic Siblings” they have, the more challenging it becomes for the model to learn to differentiate between them. In this paper, we aim to mitigate “Toxic Siblings” biases modifications in the learning mechanism, avoiding additional pretraining data or heavy external models such as LLMs and Diffusion models. Instead, we introduce only minor modifications to the interaction detector, while emphasizing two novel debiasing objectives—“contrastive-then-calibration” and “merge-then-split”—to address in-and-out “Toxic Siblings” biases, respectively as shown in Fig 2 . The contrastive-then-calibration (C2C) objective samples biased versions of each HOI triplet from its sibling triplets

Read the complete text