DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary

Jiazhi Guan ∗, Quanwei Yang ∗, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang †, Hang Zhou †, Jingdong Wang · 2026

Read DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary by Jiazhi Guan ∗, Quanwei Yang ∗, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang †, Hang Zhou †, Jingdong Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09883.txt.

Introduction

Jiazhi Guan and coauthors generate videos of people handling objects under light, direct control. Past methods need dense signals, template clips or fussy prompts, which limits use with new objects. Their DISPLAY system guides motion with just wrist positions and a plain box around the object. An object stressed attention step plus multi task training on curated data keep the results sharp and physical.

Catalog record

CREATORJiazhi Guan ∗, Quanwei Yang ∗, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang †, Hang Zhou †, Jingdong Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48038
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09883

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary Jiazhi Guan ∗, Quanwei Yang ∗, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang †, Hang Zhou †, Jingdong Wang Abstract Human-centric video generation has advanced rapidly, yet existing methods struggle to produce controllable and physically consistent Human-Object Interaction (HOI) videos. Existing works rely on dense control signals, template videos, or carefully crafted text prompts, which limit flexibility and generalization to novel objects. We introduce a framework, namely DISPLAY, guided by Sparse Motion Guidance , composed only of wrist joint coordinates and a shape-agnostic object bounding box. This lightweight guidance alleviates the imbalance between human and object representations and enables intuitive user control. To enhance fidelity under such sparse conditions, we propose an Object-Stressed Attention mechanism that improves object robustness. To address the scarcity of high-quality HOI data, we further develop a Multi-Task Auxiliary Training strategy with a dedicated data curation pipeline, allowing the model to benefit from both reliable HOI samples and auxiliary tasks. Comprehensive experiments show that our method achieves high-fidelity, controllable HOI generation across diverse tasks. The project page can be found at https://mumuwei.github.io/DISPLAY/ . * * footnotetext: Equal Contribution. † † footnotetext: Corresponding Authors. Figure 1 : Human-Object Interactions Synthesis . Our method generates user-intended human-object interactions from zero-shot references and user-specified sparse motion guidance. Blue: a generated video performing human-object interactions with novel visual and object references. Gray: a generated video demonstrating environmental interactions conditioned solely on the visual reference. 1 Introduction Human-centric video generation is becoming pervasive in our daily lives, empowering a diverse array of service scenarios such as news media, e-commerce, education, and multimedia entertainment. This expansion has been paralleled by a substantial surge in research focusing on the synthesis of human faces [ 51 , 56 , 16 , 14 , 61 ] and human bodies [ 15 , 23 , 67 , 13 , 19 , 39 ] in the past few years. However, the majority of existing human video generation approaches are limited to simple actions like lip-synchronization or basic gestures. To significantly enhance user experience, there is a growing need to embed more interactive functionalities in digital humans, particularly the ability to create videos with controllable human-object interaction. Recently, large video generation models (LVGM) [ 2 , 69 , 55 , 29 ] have allowed users to generate highly creative content conditional on specific text prompts. Despite their powerful capabilities, these large models still exhibit a fundamental over-reliance on meticulously crafted text prompts, and the generation process is often non-deterministic. While several variants [ 27 , 77 , 73 , 32 ] have explored integrating multiple concepts, their outputs frequently suffer from physical inconsistencies in interactions and generally lack precise control. Moreover, achieving fine-grained and spatially precise control—such as directing a subject to grasp a specific object at an exact location—is exceptionally challenging when relying solely on textual inputs. This inherent deficiency in spatio-temporal controllability prevents users from accurately dictating complex Human-Object Interaction (HOI) tasks, which severely restricts the practical deployment and application scope of these models. On the other hand, several studies [ 8 , 64 , 24 , 65 , 27 ] have taken initial steps toward achieving more controllable HOI synthesis through explicit guidance. These methods typically operate within a video-to-video inpainting framework [ 8 , 65 ] or a pose-guided animation setting [ 64 , 24 ] , which require complex and high-dimensional control signals. Nevertheless, these approaches face two critical limitations. First, a key imbalance exists in the representation of the interacting components: while these methods frequently employ strong control signals for the human hand regions (e.g., 2D human pose keypoints, 3D hand meshes), the interacting objects typically lack similarly explicit structural representations. This representation asymmetry causes the generation models to frequently overfit to the dominant control signals, leading to failure modes such as geometric interpenetration and object deformation, particularly on novel or unseen items. Second, the necessity of a template video or HOI representations (e.g., human poses [ 24 , 8 ] and object depth maps [ 64 ] ) extracted from a driving source imposes significant constraints on these approaches. This dependency restricts editing capabilities to mere modifications of existing footage, thus preventing the freedom to arbitrarily generate content according to higher-level user demands. In this paper, we propose a D irectable human-object I nteraction video generation framework via SP arse motion guidance and mu L ti-task A uxiliar Y , named DISPLAY , which tackles the aforementioned problems. Our key insight is to introduce Sparse Motion Guidance that enables more robust and high-fidelity HOI generation with elevated degrees of freedom. Concretely, the Sparse Motion Guidance is composed of the coordinates of hand wrist joints and a shape-agnostic bounding box of the target object, which offers two significant advantages: 1) The sparse guidance effectively alleviates the imbalance between hand and object representations, as well as the discrepancy between training and inference. On one hand, neither the wrist points nor the shape-agnostic boxes dominate the training, avoiding over-dense priors on hand gestures or object boundaries. On the other hand, this form of guidance ensures consistent representations during inference, even when novel objects with significantly different shapes are introduced. 2) In contrast to methods [ 24 , 64 , 8 ] that depend on template or driving videos, our sparse guidance requires only minimal user input, such as simple clicking on a canvas at a few key frames. This lightweight interaction enables intuitive generation without relying on external video sources or complex representations, thereby offering substantially greater flexibility. However, directly adopting such a sparse condition, while beneficial for control freedom, introduces two interlinked challenges: 1) The inherent sparsity of the guidance inevitably raises the difficulty of learning fine-grained generation, particularly within the complex, localized HOI regions. 2) HOI data are frequently compromised by issues such as occlusion. Consequently, the amount of clean and usable HOI data remains limited, constraining the model’s generalization capability. To address the first issue, we propose the Object-Stressed Attention mechanism, which applies weighted coefficients to emphasize object reference tokens and their interactions during HOI modeling, thereby enhancing the quality of HOI generation. In terms of the data quantity issue, we propose a novel Multi-Task Auxiliary Training strategy together with a carefully designed data curation pipeline. By leveraging this mixed training paradigm, the model learns more effectively from a broader and more diverse data corpus, which in turn enhances all inference modalities, including object replacement, insertion, and environmental interaction. Our contributions can be summarized as follows: 1) We propose a novel framework anchored by Sparse Motion Guidance that enables arbitrary, high-fidelity, and robust Human-Object Interaction (HOI) generation, supporting effortless user interaction. 2) We introduce the Object-Stressed Attention mechanism to enhance object synthesis robustness under sparse guidance, which guarantees the generated objects are physically consistent with the surrounding scene and human pose. 3) We propose a comprehensive Multi-Task Auxiliary Training strategy, paired with a corresponding data curation pipeline, specifically to overcome the bottleneck imposed by the scarcity of high-quality HOI data. 2 Related Works Human Video Animation . Human video animation generation has become a widely studied research topic. Early GAN-based works [ 21

Read the complete text