HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models
Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J. Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, Hao Tang · 2025
Read HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models by Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J. Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, Hao Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19157.txt.
Introduction
Mingzhen Huang and a large team present HOIGPT, a system that both reads and makes 3D scenes of hands grasping objects. Their twenty twenty-five paper unifies two tasks: writing a caption for a hand-object interaction, and building such a scene from text, an object, or a partial sequence.
The core is a language model trained on tokens for both words and motion. A new tokenizer, built as a split-hand-and-object autoencoder, turns motion into tokens the model can handle, with physics kept in mind. A motion-aware model then works both ways, text to scene and scene to text. Tests set new best scores on both caption and generation benchmarks.
Catalog record
| CREATOR | Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J. Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, Hao Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J. Liang, Haoyu Ma, Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, Hao Tang Abstract We introduce HOIGPT, a token-based generative method that unifies 3D hand-object interactions (HOI) perception and generation, offering the first comprehensive solution for captioning and generating high-quality 3D HOI sequences from a diverse range of conditional signals ( e.g . text, objects, partial sequences). At its core, HOIGPT utilizes a large language model to predict the bidrectional transformation between HOI sequences and natural language descriptions. Given text inputs, HOIGPT generates a sequence of hand and object meshes; given (partial) HOI sequences, HOIGPT generates text descriptions and completes the sequences. To facilitate HOI understanding with a large language model, this paper introduces two key innovations: (1) a novel physically grounded HOI tokenizer, the hand-object decomposed VQ-VAE, for discretizing HOI sequences, and (2) a motion-aware language model trained to process and generate both text and HOI tokens. Extensive experiments demonstrate that HOIGPT sets new state-of-the-art performance on both text generation (+2.01% R Precision) and HOI generation (-2.56 FID) across multiple tasks and benchmarks. 1 1 footnotetext: Work done when interning at Meta 1 Introduction Figure 1 : HOIGPT can interpret a variety of input prompts for diverse HOI-related tasks. We illustrate examples of text to HOI generation, HOI completion conditioned on object movement, and HOI captioning. HOIGPT generates or interprets hand-object interaction sequences in response to user queries, showcasing its capability to understand and produce contextually relevant HOI motions. The sequences represent time order from left to right. Hand-object interaction (HOI) refers to how humans use one or both hands to directly manipulate objects, including for tool manipulation, skilled activity, or exploration purposes. These kinds of actions are among the primary ways that humans take action in the world. Generating and understanding HOI in 3D is thus an important task pivotal to advancing machine capabilities in human-centric applications, such as human activity understanding, AR/VR, and robotics. A recent research trend has been the generation of HOI motion sequences from text Christen et al. 2024 ; Cha et al. 2024 , primarily based on diffusion models Sohl-Dickstein et al. 2015 . Such models are capable of generating HOI motion sequences with realistic appearances, but they have certain shortcomings that prevent them from being applied in broader HOI settings: 1) Limited flexibility in conditioning: Text prompts are often the only method of steering generation. This limits their ability to condition the motion as needed for motion completion, prediction, or blending; 2) Limited to short sequences: Diffusion models struggle to generate coherent, longer sequences of motion that includes multiple actions, partially due to challenges in leveraging prior knowledge and enforcing temporal consistency. Moreover, these works only focus on text-to-3D HOI generation and lack the capability to summarize actions, which restricts their applicability for other tasks. In a seemingly unrelated field, large language models (LLMs) have made stunning recent advances Devlin et al. 2019 ; Raffel et al. 2020 ; Dubey et al. 2024 ; Achiam et al. 2023 . While not perfect, LLMs can now generate long, coherent text or even converse in multi-turn dialogue. Interestingly, LLMs have also proven highly compatible with modalities beyond text, demonstrating increasing viability as universal interfaces and enabling transferr of the extensive knowledge distilled in pretrained LLMs to other modalities. The latter is particularly attractive for HOI due to the difficulty of data collection (and subsequent small scale of existing data). Other characteristics of multimodal LLMs, such as bidirectionality, descriptive abilities, flexible conditioning, and long-sequence coherence, pose a potential solution to prior text-conditioned diffusion approaches to HOI. Recognizing the potential of combining large language models (LLMs) with motion data, several recent studies have utilized LLMs to model human body motion Jiang et al. 2024 ; Zhang et al. 2024b ; Radosavovic et al. 2024 ; Ribeiro-Gomes et al. 2024 ; Wang et al. 2024 . These approaches involve tokenizing motion data (e.g., using a VQVAE Van Den Oord et al. 2017 ) and fine-tuning the LLM to interpret these motion tokens as a novel “language.” A similar methodology can be applied to LLMs and HOI, but several challenges unique to HOI emerge. Chief among these is that while human body motion effectively models a single entity, HOI involves the interaction of up to three. Modeling these interactions requires a next-level reasoning ability in physical world. Furthermore, the set of possible objects in HOI is very diverse, each with varying interaction patterns and affordances, leading to virtually limitless possibilities; this variability in objects contrasts with human bodies and hands, which are often parameterizable with specialized and compact models Romero et al. 2017 ; Loper et al. 2015 . The complex interplay between hands and objects and the objects’ high diversity results in an enormous joint space of possible movements, which naive tokenization implementations can struggle to model. In this paper, we propose HOIGPT , an autoregressive generative model designed for bidirectional transformations between 3D HOI motion and corresponding textual descriptions. HOIGPT contains several key innovations. Core to our work is a novel tokenizer specifically tailored for HOI that effectively factorizes HOI motion token space, reducing the required complexity and improving generality while still enforcing physical and semantical constraints to 3D hand-object interactions—a challenge not present in single-entity human body motion yet critical for generating plausible and coherent long HOI sequences. We also propose a pretraining approach for HOI tokens that enables the language model to capture the relationships between various HOI tokens more effectively. This enables several downstream HOI generation and understanding tasks, including HOI comprehension, generation, motion completion, and motion prediction—all using the same model with different prompts. Our contributions are summarized as follows: • We introduce the first unified model for bidirectional transformation between HOI motion and text through autoregressive language models, enabling a single model to understand, generate, predict, and complete HOI motions. • We present a HOI tokenizer, the first tokenizer designed to discretize HOI motions, addressing the lack of physical grounding in language tokens. Our approach incorporates novel geometric loss functions and a HOI-decomposed VQ-VAE architecture to enhance physical grounding. • With these innovations, HOIGPT achieves state-of-the-art performance in both HOI generation and understanding, significantly outperforming all previous methods. 2 Related Work Human motion generation with Multi-Modal Language Models . Text to human motion generation Tevet et al. 2022 ; Zhai et al. 2023 ; Guo et al. 2022a ; Azadi et al. 2023 ; Zhang et al. 2024a ; Chen et al. 2023 ; Dai et al. 2024 ; Wang et al. 2023 ; Zhong et al. 2023 ; Lou et al. 2023 has become a well-studied task in recent years. Among those methods, Diffusion Models Ho et al. 2020 has been widely used for human motion synthesis Cohan et al. 2024 ; Shafir et al. 2024 ; Tevet et al. 2023 . However, those methods only generate human motion from text descriptions but do not generate text descriptions to demonstrate an understanding of hand motions. Recently, the language model based motion generators are proposed to also achieve both generation and understanding of human motion. Zhang et al. 2023