Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning

Mengyuan Liu, Xinshun Wang, Zhongbin Fang, Deheng Ye, Xia Li, Tao Tang, Songtao Wu, Xiangtai Li, Ming-Hsuan Yang · 2025

Read Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning by Mengyuan Liu, Xinshun Wang, Zhongbin Fang, Deheng Ye, Xia Li, Tao Tang, Songtao Wu, Xiangtai Li, Ming-Hsuan Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10897.txt.

Introduction

Mengyuan Liu and co-authors present a 2025 computer vision paper on one model for all kinds of 3D human motion data. Pose and mesh data come from many tasks and datasets, and earlier unified models needed special parts per domain plus staged training. The authors propose training a single model in one pass, taught by examples in context.

Their earlier pose-only system generalized across tasks but stumbled over mixed data types. The new system, Human-in-Context, unites pose and mesh inputs, widens task coverage, and trains on larger data. A max-min sampling rule picks instructive context examples, and a two-branch design injects that context more carefully. Across domains it beat its predecessor on flexibility and scores.

Catalog record

CREATORMengyuan Liu, Xinshun Wang, Zhongbin Fang, Deheng Ye, Xia Li, Tao Tang, Songtao Wu, Xiangtai Li, Ming-Hsuan Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning Mengyuan Liu, Xinshun Wang, Zhongbin Fang, Deheng Ye, Xia Li, Tao Tang, Songtao Wu, Xiangtai Li, Ming-Hsuan Yang Abstract This paper aims to model 3D human motion across domains, where a single model is expected to handle multiple modalities, tasks, and datasets. Existing cross-domain models often rely on domain-specific components and multi-stage training, which limits their practicality and scalability. To overcome these challenges, we propose a new setting to train a unified cross-domain model through a single process, eliminating the need for domain-specific components and multi-stage training. We first introduce Pose-in-Context (PiC), which leverages in-context learning to create a pose-centric cross-domain model. While PiC generalizes across multiple pose-based tasks and datasets, it encounters difficulties with modality diversity, prompting strategy, and contextual dependency handling. We thus propose Human-in-Context (HiC), an extension of PiC that broadens generalization across modalities, tasks, and datasets. HiC combines pose and mesh representations within a unified framework, expands task coverage, and incorporates larger-scale datasets. Additionally, HiC introduces a max-min similarity prompt sampling strategy to enhance generalization across diverse domains and a network architecture with dual-branch context injection for improved handling of contextual dependencies. Extensive experimental results show that HiC performs better than PiC in terms of generalization, data scale, and performance across a wide range of domains. These results demonstrate the potential of HiC for building a unified cross-domain 3D human motion model with improved flexibility and scalability. The source codes and models are available at https://github.com/BradleyWang0416/Human-in-Context . I Introduction As a core topic in computer vision, cross-domain 3D human motion modeling aims to develop a model capable of handling multiple domains, including different tasks, modalities, and datasets. To achieve this, poses [ 1 , 2 ] and meshes [ 3 ] are two widely adopted human motion representations, as they provide greater efficiency, compactness, and richer information compared to RGB-based representations [ 4 , 5 ] . Leveraging pose and mesh representations, cross-domain 3D human motion modeling has found diverse applications, such as motion prediction for autonomous driving [ 6 , 7 ] , pose estimation for human-robot collaboration [ 8 , 9 , 10 ] , and mesh recovery for virtual reality [ 11 ] . Cross-domain models have been extensively explored in computer vision, robotics, and natural language processing, leveraging diverse backbones [ 12 , 13 , 14 ] and training paradigms [ 15 , 16 ] . However, applying cross-domain modeling to 3D human motion presents even greater challenges due to the multi-dimensional and spatiotemporal complexity of 3D motion data [ 17 ] . As a result, existing cross-domain models [ 18 , 19 , 20 , 21 , 22 ] suffer from two major limitations. First, their applicability is often restricted to a narrow scope, such as performing the same task across a few datasets [ 23 , 24 ] or handling similar tasks within single-modal data [ 19 , 25 ] . Second, they largely rely on additional domain-specific model heads [ 18 , 20 ] and require complex multi-stage training [ 21 , 22 ] , limiting their generalizability and scalability. To overcome these limitations, we introduce a new setting that enables training a unified cross-domain model in a single process, eliminating the need for domain-specific components and complex multi-stage training. Motivated by in-context learning [ 16 , 26 ] in natural language processing, which allows models to perform multiple tasks without explicit fine-tuning or retraining, we find that this paradigm aligns well with our proposed setting. While in-context learning has been extended to image-based tasks [ 27 , 28 ] and point cloud-based tasks [ 29 ] , its application to 3D human motion modeling remains unexplored, to the best of our knowledge. Fig. 1 : Motivation (left) and performance overview (right) for our proposed Pose-in-Context and Human-in-Context. Compared to previous methods, which require multiple models/heads along with multiple training stages, PiC and HiC unify various human-centric tasks and datasets through one-time training with a fully unified model. Furthermore, HiC extends PiC into a more inclusive, scalable, and generalizable cross-domain model by introducing designs in both functionality and technicality. We present PiC, the first approach that incorporates in-context learning into 3D human motion modeling to facilitate a pose-centric cross-domain model. PiC achieves generalization across various pose-based tasks and datasets, showing competitive performance. However, PiC has three key limitations: 1) PiC focuses on the scope of pose-based tasks and datasets without the ability to generalize across different modalities; 2) PiC uses a random selection-based prompting strategy, selecting prompts independently of queries, which could result in a large gap between the query and the selected prompt; and 3) PiC employs an attention-based network architecture, which does not exploit contextual dependencies other than global context. To overcome the limitations of PiC, we develop Human-in-Context, a cross-domain model in 3D human motion modeling that achieves cross-modality, cross-task, and cross-dataset generalization within a single unified framework. Unlike PiC, which addresses only pose-based tasks, HiC can handle both pose- and mesh-based tasks through one-time training. HiC extends PiC in three aspects (see Figure 1 ). First, HiC extends the model’s generalization capability to cover a broader range of domains, including both pose and mesh modalities. By representing pose- and mesh-based human motion within a unified formulation, HiC supports cross-modal learning within a single in-context framework. Regarding task coverage, for each pose-based task in PiC, HiC includes a corresponding mesh-based variant, effectively doubling the number of supported tasks. Second, HiC employs a max-min similarity prompt sampling strategy to enhance generalization. Instead of random selection, it samples representative anchors from the training set, retrieving the closest-matching anchor for each query to ensure contextual alignment. This approach dynamically pairs hard anchors (fixed representatives) with soft anchors (adaptable refinements), improving generalization across diverse domains (both in-distribution and out-of-distribution). Third, HiC introduces a new network, X-Fusion Net, with a dual-branch architecture, enabling context integration across prompts and queries. Each branch consists of a sequence of X-Fusion blocks to handle contextual information at multiple levels. Each block performs two operations: multi-level context aggregation and cross-level context update. In the aggregation step, features are processed using state-space models, self-attention, and graph convolution to capture dependencies across different views, scopes, and feature spaces. In the update step, the network adjusts feature representations across levels by estimating their relative importance for the target task. These operations support context-aware representation learning across diverse motion domains. Extensive results show that PiC and HiC perform favorably against state-of-the-art domain-specific and cross-domain approaches. Figure 1 shows the overall framework where PiC 1 1 1 Pose-in-Context is referred to as “Skeleton-in-Context” in our prior work. For terminological consistency in the cross-domain context of this journal version, we use “pose” instead of “skeleton” throughout this paper. is first proposed in our CVPR 2024 conference paper [ 30 ] . This work, HiC, extends PiC in several aspects: • HiC is an in-context cross-domain model in 3D human motion modeling that simultaneously possesses cross-modality, cross-task, and cross-dataset generalization capability. • HiC incorporates a larger scope of domains, including two modalities, ten tasks, and four datasets, while expanding the scale of data by \sim 21\times . • HiC proposes a prompting strategy,

Read the complete text