Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation Xinshun Wang, Peiming Li, Ziyi Wang, Zhongbin Fang, Zhichao Deng, Songtao Wu, Jason Li, Mengyuan Liu Abstract † † footnotetext: * Equal contribution. Corresponding author is Mengyuan Liu. Human motion analysis tasks, such as temporal 3D pose estimation, motion prediction, and motion in-betweening, play an essential role in computer vision. However, current paradigms suffer from severe fragmentation. First, the field is split between “perception” models that understand motion from video but only output text, and “generation” models that cannot perceive from raw visual input. Second, generative MLLMs are often limited to single-frame, static poses using dense, parametric SMPL models, failing to handle temporal motion. Third, existing motion vocabularies are built from skeleton data alone, severing the link to the visual domain. To address these challenges, we introduce Superman, a unified framework that bridges visual perception with temporal, skeleton-based motion generation. Our solution is twofold. First, to overcome the modality disconnect, we propose a Vision-Guided Motion Tokenizer. Leveraging the natural geometric alignment between 3D skeletons and visual data, this module pioneers robust joint learning from both modalities, creating a unified, cross-modal motion vocabulary. Second, grounded in this motion language, a single, unified MLLM architecture is trained to handle all tasks. This module flexibly processes diverse, temporal inputs, unifying 3D skeleton pose estimation from video (perception) with skeleton-based motion prediction and in-betweening (generation). Extensive experiments on standard benchmarks, including Human3.6M, demonstrate that our unified method achieves state-of-the-art or competitive performance across all motion tasks. This showcases a more efficient and scalable path for generative motion analysis using skeletons. Figure 1 : A Unified Framework for Human Motion Perception and Generation. Our unified model achieves state-of-the-art performance across three traditionally disparate tasks. Our method consistently outperforms existing methods across all tasks simultaneously: (1) 3D Pose Estimation , (2) Motion Prediction , and (3) Motion In-betweening . 1 Introduction Table 1 : Comparison with existing traditional and LLM/MLLM-based models for perceiving and/or generating human poses Perception Generation Tokenizer Repr. vision motion ∗ motion ∗ visual guidance Traditional Multi-Task Models MotionBERT 39 \times ✓ \times N/A skeleton SiC 30 \times ✓ \times N/A skeleton HiC 18 \times ✓ \times N/A skeleton LLM / MLLM-based Models MotionLLM 4 video ✓ \times \times SMPL MotionGPT 14 \times ✓ ✓ \times SMPL MotionGPT3 38 \times ✓ ✓ \times SMPL LocLLM 28 image \times \times \times \times PoseLLaVA 9 image \times \times \times SMPL UniPose 16 image \times \times \times SMPL Superman video&image ✓ ✓ ✓ skeleton * “Motion” refers to a temporal sequence of poses. Human motion analysis is a cornerstone of computer vision and robotics, with critical applications in human-computer interaction, augmented reality, and autonomous systems 34 ; 1 ; 19 ; 35 . Several tasks such as 3D pose estimation 34 ; 19 ; 35 , motion prediction 6 ; 31 ; 29 , and in-betweening 37 ; 15 ; 12 ; 11 ; 8 form the core of this field. Traditionally, these tasks have been tackled by highly specialized models, each optimized for a single objective. While this focused approach has yielded significant success, it has led to a fragmented and inflexible ecosystem. The recent paradigm shift driven by Multi-modal Large Language Models (MLLMs) offers a compelling path toward unification. However, a fundamental disconnect persists, cleaving the field into two distinct camps. On one side, Multi-modal Large Language Models (MLLMs) like MotionLLM 4 and LLaVA-Pose 33 excel at understanding motion. They can reason about poses from raw video and then generate textual descriptions, but lack the capacity to generate new, plausible human poses. On the other hand, specialized generative models like MotionGPT 14 excel at generating motion from text but struggle to process or perceive raw visual inputs. This “read-only” vs. “write-only” dichotomy forces researchers to use disparate architectures for tasks that are inherently interconnected. Moreover, constraints on input modality are also a limitation. Many generative methods operate only on abstract text or single images ( e.g., ChatPose 10 , PoseLLaVA 9 , UniPose 16 ), ignoring the rich temporal dynamics of video. Conversely, video-processing models such as SKI Models 22 are typically confined to understanding, not perception, i.e., generating textual results instead of human poses. This fragmentation not only hinders research efficiency but also prevents models from leveraging the synergistic relationship between perception and generation. This leaves a critical gap: the absence of a single, unified model that can fluidly process multi-modal inputs , from full videos for pose estimation to skeleton sequences for prediction, and generate coherent, structured 3D motion as output. To unify perception with generation, we conceptualize motion as a universal language. Our core idea is to train a single MLLM to serve as a unified motion processor. The key to achieving this is a cross-modal motion vocabulary. Existing methods, however, build this vocabulary only from skeletal data, severing the link to the visual domain. Figure 2 : Architecture of our Vision-Guided Motion Tokenizer (VGMT). VGMT creates a discrete motion vocabulary by jointly fusing information from two modalities. A Skeleton Encoder ( E_{s} ) captures geometry while a Visual-Skeleton Attention (VSA) module and a subsequent Visual Encoder ( E_{v} ) ground the pose in visual features. The fused representation is quantized against a learnable hybrid codebook, and a decoder reconstructs the 3D poses. To address this, we introduce Superman , a unified, multi-task, and multi-modal generative framework that reformulates these disparate challenges as a conditional sequence generation problem. Our approach has two key advantages: 1) it unifies perception and generation in a single model, promoting knowledge sharing between tasks; 2) it is flexible with multi-modal inputs, processing video, sequential skeletons, and text within the same architecture. Our cornerstone is the Vision-Guided Motion Tokenizer, a VQ-VAE 26 architecture building a universal, discrete pose vocabulary. Unlike prior works limited to skeleton data, our tokenizer introduces a hybrid codebook for a richer, cross-modal representation. Each token comprises paired visual and geometric prototypes. This structure guides quantization using both visual appearance from frames and 3D skeleton geometry, creating a powerful representation that intrinsically links visual evidence to motion semantics. Grounded in this motion language, our MLLM flexibly processes diverse inputs within a single architecture. It can: (1) Estimate 3D Pose by translating a video into a sequence of pose tokens. (2) Predict Motion by auto-regressively completing a pose sequence. (3) Perform In-betweening by generating intermediate tokens between keyframes. As shown in Fig. 1 , our unified design is validated by extensive experiments on standard benchmarks including Human3.6M 13 and 3PDW 27 . Superman achieves state-of-the-art performance across all tasks, even when compared with traditional multitask models specializing in perception tasks. For instance, for 3D pose estimation on Human3.6M 13 , Superman achieves an 11.97% improvement compared with the state-of-the-art multi-task perception method 18 , and a 10.91% improvement compared with the state-of-the-art LLM/MLLM-based method 9 . Additionally, Superman delivers strong generalization performance. Trained exclusively on Human3.6M, Superman generalizes well to unseen data, namely 3DPW 27 , and outperforms existing methods. Our contributions are threefold: • We propose a unified generative framework that leverages a single MLLM for multi-task, multi-modal human motion analysis, bridging the gap between motion perception and generation. • We introduce a novel Vision-Guided Motion Tokenizer that creates a robust, cross-modal pose vocabulary. By designing a hybrid codebook where each token is a bimodal entity comprising paired visual and geometric prototypes, we ensure that the quantization process is simultaneously guided by both appearance features from video and the geometric structure of 3D skeletons. • We demonstrate that our unified method achieves state-of-the-art across diverse tasks, validating the efficacy and scalability of our approach. 2 Related Work 3D Human Pose Estimation. 3D human pose estimation has evolved from classical methods to deep learning approaches using CNNs 25 and Transformers 36 to model human body structure. Modern methods typically follow a two-stage pipeline of lifting 2D keypoints to 3D 20 or perform end-to-end regression from images 24 , while capturing spatio-temporal relationships to ensure temporal coherence 21 . However, a common thread among these methods 20 ; 24 ; 21 ; 36 is their specialization on the single task of pose estimation. In contrast, our work integrates pose estimation into a unified generative framework, allowing the model to perform both perception and generation tasks, rather than being confined to a single objective. Human Motion as a Language. A groundbreaking shift in motion analysis involves representing continuous human motion as a sequence of discrete tokens, a paradigm pioneered by early works in gesture synthesis 14 . A common technique for this is using a Vector Quantized Variational Autoencoder (VQ-VAE) to learn a “codebook” of atomic poses. As summarized in Tab. 1 , recent models have framed tasks like motion generation and prediction as language modeling problems 14 ; 35 . However, their scope is often limited to generating from non-visual modalities, failing to ground the motion language in raw visual input. Our work builds directly upon this paradigm but makes a crucial extension by integrating vision into the tokenization process, creating a visually grounded motion language. Multi-modal Large Language Models for Pose Perception and Generation. The remarkable success of Multi-modal Large Language Models (MLLMs), which integrate powerful vision encoders with LLMs for complex visual reasoning 17 ; 32 ; 7 , has inspired their application to human-centric tasks. In the context of human motion, models such as MotionLLM 4 and LLaVA-Pose 33 have demonstrated the ability to comprehend and answer questions about human actions and poses depicted in images and videos. They effectively connect the visual and semantic aspects of human motion. However, a significant limitation of these models is their primary focus on perception. They can interpret and describe poses, but they are not architecturally designed to generate new, structured motion sequences. Recent works, such as ChatPose 10 and UniPose 16 , have begun to bridge this gap by enabling pose generation conditioned on both visual and textual inputs. These models, however, typically operate on static, single-frame images. Our work distinguishes itself by being a fully versatile framework that not only perceives motion from videos but is also a powerful generative model capable of multiple synthesis tasks, thereby unifying the reading and writing of motion within a single, coherent architecture. Figure 3 : Network architecture and training paradigm. Superman fine-tune a single LLM to integrate information from text, video, and 3D skeleton modalities. Optionally, a Motion-Aware Fine-Tuning (MAFT) module can be integrated. With < 0.2% extra parameters, MAFT enhances motion perception by enabling cross-video-motion fusion, leading to substantial improvement on tasks with visual input, as validated by the experiment results. 3 Method Our method unifies multi-task human motion analysis within a conditional sequence generation framework. It involves two key stages: (1) a novel Vision-Guided Motion Tokenizer (Section 3.1 ) that converts continuous, high-dimensional motion into a discrete sequence of semantic tokens, and (2) a Multi-modal Large Language Model (MLLM) (Section 3.2 ) that autoregressively models these token sequences to perform our diverse set of tasks. The overall network architecture is depicted in Fig. 3 . 3.1 Vision-Guided Motion Tokenizer The cornerstone of our approach is a discrete vocabulary for continuous human motion. Unlike prior tokenizers that rely on skeletons alone 14 , we design a Vision-Guided Motion Tokenizer, built upon a VQ-VAE, which is explicitly trained to fuse visual appearance and 3D skeletal geometry . This ensures each token is grounded in both modalities. The architecture is illustrated in Fig. 2 . Table 2 : Comparison of our model with traditional multi-task models and LLM / MLLM-based models on three human motion tasks: pose estimation (PE), motion prediction (MP), and motion in-betweening (MIB) on Human3.6M 13 . All tasks are evaluated using Mean Per Joint Position Error (MPJPE) in millimeters, averaged over all test data, where lower is better. “ T ” means how many frames the model inputs and outputs. (“ T =1” means the video has to be processed image-by-image.) “N/A” means the model is not applicable to the task. For example, single-frame-only models 28 ; 16 ; 9 cannot do MP or MIB; LLM-based models 14 ; 38 cannot do PE. Models Venue T PE ‡ MP † MIB † N-MPJPE MPJPE Avg 80ms 160ms 320ms Avg mid last Traditional Multi-Task Models MotionBERT 39 ICCV’23 16 47.07 56.70 29.94 18.67 26.82 50.33 42.37 44.86 53.16 Skeleton-in-Context 30 CVPR’24 16 45.26 55.57 33.42 21.09 30.48 54.62 31.27 36.66 36.84 Human-in-Context 18 Arxiv’25 16 44.77 53.86 26.66 14.36 23.58 47.34 31.13 37.01 35.49 Superman (w/ MAFT) Ours’26 16 39.41 51.61 26.13 † 13.70 23.30 44.90 30.61 † 35.99 35.13 LLM / MLLM-based Models LocLLM 28 CVPR’24 1 49.32 62.19 N/A N/A UniPose 16 CVPR’25 1 65.24 106.96 N/A N/A PoseLLaVA 9 AAAI’25 1 50.40 62.43 N/A N/A MotionGPT 14 NeurIPS’23 16 N/A 48.81 42.99 49.27 55.31 52.96 56.07 66.45 MotionGPT3 38 Arxiv’25 16 N/A 42.30 37.25 42.70 47.94 47.66 50.47 59.81 Superman Ours’26 16 44.90 61.39 26.13 † 13.70 23.30 44.90 30.61 † 35.99 35.13 \ddagger Traditional models adopt “RGB \to 2D \to 3D” paradigm, leveraging a fixed 2D pose estimator; LLM/MLLM-based models adopt “RGB \to 3D” paradigm. \dagger For MP and MIB, existing methods take only 3D poses as input (no vision). For fair comparison, our MAFT module (which enhances vision) is not active, hence the identical results for Superman and Superman (w/ MAFT) on these tasks. Dual-Stream Encoding. Given a video clip \mathcal{V}\in\mathbb{R}^{F\times H\times W\times 3} and its corresponding 3D pose sequence \mathbf{X}\in\mathbb{R}^{F\times N\times 3} , our tokenizer employs two parallel streams. 1) Visual Encoding Stream: This stream extracts pose-centric visual features. We first use a vision backbone (e.g., HRNet 23 ) to obtain feature maps \mathcal{F}_{f} for each frame. For each joint j at frame f , its projected 2D location \mathbf{p}_{j,f} serves as a reference point to sample an initial query feature \mathbf{q}_{j,f} from \mathcal{F}_{f} . To enhance feature representation and robustness to occlusions, we introduce a Visual-Skeleton Attention (VSA) module. The VSA adaptively aggregates features by predicting sampling offsets and aggregation weights from the query. The visual feature for the joint is computed as: \mathbf{v}_{j,f}=\text{VSA}(\mathbf{q}_{j,f},\mathcal{F}_{f},\mathbf{p}_{j,f}), (1) yielding a sequence of contextually-aware visual features \mathbf{V}\in\mathbb{R}^{F\times N\times D} . 2) Skeletal Encoding Stream: This stream captures the motion’s intrinsic geometry. The input pose sequence \mathbf{X} is passed through a series of lightweight 2D convolutions operating on the joint-temporal grid. This effectively models spatio-temporal kinematics and maps the coordinates into a latent space, producing skeletal features \mathbf{S}\in\mathbb{R}^{F\times N\times D} . Hybrid Codebook Quantization. To ensure each token represents a short motion pattern rather than a static pose, we first temporally downsample the feature sequences \mathbf{V} and \mathbf{S} , yielding window-level representations \mathbf{z}^{v}_{w} and \mathbf{z}^{s}_{w} for each time window w . We then introduce a hybrid codebook that pairs visual and geometric prototypes: \mathcal{C}=\{(\mathbf{c}^{v}_{k},\mathbf{c}^{s}_{k})\}_{k=1}^{K} , where \mathbf{c}^{v}_{k},\mathbf{c}^{s}_{k}\in\mathbb{R}^{D} . For each window w , k_{w} is the token index that minimizes the joint distance: k_{w}=\underset{k}{\arg\min}\left(\|\mathbf{z}^{v}_{w}-\mathbf{c}^{v}_{k}\|_{2}^{2}+\|\mathbf{z}^{s}_{w}-\mathbf{c}^{s}_{k}\|_{2}^{2}\right). (2) This hybrid selection, based on minimizing the joint Euclidean distance, grounds the discretization process in both vision and geometry. The selected code vectors for the window are then \hat{\mathbf{c}}^{v}_{w}=\mathbf{c}^{v}_{k_{w}} and \hat{\mathbf{c}}^{s}_{w}=\mathbf{c}^{s}_{k_{w}} . Reconstruction and Training. The skeletal code \hat{\mathbf{c}}^{s}_{w} is upsampled and passed to a decoder to reconstruct the 3D poses \hat{\mathbf{X}}_{w} . The tokenizer is trained end-to-end with a VQ objective that combines reconstruction with modality-wise commitment losses: \mathcal{L}_{\text{VQ}}=\|\mathbf{X}_{w}-\hat{\mathbf{X}}_{w}\|_{2}^{2}+\beta_{s}\|\text{sg}[\mathbf{z}^{s}_{w}]-\hat{\mathbf{c}}^{s}_{w}\|_{2}^{2}+\beta_{v}\|\text{sg}[\mathbf{z}^{v}_{w}]-\hat{\mathbf{c}}^{v}_{w}\|_{2}^{2}, (3) where \text{sg}[\cdot] is the stop-gradient operator, and we set the commitment weights \beta_{s}=0.5 and \beta_{v}=0.5 . This process effectively converts a continuous motion clip of F frames into a discrete sequence of T integer tokens \mathbf{K}_{1:T} . After training, the tokenizer provides a robust, visually-grounded motion vocabulary. 3.2 Unified Multi-Task Modeling with LLM Our discrete motion tokenizer enables us to frame diverse motion tasks as a unified conditional sequence generation problem. We employ a decoder-only Multi-modal Large Language Model (MLLM), Qwen2.5-VL-7B 2 , as our central sequence processor, which learns to predict motion tokens autoregressively. Connecting Tokenizer to Qwen-VL. Besides the standard tokenizer-MLLM integration paradigm 14 ; 16 , we additionally design an optional light-weight module to inject skeletal geometry into the MLLM’s visual stream, Motion-Aware Fine-tuning (MAFT). First, we enhance the ViT features using a Visual-Skeleton Attention (VSA) block. The ViT backbone extracts patch features \mathbf{Z}_{\text{grid}} . Concurrently, 2D joint projections serve as reference points for a multi-scale deformable sampler 40 to aggregate pose-centric features \mathbf{Z}_{\text{pose}} . We then fuse this information using VSA, implemented as a cross-attention layer followed by an FFN, where grid tokens are queries and pose tokens are keys/values. The visual tokens \hat{\mathbf{Z}}_{\text{grid}} are computed as: \hat{\mathbf{Z}}_{\text{grid}}=\text{VSA}(\mathbf{Z}_{\text{grid}},\mathbf{Z}_{\text{pose}}). (4) These enhanced visual tokens, along with any textual prompts, are fed into the LLM decoder, which generates the final sequence of motion tokens. Table 3 : Action-specific N-MPJPEs for pose estimation on Human3.6M 13 . Models Venue T Actions Sit SitDown Smoke Photo Wait Walk WalkDog WalkTwo Traditional Multi-Task Models MotionBERT 39 ICCV’23 16 62.22 69.81 49.21 49.66 39.49 40.80 44.94 40.95 Skeleton-in-Context 30 CVPR’24 16 59.34 65.89 48.70 48.11 37.28 38.52 42.12 38.97 Human-in-Context 18 Arxiv’25 16 59.95 65.21 47.81 47.02 36.37 38.17 41.76 37.31 Superman (w/ MAFT) Ours’26 16 47.12 48.97 41.31 34.37 41.21 36.12 39.23 43.23 LLM / MLLM-based Models LocLLM 28 CVPR’24 1 64.48 68.76 50.96 55.16 42.55 39.11 49.96 43.10 UniPose 16 CVPR’25 1 72.30 84.24 69.20 62.65 60.02 66.97 65.71 70.69 PoseLLaVA 9 AAAI’25 1 64.64 71.36 52.67 54.21 44.67 46.90 48.90 44.90 Superman Ours’26 16 62.68 71.93 49.09 43.54 35.21 30.89 45.91 39.60 Models Venue T Actions Direct Discuss Eat Greet Phone Pose Purchase Traditional Multi-Task Models MotionBERT 39 ICCV’23 16 39.85 46.10 41.77 43.35 46.57 42.04 43.72 Skeleton-in-Context 30 CVPR’24 16 38.03 44.23 41.15 41.97 45.26 40.09 41.95 Human-in-Context 18 Arxiv’25 16 37.51 44.30 40.30 41.33 44.74 39.68 42.02 Superman (w/MAFT) Ours’26 16 42.72 36.50 33.65 35.06 40.77 34.21 36.16 LLM / MLLM-based Models LocLLM 28 CVPR’24 1 42.12 49.09 41.70 44.22 51.81 45.04 43.70 UniPose 16 CVPR’25 1 52.57 61.62 59.55 60.33 71.57 54.79 63.35 PoseLLaVA 9 AAAI’25 1 43.00 47.80 45.64 47.58 49.21 44.49 47.16 Superman Ours’26 16 34.74 43.09 45.01 36.03 48.98 34.28 44.90 Our framework handles multiple tasks by varying the conditioning inputs: 3D Pose Estimation from Video: Given a video clip \mathbf{I}_{1:F} , the MLLM is conditioned on its enhanced visual features \hat{\mathbf{Z}}_{\text{grid}} to generate the corresponding motion token sequence \mathbf{K}_{1:T} . The objective is to maximize the log-likelihood of the ground-truth sequence: \mathcal{L}_{\text{est}}=\sum_{t=1}^{T}\log P(k_{t}|\mathbf{K}_{