HiPART: Hierarchical Pose AutoRegressive Transformer for Occluded 3D Human Pose Estimation Hongwei Zheng, Han Li, Wenrui Dai, Ziyang Zheng, Chenglin Li, Junni Zou, Hongkai Xiong Abstract Existing 2D-to-3D human pose estimation (HPE) methods struggle with the occlusion issue by enriching information like temporal and visual cues in the lifting stage. In this paper, we argue that these methods ignore the limitation of the sparse skeleton 2D input representation, which fundamentally restricts the 2D-to-3D lifting and worsens the occlusion issue. To address these, we propose a novel two-stage generative densification method, named Hierarchical Pose AutoRegressive Transformer (HiPART), to generate hierarchical 2D dense poses from the original sparse 2D pose. Specifically, we first develop a multi-scale skeleton tokenization module to quantize the highly dense 2D pose into hierarchical tokens and propose a Skeleton-aware Alignment to strengthen token connections. We then develop a Hierarchical AutoRegressive Modeling scheme for hierarchical 2D pose generation. With generated hierarchical poses as inputs for 2D-to-3D lifting, the proposed method shows strong robustness in occluded scenarios and achieves state-of-the-art performance on the single-frame-based 3D HPE. Moreover, it outperforms numerous multi-frame methods while reducing parameter and computational complexity and can also complement them to further enhance performance and robustness. 1 Introduction 3D human pose estimation (HPE) from monocular images has attracted wide attention due to its flexibility in adaptation on various devices. It is usually decoupled into the 2D estimation and 3D lifting stages in existing methods [ 32 , 60 , 63 ] , where the 2D estimation stage estimates 2D poses from input images via off-the-shelf 2D detectors [ 33 , 5 , 40 ] , and subsequently, the 3D lifting stage achieves 2D-to-3D estimation with a lifting model. Such decoupling enables large-scale training with a large amount of 3D MOCAP data [ 32 ] by circumventing the scarce labeled 2D/3D ground truth poses for monocular images. However, the real-world application of 3D HPE faces a challenge of the occlusion problem ( i.e. , self-occlusion and object occlusion), where the unreliable occluded 2D poses could lead to corrupted 3D lifted poses. Figure 1 : Top: Comparison of temporal-based and visual-based methods with our densification approach at the framework level. These methods enrich information in the lifting stage, while ours address a more fundamental issue, i.e. , the sparse 2D pose input. Bottom: Comparison of parameters, GFLOPs, and MPJPE across various methods on Human3.6M. The circle size indicates GFLOPs for inference. Our method achieves the SOTA result with reduced complexity compared to the temporal-based methods. Recent works have introduced additional information in the lifting stage to alleviate the occlusion issues. Some methods leverage topology priors ( e.g. , kinematic priors [ 37 , 24 , 48 ] and geometric constraints [ 2 , 53 ] ) to improve the structural understanding of joint correlations, but often prove ineffective in severe occlusion. Others utilize enriched context ( e.g. , temporal [ 35 , 62 , 28 ] and visual cues [ 42 , 60 , 30 , 64 ] ) to improve the occluded joints inference, which require extra complex temporal or visual encoders with increased parameter and computational complexity (see top of Fig. 1 ). In essence, all these methods primarily focus on enriching the information in the lifting stage while overlooking the inherent limitation of the current input 2D representation, the sparse 2D skeleton representation , which fundamentally constrains the performance of 2D-to-3D lifting and worsens the occlusions. Specifically, current 2D pose datasets typically represent the human body with few keypoints ( e.g. , 17 joints for Human3.6M [ 20 ] ). This sparse input for lifting inherently limits the ability to exploit local context. In this paper, we argue that developing a hierarchical dense 2D skeleton representation is essential for exploring the intricate skeletal context, thus enhancing the robustness of occluded 2D-to-3D lifting. We conduct a toy experiment to demonstrate the benefit of this hierarchical representation. We collect the dataset of hierarchical dense 2D poses by progressively coarsening the 3D ground truth mesh of Human3.6M (6890 vertices \rightarrow 96 joints \rightarrow 48 joints) and projecting them into the 2D pixel space. When integrating these hierarchical dense 2D poses into the 2D-to-3D lifting stage, MPJPE is significantly improved by 55% from 37.6mm to 17.5mm, as shown in Appendix E ). This indicates the significance of richer skeletal information. For example, as shown in the bottom of Fig. 2 , when the right wrist is occluded, the sparse 2D pose only provides one elbow joint for reference, while the denser 2D poses offer multiple arm joints to help predict the occlusion region. However, acquiring such hierarchical dense 2D poses remains a challenge, as the 3D ground truth mesh is not available in real-world 3D HPE scenarios. Inspired by recent autoregressive models [ 45 , 38 , 11 , 43 , 27 ] , in this paper, we propose a novel hierarchical 2D pose densification method named Hierarchical Pose AutoRegressive Transformer (HiPART) to address the challenge. HiPART obtains reliable hierarchical dense 2D poses conditioned on the original sparse 2D pose input in a generative fashion with two main modules. First, we design a Multi-Scale Skeletal Tokenization (MSST) module to progressively quantize a highly dense 2D pose into hierarchical discrete tokens via VQ-VAE [ 45 ] , and introduce a Skeleton-aware Alignment strategy to strengthen connections across multi-scale tokens. Second, we develop a Hierarchical AutoRegressive Modeling (HiARM) scheme for hierarchical dense 2D pose generation. Unlike the conventional next-token prediction widely used in image and video generation [ 3 , 11 , 52 ] , we present novel sparse-to-dense and center-to-periphery strategies specifically designed for the non-Euclidean skeletal structure and show their efficiency. Therefore, HiPART generates hierarchical 2D poses to provide enhanced local context for subsequent 2D-to-3D lifting. We perform extensive experiments to validate the effectiveness and efficiency of HiPART. Specifically, we show that, using only a vanilla spatial transformer for lifting, HiPART achieves state-of-the-art performance in the single-frame setting on various 3D HPE benchmarks, including Human3.6M [ 20 ] , 3DPW [ 46 ] , and 3DPW-Occ [ 58 ] , with strong robustness in occluded scenarios. Compared to methods that rely on complex temporal encoders to incorporate temporal cues, HiPART is superior or comparable with significantly reduced complexity, as shown in the bottom of Fig. 1 . Note that HiPART is orthogonal to existing temporal methods and can integrate these methods to further enhance performance and robustness. The contributions of this paper are summarized below. • To our best knowledge, we achieve the first hierarchical dense 2D pose generation scheme to address the occlusion problem for 3D HPE from single monocular image. • We develop a multi-scale skeleton quantization model via VQ-VAE along with a Skeleton-aware Alignment to enhance token connections. • We design a novel hierarchical pose autoregression strategy specifically for non-Euclidean skeleton structures. • We employ the proposed 2D pose generation scheme in 2D-to-3D lifting to achieve state-of-the-art performance on the single-frame setting and outperforms numerous multi-frame methods. Figure 2 : Visualization of reconstructed 3D poses and hierarchical 2D poses on Human3.6M under occlusions. Ours outperforms DiffPose [ 15 ] due to the rich skeletal context in hierarchical poses. Figure 3 : The overview of the two-stage generative densification method, Hierarchical Pose AutoRegressive Transformer (HiPART). In Stage 1, the MSST module progressively quantizes the fine 2D pose into hierarchical tokens with Skeleton-aware Alignment. We omit the reconstruction of the dense 2D pose \hat{\mathbf{x}}_{d} for simplicity. In Stage 2, we propose a HiARM scheme to generate (i+1) -th pair tokens based on a series of indices less than i+1 . One pair of discrete tokens contains a single sparse token and r dense tokens corresponding to the related part. Finally, generated hierarchical 2D poses are fed to a vanilla spatial transformer for subsequent 2D-to-3D lifting. 2 Related Work Occlusions in 3D HPE. Existing approaches typically incorporate additional information in the lifting stage. Topology priors ( e.g. , kinematic priors [ 37 , 24 , 48 ] and geometric constraints [ 2 , 53 ] ) are utilized to enhance joint correlations, often failing in severely occluded scenarios. Other approaches introduce enriched context( e.g. , temporal context [ 35 , 62 , 28 , 41 ] and visual cues [ 30 , 61 , 64 ] ) to improve the prediction accuracy for occluded joints. While highly effective, these methods consist of advanced temporal or image encoders, significantly increasing model and computational complexity. We alleviate the occlusion problem in an effective and lightweight manner by incorporating hierarchical dense 2D poses as input. It addresses the sparse 2D pose input issue, which is overlooked by existing works. Hierarchy in 3D HPE. Hierarchical information is essential in Human Mesh Recovery (HMR) [ 34 , 23 , 22 , 57 ] , as the human body can be represented at multi-level, from sparse joints to dense vertices, which contains rich skeletal context and can help address occlusions in 3D HPE. For instance, Pose2Mesh [ 7 ] uses a learnable transformation matrix to estimate poses in a coarse-to-fine manner. HGN [ 25 ] employs a hierarchical network to reconstruct poses, transitioning from sparse to dense representations. However, both methods regard the hierarchical 3D pose prediction as an auxiliary task, neglecting the accuracy of the hierarchical 2D poses, and leading to minimal performance gains. We propose a generative densification approach that enhances 2D hierarchical pose estimation and 3D lifting precision. Generative Models for 3D HPE. Generative models have shown promising results in various domains, including image, text, and audio generation [ 16 , 17 , 4 , 19 , 51 ] , and have recently been applied in 3D HPE. For example, diffusion models progressively refine pose distributions, reducing uncertainty during estimation [ 8 , 15 , 18 ] . However, they suffer from low inference speed due to reliance on numerous sampling steps. A vector-quantized-based method, i.e. , PCT [ 14 ] , uses a two-stage process to construct a codebook capturing sub-structures and casts pose estimation as a classification task. While effective, the classification approach struggles to fully model the latent distribution of discrete tokens. We address this in an auto-regression manner in Stage 2 and introduce Skeleton-aware Alignment to strengthen token connections. Besides, our method has a faster generation speed than diffusion models due to the posed-based auto-regression. More discussion is in the Appendix D . 3 Collection of Hierarchical 2D Poses The goal of our method is to generate reliable hierarchical 2D poses with denser joints conditioned on the original sparse 2D pose \mathbf{x}_{s}\in\mathbb{R}^{J_{s}\times 2} with J_{s} joints, benefiting subsequent 2D-to-3D lifting. However, existing datasets lack hierarchical dense 2D poses that could serve as ground truth for training. To address this, we use the 3D human mesh from Human3.6M [ 20 ] and apply the dense vertices coarsening method [ 7 ] along with camera projection to create the dense and fine ( i.e. , highly dense) 2D poses, denoted as \mathbf{x}_{d}\in\mathbb{R}^{J_{d}\times 2} and \mathbf{x}_{f}\in\mathbb{R}^{J_{f}\times 2} , where J_{f}>J_{d}>J_{s} . The detailed process is described in the Appendix B . 4 Hierarchical Pose AutoRegressive Transformer (HiPART) The overview of our HiPART is shown in Fig. 3 . We first learn the Multi-Scale Skeletal Tokenization (MSST) module to quantize the fine \mathbf{x}_{f}\in\mathbb{R}^{J_{f}\times 2} into hierarchical discrete tokens, i.e., sparse tokens \mathbf{q}_{s}\in\mathbb{Z}^{J_{s}} and dense token \mathbf{q}_{d}\in\mathbb{Z}^{J_{d}} , where different colors in Fig. 3 indicate different parts of the human body, and different saturations represent the number ( r ) of dense tokens. Then, we learn a pose-aware generative transformer by the Hierarchical AutoregRessive Modeling (HiARM) scheme to generate hierarchical tokens conditioned on the sparse 2D pose \mathbf{x}_{s} . With the decoders of HiPART, we can reconstruct the hierarchical dense 2D poses from the hierarchical tokens \mathbf{q}_{s} and \mathbf{q}_{d} . 4.1 Multi-Scale Skeletal Tokenization (MSST) Multi-Scale Skeletal Quantized-Autoencoder. We use an architecture similar to VQ-VAE-2 [ 38 ] to construct our multi-scale skeletal quantized autoencoder. Encoders and decoders are implemented with MLP-Mixer [ 44 ] , which is well-suited for skeletal data. Specifically, two encoders \mathcal{E}_{f}(\cdot) and \mathcal{E}_{d}(\cdot) sequentially project the ground truth \mathbf{x}_{f} into dense and sparse pose embeddings, \mathbf{z}_{d}\in\mathbb{R}^{J_{d}\times D} and \mathbf{z}_{s}\in\mathbb{R}^{J_{s}\times D} , where D is the embedding dimension. Next, \mathbf{z}_{s} is quantized into sparse tokens \mathbf{q}_{s}\in\mathbb{Z}^{J_{s}} via a quantizer \mathcal{Q}(\cdot) using the sparse codebook \mathbf{C}_{s}\in\mathbb{R}^{K_{s}\times D} , which contains K_{s} learnable vectors. Each element of \mathbf{q}_{s} then looks up \mathbf{C}_{s} to obtain \mathbf{\hat{z}}_{s} , i.e. , the approximation of \mathbf{z}_{s} : \mathbf{q}_{s}=\mathcal{Q}(\mathbf{z}_{s};\mathbf{C}_{s}),\quad\mathbf{\hat{z}}_{s}=\mathbf{C}_{s}[\mathbf{q}_{s}]. (1) Subsequently, \mathbf{\hat{z}}_{s} is upsampled into \mathbf{\hat{z}}^{\prime}_{s}\in\mathbb{R}^{J_{d}\times D} using decoder \mathcal{D}_{s}(\cdot) . This upsampling embedding is then concatenated with \mathbf{z}_{d} to form \mathbf{z}^{\prime}_{d} , which is quantized into dense tokens \mathbf{q}_{d}\in\mathbb{Z}^{J_{d}} using the dense codebook \mathbf{C}_{d}\in\mathbb{R}^{K_{d}\times D} , which contains K_{d} vectors. The approximated dense pose embeddings \mathbf{\hat{z}}_{d} is obtained by looking up \mathbf{C}_{d} from \mathbf{q}_{d} . \mathbf{q}_{d}=\mathcal{Q}(\mathbf{z}^{\prime}_{d};\mathbf{C}_{d}),\quad\mathbf{\hat{z}}_{d}=\mathbf{C}_{d}[\mathbf{q}_{d}], (2) Finally, the reconstructed dense pose \hat{\mathbf{x}}_{d} is decoded using decoder \mathcal{D}_{d}(\cdot) given \mathbf{\hat{z}}_{s} , and the reconstructed fine pose \hat{\mathbf{x}}_{f} is decoded using decoder \mathcal{D}_{f}(\cdot) given \mathbf{\hat{z}}_{s} and \mathbf{\hat{z}}_{d} . Skeleton-aware Alignment. To enhance the tokenization process for skeletal data, we propose a Skeleton-aware Alignment strategy. This strategy incorporates skeletal structure into the tokenization process, improving the mutual connections across different skeleton levels. Part-wise Local Alignment (LA): The part-wise Local Alignment is introduced to align the sparse and dense tokens within the same local body part, forming a more unified representation. The Info Noise Contrastive Estimation (Info-NCE) loss is used to match part-wise pairs: \mathcal{L}_{local}=\sum_{i=1}^{J_{s}}-\log\left(\frac{\exp(\hat{\mathbf{z}}_{s}^{i}\cdot\operatorname{avg}(\hat{\mathbf{z}}_{d}^{i})/\tau)}{\sum_{k=1}^{J_{s}}\exp(\hat{\mathbf{z}}_{s}^{i}\cdot\operatorname{avg}(\hat{\mathbf{z}}_{d}^{k})/\tau)}\right), (3) where \tau is the temperature parameter, \operatorname{avg}(\cdot) is the average function. The r approximated dense pose embeddings \hat{\mathbf{z}}_{d}^{i}=\{\hat{\mathbf{z}}_{d}^{(i,j)}\}_{j=1,2,\dots,r} correspond to the related part of single \hat{\mathbf{z}}_{s}^{i} , where the ratio r=J_{d}/J_{s} . Action-wise Global Alignment (GA): We further introduce an action-wise Global Alignment to align the hierarchical tokens with the action label of the pose (contained in the Human3.6M [ 20 ] dataset), thus offering consistent semantic information for the discrete spaces. We define the global loss \mathcal{L}_{global} as a cross-entropy loss between the predicted classification vector \mathbf{p}_{A} and the action label \mathbf{y}_{A} : \displaystyle\mathbf{p}_{A} \displaystyle=\mathcal{P}_{A}(\operatorname{concat}(\hat{\mathbf{z}}_{s},\hat{\mathbf{z}}_{d})), (4) \displaystyle\mathcal{L}_{global} \displaystyle=\operatorname{CrossEntropy}(\mathbf{y}_{A},\mathbf{p}_{A}), (5) where \mathcal{P}_{A}(\cdot) is the action projector composed of two MLP-Mixer blocks, \operatorname{concat}(\cdot) is the concatenate operation. Loss Function. The Stage 1 loss \mathcal{L}_{1} includes the reconstruction loss for the poses and hierarchical pose embeddings, as well as the loss of Skeleton-aware Alignment. \displaystyle\mathcal{L}_{1}= \displaystyle\|\mathbf{x}_{f}-\hat{\mathbf{x}}_{f}\|_{2}^{2}+\|\mathbf{x}_{d}-\hat{\mathbf{x}}_{d}\|_{2}^{2}+\sum_{j\in\{s,d\}}\Big(\|\operatorname{sg}(\mathbf{z}_{j})\!-\!\hat{\mathbf{z}}_{j}\|_{2}^{2} \displaystyle+\beta\|\mathbf{z}_{j}\!-\!\operatorname{sg}(\hat{\mathbf{z}}_{j})\|_{2}^{2}\Big)\!+\!\lambda_{l}\cdot\mathcal{L}_{local}\!+\!\lambda_{g}\cdot\mathcal{L}_{global}, (6) where \beta controls the commitment loss, \lambda_{l} and \lambda_{g} are weighting factors for the local and Global Alignment losses, sg() is the stop gradient operation. Codebooks are updated via Exponential Moving Average. 4.2 Hierarchical AutoRegressive Modeling (HiARM) In Stage 2, we propose Hierarchical AutoRegressive Modeling (HiARM) for hierarchical dense 2D pose generation. We first define a hierarchical pose auto-regressive objective function and then present the model structure details. Objective Function. We generate the discrete tokens in an auto-regression manner based on the learned codebooks in MSST. Our objective function aims at maximizing the log-likelihood of the i -th pair discrete tokens based on a series of indices less than i : \mathbb{P}_{\theta}(\mathbf{q}_{d},\mathbf{q}_{s})=\prod_{i=1}^{J_{s}}\mathbb{P}_{\theta}(\mathbf{q}_{d}^{i},q_{s}^{i}|\mathbf{q}_{d}^{