SCJD: Sparse Correlation and Joint Distillation for Efficient 3D Human Pose Estimation
Weihong Chen, Xuemiao Xu, Haoxin Yang, Yi Xie, Peng Xiao, Cheng Xu, Huaidong Zhang, Pheng-Ann Heng · 2025
Read SCJD: Sparse Correlation and Joint Distillation for Efficient 3D Human Pose Estimation by Weihong Chen, Xuemiao Xu, Haoxin Yang, Yi Xie, Peng Xiao, Cheng Xu, Huaidong Zhang, Pheng-Ann Heng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14097.txt.
Introduction
Weihong Chen and co-authors present SCJD, a 2025 method for finding the 3D pose of a person from video. Fast pose tools often lose accuracy, while exact ones run too slowly for live use. Teaching a small model from a big one helps, but past teaching methods ignore how joints link in space and time.
SCJD trims the frames fed to the small model while keeping the links between frames. It then teaches the small model in three ways: richer joint traits, focus on joints next to each other, and matching motion patterns over time. Tests show the method reaches top scores while staying light. The work suits uses like sports tracking and live camera apps that need speed and care at once.
Catalog record
| CREATOR | Weihong Chen, Xuemiao Xu, Haoxin Yang, Yi Xie, Peng Xiao, Cheng Xu, Huaidong Zhang, Pheng-Ann Heng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14097
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SCJD: Sparse Correlation and Joint Distillation for Efficient 3D Human Pose Estimation Thanks: Accepted by ICME 2025 (Oral Paper). Thanks: The work is supported by China National Key R&D Program (Grant No. 2023YFE0202700, 2024YFB4709200), Guangdong Provincial Natural Science Foundation for Outstanding Youth Team Project (No. 2024B1515040010), NSFC Key Project (No. U23A20391), Key-Area Research and Development Program of Guangzhou City (No. 2023B01J0022). Thanks: † Corresponding authors: Xuemiao Xu & Haoxin Yang (email: [email protected], [email protected]) Weihong Chen, Xuemiao Xu, Haoxin Yang, Yi Xie, Peng Xiao, Cheng Xu, Huaidong Zhang, Pheng-Ann Heng Abstract Existing 3D Human Pose Estimation (HPE) methods achieve high accuracy but suffer from computational overhead and slow inference, while knowledge distillation methods fail to address spatial relationships between joints and temporal correlations in multi-frame inputs. In this paper, we propose Sparse Correlation and Joint Distillation (SCJD), a novel framework that balances efficiency and accuracy for 3D HPE. SCJD introduces Sparse Correlation Input Sequence Downsampling to reduce redundancy in student network inputs while preserving inter-frame correlations. For effective knowledge transfer, we propose Dynamic Joint Spatial Attention Distillation , which includes Dynamic Joint Embedding Distillation to enhance the student’s feature representation using the teacher’s multi-frame context feature, and Adjacent Joint Attention Distillation to improve the student network’s focus on adjacent joint relationships for better spatial understanding. Additionally, Temporal Consistency Distillation aligns the temporal correlations between teacher and student networks through upsampling and global supervision. Extensive experiments demonstrate that SCJD achieves state-of-the-art performance. Code is available at https://github.com/wileychan/SCJD . I Introduction 3D Human Pose Estimation (HPE) detects 3D joint coordinates from monocular video and is widely used in downstream tasks like action recognition [ 1 , 2 ] and motion analysis [ 3 , 4 ] . To achieve high accuracy, prior methods typically use multi-frame input sequences to capture temporal dependencies [ 5 , 6 , 7 , 8 , 9 , 10 , 11 ] and large networks (e.g., Vision Transformers [ 12 ] ) for feature extraction. However, these approaches are computationally expensive and suffer from slow inference speeds due to dense inputs and significant network overhead. To balance 3D HPE computational efficiency and accuracy, recent studies have adopted strided convolutions [ 13 ] or frequency-domain features [ 14 , 15 ] to avoid processing long input sequences. However, these approaches require deeper and wider networks to sustain accuracy, still resulting in significant computational demands. Knowledge distillation (KD) [ 16 ] has also been explored to address this challenge [ 17 , 18 , 19 ] . These methods train a light student network to replicate the output logits [ 17 ] or the final-layer features [ 18 , 19 ] of heavy teacher network. Despite their potential, these methods are limited by their reliance on single-image inputs, neglecting the temporal correlations between actions in consecutive frames—an essential factor in resolving the 2D-to-3D ambiguity in monocular HPE. Furthermore, these approaches fail to emphasize that the student network should prioritize learning the relationships between human joints from the teacher network rather than merely replicating final outputs or high-level semantic features. These limitations ultimately restrict their accuracy and performance. Fig. 1: Comparison between the previous methods PoseFormerV2 [ 14 ] , HDFormer [ 8 ] , MHFormer [ 10 ] and our SCJD on Human3.6M. f represents the length of the input sequence. To address these challenges, we propose a novel S parse C orrelation and J oint D istillation (SCJD) for 3D HPE. Our key insight is to leverage sparse input sequences and KD to reduce computational overhead while maintaining accuracy. Specifically, Our approach begins with Sparse Correlation Input Sequence Downsampling , which sparsifies the student network’s inputs to reduce redundancy while preserving inter-frame correlations more effectively than single-frame methods. Next, to transfer knowledge effectively from a dense, heavy teacher network to a sparse, lightweight student network, we introduce Dynamic Joint Spatial Attention Distillation , which comprises two components: a) Dynamic Joint Embedding Distillation , which aggregates the local context information of teacher’s spatial encoder to enhance the student’s spatial feature representation. b) Adjacent Joint Attention Distillation , which employs an adjacent joint attention matrix to guide the student network in focusing on connections between adjacent joints, improving its understanding of human posture. Finally, we propose Temporal Consistency Distillation , which aligns the temporal correlation between input contexts by upsampling the student’s features to match the teacher network’s outputs, and complemented by global supervised learning with ground truth. Our extensive experiments validate our method’s superiority over existing state-of-the-art techniques in terms of both efficiency and accuracy. Our contributions are threefold: • We propose a novel framework SCJD, which enhances the efficiency and accuracy of 3D HPE by utilizing sparse correlation inputs and dynamic joint spatial attention distillation. • We introduce dynamic joint embedding distillation and adjacent joint attention distillation, enabling better alignment of joint spatial features between the teacher and student spatial encoder, significantly improving the student model’s performance. • Extensive experiments demonstrate that our method achieves superior speed-accuracy trade-offs, delivering up to 6.15 \times speedup with comparable accuracy, as illustrated in Fig. 1 . Fig. 2: Overview of the SCJD framework. SCJD comprises two transformer encoders that separately learn spatial and temporal information. The student model first employs Sparse Correlation Input Sequence Downsampling to reduce input redundancy while preserving temporal correlations. Then, through Dynamic Joint Embedding Distillation and Adjacent Joint Attention Distillation , knowledge from the teacher network’s spatial encoder is transferred to help the student model learn joint associations. Finally, Temporal Consistency Distillation facilitates the transfer of temporal knowledge from the teacher’s temporal encoder, enabling the student model to capture the temporal relationships between multi-frame postures. II Related Work II-A Efficient 3D Human Pose Estimation The goal of efficient 3D human pose estimation is to strike a balance between accuracy and computational efficiency. While recent studies [ 5 , 6 , 7 , 8 , 9 , 10 , 11 ] have prioritized improving accuracy, fewer have tackled the challenge of computational efficiency. Early efficient methods, such as Choi et al. [ 20 ] , achieved faster pose estimation by utilizing compact networks and skip concatenation to reduce computational complexity. However, these approaches rely on single-frame inputs, which lack temporal context and limit accuracy compared to more advanced methods. Multi-frame inputs improve accuracy but increase computational demands. To address this, Li et al. [ 13 ] used strided convolutions to reduce the sequence length while capturing local context. Zhao et al. [ 14 ] encoded longer sequences into low-frequency coefficients combined with shorter pose sequences. While these methods reduce input length, they still require deeper, wider networks, leading to higher computational costs. II-B Knowledge Distillation in 3D Human Pose Estimation Knowledge distillation [ 16 ] is an effective technique for compressing networks by transferring knowledge from a large teacher network to a lightweight student network, improving the latter’s accuracy. Several studies [ 17 , 18 , 19 ] have applied distillation to accelerate 3D human pose estimation. Hwang et al. [ 17 ] trained a student network to mimic the teacher’s output logits, while Kim et al. [ 18 ] used pruning and feature imitation to boost performance. Bian et al. [ 19 ] transferred multi-view stereo knowledge to