SDPose : Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation

Shuang Liang, Jing He, Chuanmeizhi Wang, Lejun Liao, Guo Zhang, Yingcong Chen, Yuan Yuan, [3pt] Rama Alpaca Technology Company · 2025

Read SDPose : Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation by Shuang Liang, Jing He, Chuanmeizhi Wang, Lejun Liao, Guo Zhang, Yingcong Chen, Yuan Yuan, [3pt] Rama Alpaca Technology Company (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24980.txt.

Introduction

Diffusion models trained to generate images quietly learn rich features of bodies and scenes, and Shuang Liang and co-authors borrow those features for human pose estimation in a 2025 paper. Their system, SDPose, targets the hard case: poses in domains unlike the training data, under noise, weather, and blur.

A systematic sweep of the layers of the diffusion network found which levels carry the most robust features for the job. Those features are fused by a small aggregation module, and training pairs keypoint supervision with a reconstruction loss that protects the borrowed prior. The authors built a new out-of-domain benchmark from a standard data set, with style-shifted and corrupted splits. Light fine-tuning on it delivered strong cross-domain results.

Catalog record

CREATORShuang Liang, Jing He, Chuanmeizhi Wang, Lejun Liao, Guo Zhang, Yingcong Chen, Yuan Yuan, [3pt] Rama Alpaca Technology Company
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24980

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SDPose : Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation Shuang Liang, Jing He, Chuanmeizhi Wang, Lejun Liao, Guo Zhang, Yingcong Chen, Yuan Yuan, [3pt] Rama Alpaca Technology Company Abstract Pre-trained diffusion models provide rich multi-scale latent features and are emerging as powerful vision backbones. While recent works such as Marigold ( Ke et al. 2024 ) and Lotus ( He et al. 2024 ) adapt diffusion priors for dense prediction with strong cross-domain generalization, their potential for structured outputs (e.g., human pose estimation) remains underexplored. In this paper, we propose SDPose , a fine-tuning framework built upon Stable Diffusion to fully exploit pre-trained diffusion priors for human pose estimation. First, rather than modifying cross-attention modules or introducing learnable embeddings, we directly predict keypoint heatmaps in the SD U-Net’s image latent space to preserve the original generative priors. Second, we map these latent features into keypoint heatmaps through a lightweight convolutional pose head, which avoids disrupting the pre-trained backbone. Finally, to prevent overfitting and enhance out-of-distribution robustness, we incorporate an auxiliary RGB reconstruction branch that preserves domain-transferable generative semantics. To evaluate robustness under domain shift, we further construct COCO-OOD , a style-transferred variant of COCO with preserved annotations. With just one-fifth of the training schedule used by Sapiens on COCO, SDPose attains parity with Sapiens-1B/2B on the COCO validation set and establishes a new state of the art on the cross-domain benchmarks HumanArt and COCO-OOD. Furthermore, we showcase SDPose as a zero-shot pose annotator for downstream controllable generation tasks, including ControlNet-based image synthesis and video generation, where it delivers qualitatively superior pose guidance. Figure 1: SDPose: OOD-robust pose via diffusion priors. On stylized paintings, SDPose surpasses Sapiens and ViTPose++ - H, matching SoTA on COCO and setting new records on HumanArt and COCO - OOD; yellow boxes show baseline failures. 1 Introduction With the recent rise of embodied AI, video generation, and 3D asset rendering, the need for cross-domain-robust human pose estimation has become critical in robotics as well as in film, animation, and game production. Although recent advances on academic benchmarks such as MS COCO ( Lin et al. 2014 ) using models such as DWPose ( Yang et al. 2023 ) , RTMPose ( Jiang et al. 2023 ) and OpenPose ( Martınez 2019 ) , as well as approaches leveraging large pretrained backbones such as ViTPose ( Xu et al. 2022 ; Xu et al. 2023 ) and Sapiens ( Khirodkar et al. 2024 ) , have achieved strong in-domain accuracy, they often exhibit severe performance degradation under domain shifts and require substantial fine-tuning efforts. Recently, pre-trained diffusion models such as Stable Diffusion ( Rombach et al. 2022 ) have emerged as robust vision backbones. A growing body of work has shown that with fine-tuning and adaptation, diffusion priors can be repurposed for 3D generation ( Cheng et al. 2023 ; Lin et al. 2025 ; Long et al. 2024 ) , segmentation ( Karmann & Urfalioglu 2025 ) , and dense prediction tasks ( Ke et al. 2024 ; He et al. 2024 ) , while consistently demonstrating strong cross-domain robustness and highlighting their potential for leveraging intra-visual multimodality in generative priors. However, their potential for structured and semantically aware outputs, particularly in human pose estimation, remains largely unexplored. Concurrent efforts like GenLoc ( Wang et al. 2025a ) and Diff-Tracker ( Zhang et al. 2024 ) indicate that generative priors can benefit keypoint localization and tracking by steering learnable condition embeddings and adapting the diffusion model’s cross-attention. We instead examine a complementary axis: can one rely purely on SD U-Net latent features, without attention read-outs or condition tokens, to produce reliable pose heatmaps? To bridge this gap and systematically study how to leverage the rich latent features for robust, cross-domain pose estimation, we propose SDPose , a fine-tuning framework with three key components: (i) Latent-space preservation. We operate entirely in the SD U-Net’s image latent space without modifying cross-attention modules or adding learnable embeddings, thus preserving pretrained visual semantics and feature geometry. (ii) Lightweight pose decoder head. We employ one deconvolution layer (kernel size 4) followed by two 1\times 1 convolutions to map intermediate U-Net features directly into keypoint heatmaps, minimizing disruption to the pretrained backbone. (iii) RGB reconstruction regularization. We add an auxiliary reconstruction branch that regularizes fine-tuning, helping to maintain domain-transferable generative semantics and improve out-of-distribution robustness. Furthermore, to systematically evaluate robustness under domain shift, we introduce COCO-OOD , a style-transferred extension of COCO annotations featuring oil painting domains, which bridges the gap in benchmarking for generalization robustness evaluation. On COCO ( Lin et al. 2014 ) and COCO-WholeBody ( Jin et al. 2020 ) , SDPose delivers in-domain performance on par with the current SoTA, Sapiens ( Khirodkar et al. 2024 ) . Under domain shift (HumanArt ( Ju et al. 2023 ) , COCO-OOD), it sets a new state of the art while using only one-fifth of Sapiens’s fine-tuning epochs, highlighting the efficiency and cross-domain robustness of generative priors. Beyond quantitative benchmarks, we further demonstrate SDPose as a zero-shot pose annotator for downstream controllable generation tasks, including ControlNet-based image synthesis and video generation, where it provides reliable and qualitatively superior pose guidance. 2 Related Works 2.1 Latent Diffusion Models Latent diffusion models (LDMs), built on DDPM and further advanced by ODE and SDE samplers ( Ho et al. 2020 ; Song et al. 2020b ; Song et al. 2020a ; Lu et al. 2022 ; Rombach et al. 2022 ) , have gained traction over the past few years. Classic architectures, such as the UNet-based Stable Diffusion and Diffusion Transformers (DiT) ( Peebles & Xie 2023 ; Esser et al. 2024 ) , have demonstrated strong performance across diverse conditional generation tasks ( Zhang et al. 2023 ) . Pretrained on large-scale datasets such as LAION-5B ( Schuhmann et al. 2022 ) , generative models like Stable Diffusion provide rich visual priors that can be effectively leveraged for a wide range of tasks. Recent advances in flow-matching ( Lipman et al. 2022 ; Esser et al. 2024 ; Xie et al. 2024 ) further show that latent diffusion models can achieve high-quality synthesis with only a few sampling steps. These developments highlight the power of latent generative priors as a strong visual foundation. 2.2 Leveraging Diffusion Priors for Prediction Tasks A growing body of work has explored repurposing pretrained latent diffusion priors for dense prediction tasks. Marigold ( Ke et al. 2024 ) adapts Stable Diffusion by fine-tuning only the denoising U-Net using synthetic data, delivering high-quality depth results. Later, subsequent methods such as Lotus ( He et al. 2024 ) and GenPercept ( Xu et al. 2024 ) both adopt a deterministic one-step fine-tuning strategy, removing the multi-step stochastic diffusion process and directly predicting task annotations, which significantly improves both accuracy and inference speed. In contrast, leveraging latent diffusion priors for structured outputs (e.g., human pose) remains underexplored. Prior works such as GenLoc ( Wang et al. 2025a ) and Diff - Tracker ( Zhang et al. 2024 ) freeze Stable Diffusion backbone and use learnable condition or prompt embeddings to read cross-attention maps, rather than decoding from U-Net latent features, aiming at zero-/few-shot and schema-flexible generalization. We instead

Read the complete text