CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks
Mingshuang Luo, Ruibing Hou, Bo Chao, Hong Chang, Zimo Liu, Yaowei Wang, Shiguang Shan · 2026
Read CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks by Mingshuang Luo, Ruibing Hou, Bo Chao, Hong Chang, Zimo Liu, Yaowei Wang, Shiguang Shan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13133.txt.
Introduction
Mingshuang Luo and co-authors build a train path for vision tasks that centre on people. Such tasks span watch cams, health care, and how people use machines. Huge sets of unmarked photos of people now exist. The field needs one broad pre train model that can serve many such tasks.
Their path is self taught, so it needs no hand marks to start. It is led by CLIP, a model that links text and image. It can also adapt to fit each new task. The aim is a single base that lifts scores across many people centred tasks at once. Teams gain a start point that is both broad and cheap to tune.
Catalog record
| CREATOR | Mingshuang Luo, Ruibing Hou, Bo Chao, Hong Chang, Zimo Liu, Yaowei Wang, Shiguang Shan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks Mingshuang Luo, Ruibing Hou, Bo Chao, Hong Chang, Zimo Liu, Yaowei Wang, Shiguang Shan Abstract Human-centric visual analysis plays a pivotal role in diverse applications, including surveillance, healthcare, and human-computer interaction. With the emergence of large-scale unlabeled human image datasets, there is an increasing need for a general unsupervised pre-training model capable of supporting diverse human-centric downstream tasks. To achieve this goal, we propose CLASP (CLIP-guided Adaptable Self-suPervised learning), a novel framework designed for unsupervised pre-training in human-centric visual tasks. CLASP leverages the powerful vision-language model CLIP to generate both low-level ( e.g. body parts) and high-level ( e.g. attributes) semantic pseudo-labels. These multi-level semantic cues are then integrated into the learned visual representations, enriching their expressiveness and generalizability. Recognizing that different downstream tasks demand varying levels of semantic granularity, CLASP incorporates a Prompt-Controlled Mixture-of-Experts (MoE) module. MoE dynamically adapts feature extraction based on task-specific prompts, mitigating potential feature conflicts and enhancing transferability. Furthermore, CLASP employs a multi-task pre-training strategy, where part- and attribute-level pseudo-labels derived from CLIP guide the representation learning process. Extensive experiments across multiple benchmarks demonstrate that CLASP consistently outperforms existing unsupervised pre-training methods, advancing the field of human-centric visual analysis. I Introduction Human-centric visual analysis plays a crucial role in various applications, including video surveillance [ 18 , 19 , 86 , 29 ] , autonomous driving [ 11 , 51 , 8 ] , and augmented reality [ 15 , 81 ] . This field encompasses a wide range of tasks, such as person re-identification (reID) [ 21 , 48 , 66 ] , human parsing [ 37 , 67 , 33 ] , pose estimation [ 85 , 60 ] , pedestrian detection [ 45 , 17 ] , and attribute recognition [ 44 , 69 ] . Despite significant progress in each individual task, most existing models are tailored for single task, leading to substantial overhead in model design and pre-training. To facilitate efficient and scalable deployment in real-world scenarios, it is essential to develop a unified human-centric pre-training model capable of effectively adapting to a wide range of downstream tasks. Human-centric pre-training typically follows two mainstream paradigms: supervised multi-task pre-training and unsupervised pre-training. The supervised multi-task pre-training paradigm leverages large-scale annotated datasets to jointly learn transferable representations across multiple tasks. For example, HCMoCo [ 28 ] simultaneously optimizes for both pose estimation and human parsing, encouraging the model to capture shared structural semantics. UniHCP [ 9 ] further enhances inter-task parameter sharing by employing a shared decoder equipped with task-specific queries. Extending to multimodal settings, Hulk [ 71 ] proposes a generalist model that integrates 2D vision, 3D vision, skeleton-based representations, and vision-language tasks within a single framework. In contrast, unsupervised pre-training aims to learn generalizable representations from large-scale unlabeled data, providing a scalable and cost-effective alternative to supervised approaches. Earlier studies [ 49 , 75 ] directly adopt DINO [ 1 ] for self-supervised representation learning in human-centric scenarios. SOLIDER [ 4 ] further introduces a semantic controller to regulate appearance and semantic information across tasks, while HAP [ 78 ] incorporates human part priors into the masked autoencoding (MAE) frameworks [ 25 ] . Given the superior scalability and reduced reliance on manual annotations, our goal is to develop an unsupervised human-centric pre-training model that fully leverages unlabeled data to support a wide range of downstream tasks. Fig. 1: Feature representation space for person re-identification, attribute recognition, and human parsing tasks. Each task is trained independently using Swin-Tiny as the backbone, with features extracted via average pooling. The visualization highlights that different tasks demand distinct feature representations, highlighting the challenge of learning a unified representation across diverse human-centric tasks. Despite recent advances, developing a general-purpose unsupervised pre-training model for human-centric tasks remains highly challenging, primarily due to the heterogeneous representation requirements inherent to different tasks. As illustrated in Fig. 1 , each task demands distinct feature characteristics. For instance, person reID relies heavily on global appearance consistency , requiring the preservation of discriminative visual cues to distinguish between individual identities. In contrast, semantic understanding tasks, such as human parsing, pose estimation and attribute recognition, require multi-level semantic consistency to capture fine-grained human-centric details. More specifically, human parsing and pose estimation emphasize low-level part-aware semantics , necessitating precise localization and differentiation of body parts. Meanwhile, attribute recognition depends on high-level global semantics to infer abstract human attributes, such as gender or age. However, as highlighted in [ 4 , 79 ] , mainstream unsupervised pre-training approaches, such as DINO [ 1 ] and MAE [ 25 ] , struggle to learn universally transferable representations across diverse human-centric tasks. Specifically, DINO [ 1 ] primarily captures appearance-based representations, which are well-suited for person reID task but insufficient for semantic understanding tasks that require rich multi-level semantic cues. On the other hand, MAE [ 25 ] , with its high masking ratio and reconstruction-based objective, tends to emphasize low-level visual features, thereby limiting its capacity to model high-level semantics and global appearance cues. SOLIDER [ 4 ] makes partial progress by introducing coarse-grained pseudo labels ( e.g. , upper body, lower body, and shoes) to provide semantic supervision during DINO pre-training. However, it still faces two fundamantal limitations: (1) It entangles appearance and coarse semantic representations, lacking the ability to model fine-grained semantics (e.g., hat, hair) and high-level concepts (e.g., gender, age); (2) It relies on manually tuned external parameters for adapting to each downstream task, which not only increases the adaptation cost but also limits scalability and flexibility in real-world deployments. In this work, we propose CLASP ( CL IP-guided A daptable S elf-su P ervised Learning), a unified unsupervised pre-training framework designed to address the heterogeneous representation requirements across diverse human-centirc tasks. First, CLASP leverages the powerful vision-language model CLIP to automatically generate multi-level semantic pseudo labels , covering both low-level body parts and high-level human attributes . These pseudo labels supervise two pretext tasks—body part classification and attribute classification—guiding the model to acquire rich and robust semantic representations across different levels. Second , to alleviate potential feature conflicts among different tasks, we introduce a Prompt-Controlled Mixture-of-Experts ( PC-MoE ) module. This module comprises a set of expert sub-networks, a channel-wise gating mechanism, and a global-wise gating mechanism, all dynamically modulated by a task-specific learnable prompt. The prompt serves as a control signal to steer the expert selection process, enabling adaptive and task-aware representation learning. Finally , CLASP is optimized through a set of hierarchical supervision signals that jointly supervise both appearance features and multi-level semantic cues. This combination of prompt-guided expert control and multi-level joint supervision empowers the pre-trained model with strong generalization and adaptability across a wide range of downstream human-centric tasks. Through comprehensive quantitative analysis, we demonstrate that CLIP can effectively generate reliable semantic pseudo labels for both human attributes and body parts. In addition, qualitative results show that task-specific prompt embeddings in CLASP successfully guide the model to learn adaptable representations. Extensive experiments on six diverse human-centric downstream tasks validate the effectiveness of our framework: CLASP achieves significant improvements on semantic tasks, while maintaining competitive performance on appearance-centric tasks. These results highlight the versatility and generalization capability of CLASP