Role-SynthCLIP: A Role Play Driven Diverse Synthetic Data Approach
Yuanxiang Huangfu, Chaochao Wang, Weilei Wang · 2025
Read Role-SynthCLIP: A Role Play Driven Diverse Synthetic Data Approach by Yuanxiang Huangfu, Chaochao Wang, Weilei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05057.txt.
Introduction
Yuanxiang Huangfu, Chaochao Wang, and Weilei Wang present Role-SynthCLIP, a way to make synthetic image-text pairs with richer meaning. CLIP models learn from such pairs, but past synthetic sets add bulk while their captions stay shallow and alike.
Role-SynthCLIP asks a large multimodal model to write captions from set roles, such as a layout analyst or a reader of scene context. Each role sees the image in a fresh way, so captions vary more and match fine detail better. The result is training data with broader sense and tighter image-text fit.
Catalog record
| CREATOR | Yuanxiang Huangfu, Chaochao Wang, Weilei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05057
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Role-SynthCLIP: A Role Play Driven Diverse Synthetic Data Approach Yuanxiang Huangfu, Chaochao Wang, Weilei Wang Abstract The effectiveness of Contrastive Language-Image Pre-training (CLIP) models critically depends on the semantic diversity and quality of their training data. However, while existing synthetic data generation methods primarily focus on increasing data volume, such emphasis often leads to limited semantic diversity and redundant or shallow captions. To address this limitation, we propose Role-SynthCLIP , a novel data synthesis framework that leverages multi-perspective role-playing prompts (e.g., a compositional analyst, an interpreter of image context) to guide Multimodal Large Language Models (MLLMs) in generating semantically diverse captions from distinct viewpoints. This mechanism enhances the semantic diversity and fine-grained image-text alignment of synthetic pairs, thereby improving caption expressiveness and accuracy while keeping the total number of image-text pairs unchanged. Experimental results demonstrate the effectiveness and efficiency of our method. A CLIP-B/16 model trained on only 1 million Role-SynthCLIP pairs achieves a Recall@1 of 64.1 % \mathbf{64.1\%} on the MS COCO validation set, surpassing the best existing synthetic data baseline (trained on 5M pairs) by 2.8 2.8 percentage points. The code and trained models are released at https://github.com/huangfu170/Role-SynthCLIP . 1 Introduction The paradigm of vision-language models (VLMs), such as CLIP Radford et al. 2021 , has profoundly reshaped multimodal learning by aligning vision and language representations luo et al. 2023 . Traditionally, their success has relied on massive web-crawled datasets containing millions to billions of image-text pairs Zhang et al. 2024 . However, this pursuit of data scale—from millions to billions of pairs—has created a critical bottleneck: the growing data volume demands prohibitive computational resources and inevitably incorporates noisy, low-quality, and redundant web samples Sun et al. 2023 ; Fang et al. 2024 . In this context, controllable high-quality synthetic data generation has emerged as a promising alternative to large-scale web scraping. Methods such as SynthCLIP Hammoud et al. 2024 have demonstrated that competitive performance can be achieved with fully synthetic corpora of up to 30M pairs. Nevertheless, as attention shifts from data scale to data quality, a new challenge arises: semantic impoverishment. Captions generated by generic prompts often exhibit limited diversity: they tend to be uniform, single-perspective, and semantically repetitive, failing to capture the rich, multifaceted nature of visual content. This redundancy constrains the diversity and discriminative capacity of learned visual–language representations. Figure 1 : Comparison of synthetic data generation paradigms. Our proposed Role-SynthCLIP framework addresses the semantic impoverishment of existing methods by leveraging multi-expert role-playing prompts. (Top) Under the conventional paradigm, generic prompts fed into the MLLM yield shallow, single-perspective descriptions. (Bottom) Our approach introduces role-play templates (e.g., Compositional Analyst, Narrative Setter), which guide the MLLM to perceive the image from diverse cognitive perspectives. This mechanism generates semantically rich captions that focus on fine-grained elements (e.g., specific objects, visual structure, context), maximizing the diversity of representations learned during VLM pre-training. Existing efforts to improve synthetic data quality primarily target distributional balance or prompt variety. For instance, SynthCLIP Hammoud et al. 2024 maintains entity distribution balance across the corpus. Other concurrent works, like FIX-CLIP Wang et al. 2025 , attempt to capture diversity by employing ensembles of multiple generic prompts (e.g., 20 variants). However, these ensemble approaches lack cognitive depth: their variations are often shallow paraphrases that do not introduce distinct interpretative dimensions-such as the compositional, contextual, or emotional perspectives-of the same image. To address this limitation, we propose Role-SynthCLIP , a novel framework that amplifies caption diversity by simulating multi-expert perspectives through role-playing prompts. As illustrated in Figure 1 , our method assigns distinct expert roles to a Multimodal Large Language Model (MLLM). For instance, an Observer of Details highlights fine-grained attributes, while a Compositional Analyst focuses on global structure and visual organization. By combining these complementary perspectives, Role-SynthCLIP produces semantically rich and well-aligned captions that better reflect the complexity of visual scenes. Our design is motivated by the proven effectiveness of role-playing prompting in LLMs, which has been shown to elicit diverse responses and improve task performance without architectural modifications or additional training Zhao et al. 2025 ; Kong et al. 2024 . By extending this paradigm to MLLMs for caption generation, we can efficiently and flexibly steer the model to produce the desired semantic diversity. In summary, the main contributions of this paper can be summarized as threefold: • We propose Role-SynthCLIP, a novel data synthesis framework. By leveraging role-playing prompts, it enables an MLLM to perceive an image from diverse, multi-faceted perspectives (e.g., compositional, detailed, contextual), generating captions with superior semantic diversity and alignment. • We establish new state-of-the-art (SOTA) performance for data-efficient CLIP training. A CLIP-B/16 model trained on only 1M Role-SynthCLIP pairs attains a Recall@1 of 64.1% on COCO-val, surpassing the best 5M-pair synthetic baseline. • Our work includes a detailed ablation study and empirical analysis across 8 downstream tasks, providing deep insights into the crucial impact of various expert role combinations on the learned representation diversity and model performance. 2 Related Works 2.1 Vision-Language Models Recent years have witnessed rapid progress in vision-language models (VLMs) that bridge visual perception and natural language understanding. Among them, Contrastive Language-Image Pretraining (CLIP) Radford et al. 2021 ; Peleg et al. 2025 ; Chen et al. 2023 ; Lai et al. 2024 has emerged as a representative framework, demonstrating how large-scale image-text alignment can lead to transferable multimodal representations. Subsequent studies have investigated extensions of this paradigm, including scaling pretraining corpora Sun et al. 2023 ; Fang et al. 2024 , improving robustness under distribution shifts Nguyen et al. 2023 , and adapting CLIP-style embeddings for tasks that require multimodal reasoning tasks Guzhov et al. 2021 ; Alyami and Luqman 2025 ; Zhao et al. 2025 . Despite this progress, a key challenge remains in effectively increasing semantic richness and alignment of the training data, particularly when operating within a constrained training corpus or relying on synthetic generation. Our work specifically addresses this quality bottleneck in the constrained data budget. 2.2 Synthetic Captions for VLM training Recent advances have demonstrated that caption diversity is paramount for improving the quality of contrastive pre-training. Prior work has explored data volume expansion Xu et al. 2024 ; Chuang et al. 2025 , simple caption rewriting Liu et al. 2024 , or linguistic transformations. More recently, with the development of large multimodal models, it has become possible to generate relevant captions for images Lai et al. 2024 ; Zheng et al. 2024 . While works like SynthCLIP Hammoud et al. 2024 focus on balancing entity distributions, and others like FIX-CLIP Wang et al. 2025 use multiple fixed prompts for variations. However, these generative approaches are typically constrained by generic prompts. They produce repetitive and shallow descriptions from a narrow viewpoint, resulting in surface-level paraphrasing rather than deep cognitive diversity. In contrast, our proposed Role-SynthCLIP systematically guides the MLLM through multi-experts role-playing prompts, enabling it to describe an image from diverse semantic perspectives. This approach unlocks the latent reasoning capacity of MLLMs to produce captions that are both fine-grained and semantically complementary. 2.3 Role-Play Prompting Role-playing prompting is a powerful, training-free technique widely employed in Large Language Models (LLMs) across specialized domains, such as education Dan et al. 2023 , healthcare Abbasian et al. 2024 ; Yu et al.