RealSyn : An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm
Tiancheng Gu ♥ {}^{\text{♥}} * , Kaicheng Yang ♠ {}^{\text{♠}} * , Chaoyi Zhang ♥ {}^{\text{♥}} , Yin Xie ♠ {}^{\text{♠}} , Xiang An ♠ {}^{\text{♠}} , Ziyong Feng ♠ {}^{\text{♠}} , Dongnan Liu ♥ {}^{\text{♥}} , Weidong Cai ♥ {}^{\text{♥}} † \dagger , Jiankang Deng ❉ {}^{\text{❉}} † \dagger ♥ {}^{\te · 2025
Read RealSyn : An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm by Tiancheng Gu ♥ {}^{\text{♥}} * , Kaicheng Yang ♠ {}^{\text{♠}} * , Chaoyi Zhang ♥ {}^{\text{♥}} , Yin Xie ♠ {}^{\text{♠}} , Xiang An ♠ {}^{\text{♠}} , Ziyong Feng ♠ {}^{\text{♠}} , Dongnan Liu ♥ {}^{\text{♥}} , Weidong Cai ♥ {}^{\text{♥}} † \dagger , Jiankang Deng ❉ {}^{\text{❉}} † \dagger ♥ {}^{\te (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12513.txt.
Introduction
Tiancheng Gu and co-authors present RealSyn, a data set and method to turn interleaved documents into training data for vision-language models, in a research paper from 2025. Models such as CLIP learn from image-text pairs. Vast stores of mixed documents, where images and text interleave, sit unused because their links are loose and noisy.
A pipeline extracts clean images and texts from real documents. A layered retrieval step pairs each image with several relevant real texts. A generation module adds synthetic text for fine detail, and balanced sampling covers rare concepts. Sets at three scales result. Models trained on them beat rivals at like scale across tasks.
Catalog record
| CREATOR | Tiancheng Gu ♥ {}^{\text{♥}} * , Kaicheng Yang ♠ {}^{\text{♠}} * , Chaoyi Zhang ♥ {}^{\text{♥}} , Yin Xie ♠ {}^{\text{♠}} , Xiang An ♠ {}^{\text{♠}} , Ziyong Feng ♠ {}^{\text{♠}} , Dongnan Liu ♥ {}^{\text{♥}} , Weidong Cai ♥ {}^{\text{♥}} † \dagger , Jiankang Deng ❉ {}^{\text{❉}} † \dagger ♥ {}^{\te |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12513
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RealSyn : An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm Thanks: * Equal Contribution † \dagger Corresponding Author Tiancheng Gu ♥ {}^{\text{♥}} * , Kaicheng Yang ♠ {}^{\text{♠}} * , Chaoyi Zhang ♥ {}^{\text{♥}} , Yin Xie ♠ {}^{\text{♠}} , Xiang An ♠ {}^{\text{♠}} , Ziyong Feng ♠ {}^{\text{♠}} , Dongnan Liu ♥ {}^{\text{♥}} , Weidong Cai ♥ {}^{\text{♥}} † \dagger , Jiankang Deng ❉ {}^{\text{❉}} † \dagger ♥ {}^{\text{♥}} The University of Sydney ♠ {}^{\text{♠}} DeepGlint ❉ {}^{\text{❉}} Imperial College London [email protected], [email protected] Project Page Code Abstract . After pre-training on extensive image-text pairs, Contrastive Language-Image Pre-training (CLIP) demonstrates promising performance on a wide variety of benchmarks. However, a substantial volume of multimodal interleaved documents remains underutilized for contrastive vision-language representation learning. To fully leverage these unpaired documents, we initially establish a Real-World Data Extraction pipeline to extract high-quality images and texts. Then we design a hierarchical retrieval method to efficiently associate each image with multiple semantically relevant realistic texts. To further enhance fine-grained visual information, we propose an image semantic augmented generation module for synthetic text production. Furthermore, we employ a semantic balance sampling strategy to improve dataset diversity, enabling better learning of long-tail concepts. Based on these innovations, we construct RealSyn , a dataset combining realistic and synthetic texts, available in three scales: 15M, 30M, and 100M. We compare our dataset with other widely used datasets of equivalent scale for CLIP training. Models pre-trained on RealSyn consistently achieve state-of-the-art performance across various downstream tasks, including linear probe, zero-shot transfer, zero-shot robustness, and zero-shot retrieval. Furthermore, extensive experiments confirm that RealSyn significantly enhances contrastive vision-language representation learning and demonstrates robust scalability. Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755098 ISBN: 979-8-4007-2035-2/2025/10 † † cc-license: by 1. Introduction The rapid proliferation of mobile networks and social platforms has led to exponential growth in large-scale data, offering a robust foundation for contrastive vision-language representation learning ( Guo et al., 2019 ; Baltrušaitis et al., 2018 ; Guo et al., 2024a ; Zhu et al., 2024a ; Yao et al., 2024 ; Zheng et al., 2024a ; Guo et al., 2024b ) . By predicting the correspondence between images and description texts, Contrastive Language-Image Pre-training (CLIP) ( Radford et al., 2021 ) pre-trains separate uni-modal encoders and achieves excellent performance across a range of downstream tasks, including image captioning ( Li et al., 2023 ; Mokady et al., 2021 ) , object detection ( Wu et al., 2023 ; Lin and Gong, 2023 ) , and semantic segmentation ( He et al., 2023 ; Zhou et al., 2023 ; Lin et al., 2023 ) . Notably, the representational capacity of such models improves significantly with the scale of the training dataset ( Li et al., 2024a ) . Designing a new data paradigm to expand the scale and diversity of image-text datasets, thereby continuously improving the capabilities of pre-trained vision-language representations, represents a significant and challenging issue. Figure 1 . Multimodal interleaved documents are unsuitable for CLIP training. We construct distinct image-text pairs from such documents via retrieval and generation. Figure 2 . The Real-World Data Extraction pipeline to extract high-quality images and texts from interleaved image-text documents. In recent years, the introduction of large-scale image-text pair datasets ( Schuhmann et al., 2021 ; Schuhmann et al., 2022 ; Thomee et al., 2016 ; Byeon et al., 2022 ; Gadre et al., 2024 ) has substantially advanced vision-language representation learning. Due to download failures and non-English captions in the YFCC ( Thomee et al., 2016 ) dataset, DeCLIP ( Li et al., 2022 ) reprocesses a new version of the YFCC15M dataset. LAION400M ( Schuhmann et al., 2021 ) provides 400 million image-text pairs collected from the web and filtered using CLIP similarity. COYO700M ( Byeon et al., 2022 ) offers 747 million high-quality pairs curated through advanced filtering strategies, including CLIP similarity, watermark detection, and aesthetic scoring. Despite these advancements, a substantial amount of non-paired data, particularly interleaved image-text documents ( Zhu et al., 2024b ; Laurençon et al., 2024 ; Li et al., 2025 ) containing multiple images and text paragraphs without explicit correspondence, remains incompatible with conventional vision-language representation learning methods. Recent studies ( Fan et al., 2024 ; Yu et al., 2024 ; Zheng et al., 2024b ; Lai et al., 2024 ) aim to enhance the quality of open-source image-text datasets using existing models. For instance, LaCLIP ( Fan et al., 2024 ) leverages large language models (LLMs) to rewrite raw captions for better alignment with images. CapsFusion ( Yu et al., 2024 ) fine-tunes an LLM using a ChatGPT-generated instruction dataset to mitigate hallucination issues. DreamLIP ( Zheng et al., 2024b ) re-captions 30 million images with detailed descriptions using a pre-trained large multimodal model. However, these methods primarily enhance data quality by focusing on synthetic data, thereby overlooking the significance of real-world data. Moreover, the diversity and distribution of synthetic captions produced by these approaches are inherently restricted by the limitations of the underlying generative models. In this paper, we explore two fundamental questions: 1) How to utilize multimodal interleaved documents for CLIP training. 2) How to effectively leverage both realistic and synthetic texts to enhance CLIP performance. As depicted in Figure 1 , we initially develop a Real-World Data Extraction pipeline to extract high-quality images and texts from real-world multimodal interleaved documents. Subsequently, we design a hierarchical retrieval method to efficiently associate each image with multiple semantically relevant texts. To improve fine-grained image understanding, we introduce a visual semantic augmented generation module for synthetic text production. Additionally, we implement a semantic balance sampling strategy to enhance dataset diversity and facilitate the learning of long-tail concepts. Leveraging these innovations, we construct the RealSyn dataset, which incorporates both realistic and synthetic texts in three sizes: 15M, 30M, and 100M. We evaluate the RealSyn dataset against other widely used datasets of similar scale for CLIP training. Models pre-trained on RealSyn consistently demonstrate state-of-the-art performance on various downstream tasks, including linear probe, zero-shot transfer, zero-shot robustness, and zero-shot retrieval. Extensive experiments validate that RealSyn is effective for contrastive vision-language representation learning and shows excellent scalability. The main contributions of this paper are summarized as follows: • We propose an effective and scalable multimodal interleaved document transformation paradigm for contrastive vision-language representation learning. • We release RealSyn , a large-scale semantic balanced dataset that integrates both realistic and synthetic texts and is available in three sizes: 15M, 30M, and 100M. • We conduct extensive experiments and demonstrate the effectiveness and scalability of our proposed RealSyn dataset for CLIP training. 2. Related Work Large-Scale Pre-training Dataset. In recent years, several large-scale image-text datasets ( Chen et al., 2015 ; Clark and Gardner, 2017 ; Goyal et al., 2017 ; Byeon et al., 2022 ; Li et al., 2024b ) collected from the Internet have been released. The YFCC100M ( Thomee et al., 2016 ) dataset provides a comprehensive overview of the evolution of photo and video documentation and sharing from the inception of Flickr in 2004 until early 2014. Due to download failures