Using Knowledge Graphs to harvest datasets for efficient CLIP model training

Simon Ging, Sebastian Walter, Jelena Bratulić, Johannes Dienert, Hannah Bast, Thomas Brox · 2025

Read Using Knowledge Graphs to harvest datasets for efficient CLIP model training by Simon Ging, Sebastian Walter, Jelena Bratulić, Johannes Dienert, Hannah Bast, Thomas Brox (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02746.txt.

Introduction

Simon Ging and co-authors harvest training sets for image-text models with knowledge graphs in a 2025 paper. CLIP models learn to match images with captions, but they normally need enormous data sets. That cost blocks custom models for narrow fields, which even the largest models cover poorly. Science teams also lose control over how their model was fed.

The team used knowledge graphs, structured webs of facts about things and their kinds, to steer web searches for images. With that guidance, a strong CLIP model was trained from scratch on far less data. Their proof case was an expert model for living organisms, built from ten million images. They also release EntityNet, a set of thirty-three million images with forty-six million captions, for training a general model much faster. The authors frame curation, not scale alone, as the lever.

Catalog record

CREATORSimon Ging, Sebastian Walter, Jelena Bratulić, Johannes Dienert, Hannah Bast, Thomas Brox
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02746

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Using Knowledge Graphs to harvest datasets for efficient CLIP model training Simon Ging, Sebastian Walter, Jelena Bratulić, Johannes Dienert, Hannah Bast, Thomas Brox Abstract Training high-quality CLIP models typically requires enormous datasets, which limits the development of domain-specific models – especially in areas that even the largest CLIP models do not cover well – and drives up training costs. This poses challenges for scientific research that needs fine-grained control over the training procedure of CLIP models. In this work, we show that by employing smart web search strategies enhanced with knowledge graphs, a robust CLIP model can be trained from scratch with considerably less data. Specifically, we demonstrate that an expert foundation model for living organisms can be built using just 10M images. Moreover, we introduce EntityNet , a dataset comprising 33M images paired with 46M text descriptions, which enables the training of a generic CLIP model in significantly reduced time. 1 Introduction Figure 1 : We demonstrate how to harvest datasets for training CLIP models with an improved quality-cost trade-off, for a generic (left) or an expert domain (right). Contrastive Language-Image Pretraining (CLIP) [ 1 ] has become a cornerstone for training Vision-Language Models (VLMs). CLIP models learn high-quality visual embeddings and establish a link to the semantic level of brief text descriptions by training on pairs of images and their corresponding text descriptions collected from the web. The features and the link between images and text have been used directly for, e.g., zero-shot classification or text-to-image retrieval, and enable dialogues with visual input, such as in the LLaVA family of models [ 2 , 3 , 4 ] . The link can also be exploited in the opposite direction to enable text-conditional image generation, e.g., Stable Diffusion [ 5 ] . Training state-of-the-art CLIP models is computationally expensive. The original CLIP model has seen 12.8B image-text pairs, and later works have scaled this further [ 6 , 7 ] . This need for scale has limited most of the research to finetuning, which comes with reduced architectural flexibility and control over the data selection. It is particularly problematic for analytic research that demands full control over training to find causes of emergent behavior. The effort to collect vast datasets is also a key bottleneck for building foundation models for expert domains. Although CLIP models are supposed to be generic and cover most of the world, they are not good enough for use in specific expert domains such as medicine or biology. Building foundation models for expert domains requires an efficient data collection process, taking into account the availability of fewer data samples in these domains. Our goal is to tackle these challenges from the dataset side while keeping the CLIP algorithm fixed. This strategy is backed by recent literature. For example, Li et al. [ 8 ] explored CLIP “along three dimensions: data, architecture, and training strategies” and they stress the “significance of high-quality training data”. For Large Language Models (LLMs), data curation was shown to reduce training time and model size, achieved through heavily filtered publicly available web data and synthetic data [ 9 ] . With the dataset creation process, we aim (1) for improved performance in the expert domain of living organisms, in order to demonstrate the creation of expert foundation models; and (2) we aim for a good trade-off between training efficiency and model performance on the broad domain of the visual world, in order to enable compute-efficient from-scratch analysis of fully functional CLIP models. We built a dataset we call EntityNet , where we leveraged knowledge graphs and targeted web image search. Specifically, from the knowledge graphs Wikidata and WordNet, we collected 135k entities (e.g. eagle ) as well as their aliases and descriptions. We extracted entity attributes from Wikidata related to color, partonomy, behavior, and other aspects, and used them to guide an LLM in generating entity-attribute queries for image search. For example, from the entity plastic and the attribute small we generated the query small plastic item . The resulting EntityNet consists of 33M images paired with 45M alt texts and 613k text labels from the knowledge graphs. The dataset is partitioned into a subset of 10M images of living organisms, capturing high-quality visual and semantic information about the taxonomy of animals, plants, and funghi, as well as a subset of 23M images covering a wide range of categories, such as tools, geographical features, materials, and buildings. Notably, from this process we obtain not only alt texts, but also a link back to the knowledge graph information that was used to create the search query for a given image. We show that this information can be used during training to achieve better performance than by training on alt texts alone. The method of creating our dataset is largely generic and can be applied to other knowledge graphs. Training on this dataset, we obtain a foundation model that is both specialized on the target expert domain and is also able to understand the overall visual world. In our domain-specific evaluations on iNaturalist and RareSpecies, the model demonstrates robust generalization and clearly surpasses CLIP models trained on much more data ( Figure 1 ). On ImageNet, we demonstrate our dataset to be highly compute efficient and to achieve a performance comparable to models trained 20x longer ( Figure 1 ). • We propose a method to automatically create a vision-language dataset based on a given knowledge graph and an image search engine. • We apply this method to create the EntityNet dataset, consisting of 33M images paired with 45M alt texts and supplementary text information from the knowledge graphs. • We train an expert CLIP model for living organisms on a single 8xL40S machine from scratch in 55 hours. This EntityNet-CLIP is highly specialized in the target expert domain of living organisms, and comparably strong on ImageNet. • We evaluate our model and a suite of other CLIP models for object classification, image retrieval, and domain shift robustness. In the expert domain of animals and plants, our model achieves higher performance than models with orders of magnitude more parameters or training data. It is also much stronger than CLIP models that specialize only for this domain. In the generic domain, our model performs remarkably well given the low amount of compute required to train it. Figure 2 : We create a dataset for vision-language pretraining: First, we extract entities from knowledge graphs, then generate attributes and natural types for them. We search for different combinations of entities, attributes, and types in image search engines, and collect alt texts for each image. Finally, we train our model on the combined data. 2 Related work Datasets. Many recent studies have investigated methods for building large-scale datasets for multimodal training. Radford et al. [ 1 ] trained the original CLIP model on a private dataset of 400M images using image-text pairs with text derived from Wikipedia and WordNet terms. Schuhmann et al. [ 10 ] further built the publicly available LAION-400M dataset by filtering HTML data from Common Crawl [ 11 ] based on the similarity estimated by the CLIP model. In a follow-up work, they [ 12 ] scaled their approach one order of magnitude

Read the complete text