D3G: Diverse Demographic Data Generation Increases Zero-Shot Image Classification Accuracy within Multimodal Models

Javon Hickmon · 2025

Read D3G: Diverse Demographic Data Generation Increases Zero-Shot Image Classification Accuracy within Multimodal Models by Javon Hickmon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15747.txt.

Introduction

Javon Hickmon presents D3G, a method that uses diverse demographic data generation to improve zero shot image classification, in a 2025 paper, dated December 10. Multimodal models like CLIP classify images by matching them to text, but unbalanced training data can bias predictions toward over represented groups. Low capacity models also struggle with fine grained classes.

D3G is training free: it uses CLIP as the base model and Stable Diffusion XL as a generator, supplying diverse demographic examples at inference time. Hickmon shows this raises classification accuracy while cutting demographic bias in the pretrained model. The study also measures how individual demographic factors affect the resulting accuracy.

Catalog record

CREATORJavon Hickmon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15747

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D3G: Diverse Demographic Data Generation Increases Zero-Shot Image Classification Accuracy within Multimodal Models Javon Hickmon Abstract Image classification is a task essential for machine perception to achieve human-level image understanding. Multimodal models such as CLIP have been able to perform well on this task by learning semantic similarities across vision and language; however, despite these advances, image classification is still a challenging task. Models with low capacity often suffer from underfitting and thus underperform on fine-grained image classification. Along with this, it is important to ensure high-quality data with rich cross-modal representations of each class, which is often difficult to generate. When datasets do not enforce balanced demographics, the predictions will be biased toward the more represented class, while others will be neglected. We focus on how these issues can lead to harmful bias for zero-shot image classification, and explore how to combat these issues in demographic bias. We propose Diverse Demographic Data Generation (D3G), a training-free, zero-shot method of boosting classification accuracy while reducing demographic bias in pre-trained multimodal models. With this method, we utilize CLIP as our base multimodal model and Stable Diffusion XL as our generative model. We demonstrate that providing diverse demographic data at inference time improves performance for these models, and explore the impact of individual demographics on the resulting accuracy metric. 1 Introduction Deep Learning systems have been a promising new paradigm in the field of image classification. Vision-Language systems are able to utilize multiple modalities to create models that are more generalizable to a broad range of downstream tasks. Despite this performance, issues such as data redundancy, noise, and class imbalance are just a few of the many difficulties that arise from collecting large amounts of training data. Multimodal models, in particular, require large amounts of high-quality training data with rich cross-modal representations in order to perform well compared to their unimodal counterparts. These models leverage the massive amounts of image-text pairs available online by learning to associate the images with their correct caption, leading to greater flexibility during inference ( 5 ) ; however, class imbalances can often lead to gender and racial bias depending on the desired task. Existing public face image datasets are strongly biased toward Caucasian faces; meanwhile, other races (i.e., Latino) are significantly underrepresented fig. 4 . As a result, the models trained from such datasets suffer from inconsistent classification accuracies, limiting the applicability of such systems to predominantly white racial groups 3 . This means that minority subpopulations can potentially be further marginalized when applied to certain downstream tasks without calibration. This is a core tenet of machine learning: poor data produces poor models. Figure 1: Images Generated with D3G for Race 4 It is important to note that we acknowledge that not all bias is harmful and often is necessary for models to generalize. This is why within this work, we focus on demographic bias which can frequently have harmful implications when applied within society. Image classification is particularly pressing, because it is the core of a myriad of computer vision tasks. Facial recognition, object detection, image search, content moderation, sentiment analysis, and many more tasks are grounded in accurate image classification systems. This is compounded by the fact that many widely used Foundational Models require multiple modalities, such as DALL-E 7 and Stable Diffusion 8 . In order to train these models, other models such as CLIP 6 are used to classify the data that the model will be trained on, in order to enforce a strong cross-modal correlation. This means that demographic biases will be compounded as the images continue to be utilized in training processes. When there are strong harmful demographic biases, these models can cause tremendous harms. One example was when Google Photo’s classified black people as gorillas within a user’s album, as shown in fig. 2 . This is the direct result of demographic bias. The dataset, used to quantify the classification accuracies of Google’s model, likely contained the standard biases where images of non-Caucasian faces are underrepresented fig. 4 . This leads to a misleading evaluation, which was likely why this model was deployed to the public with this significant issue. In this work, we explore how demographic bias affects image classification accuracy for multimodal models. We also propose D3G, a zero-shot, training-free framework to balance demographic bias and boost classification accuracies for multimodal models used for image classification. Figure 2: The image of Google Photos misclassifying black people within the Photos application. 2 Related Work 2.1 Model Ensembling for Image Classification Many state-of-the-art techniques for image classification leverage a methodology known as model ensembling. Ensemble learning broadly is an approach that aims to improve performance by combining the predictions of multiple models. There are many such ensemble learning methods, but the one most relevant to our proposed technique is called bagging. Bagging predictors originally published by 1 introduced “bootstrap aggregating” or bagging, the ensemble learning method that combines multiple models trained on different subsets of the training data. The formulation is presented as follows: A learning set of L consists of data {(y_{n},\mathbf{x}_{n}),n=1,...,N} where the y ’s are class labels, assume we form a predictor \phi(\mathbf{x},L) where if the input is \mathbf{x} , we predict y by \phi(\mathbf{x}) . Now suppose we form a sequence of replicate learning sets, {L^{(B)}} each consisting of N observations drawn at random with replacement from L . Since y is a class label in our scenario, the predictions from each of the predictors {\phi(\mathbf{x},L^{(B)})} will vote to form \phi_{B}(\mathbf{x}) , the aggregated final prediction. Bagging both empirically and theoretically prove improves accuracy for a given set of weak classifiers or “weak learners.” This technique effectively replicates our proposed method within a controlled environment. In the implementation of D3G, we are employing a strategy similar to bagging, but across modalities and with generated data. Our goal is based on the theoretical guarantees of bagging, where even though each model is trained on a subset of the data, the aggregation of the predictions begins to approximate the true distribution of the data. A model that applies this concept of model ensembling was introduced in Learning to Navigate for Fine-grained Classification by 11 . This is a state-of-the-art paper that attempts to reduce misclassification rates by developing a model called NTS-Net (Navigator-Teacher-Scrutinizer Network) to teach itself methods of identifying and scrutinizing fine-grained image details. The Navigator navigates the model to focus on the most informative regions (denoted by yellow rectangles in fig. 3 ), while the Teacher evaluates the regions proposed by the Navigator and provides feedback. After that, the Scrutinizer scrutinizes those regions to make predictions. NTS-Net achieves high classification accuracy on a pre-defined dataset; however, we wanted to explore if these foundational concepts could be applied to a training-free, zero-shot environment. In order to achieve this, we leverage pretrained open-vocabulary models with advanced attention mechanisms to discriminate the fine-grained features of a given image, to improve performance on a variety of classes without additional training. Figure 3: The NTSNet Architecture ( 11 ) Similarly, within the paper Sus-X: Training-Free Name-Only Transfer of Vision-Language Models by 9 achieves state-of-the-art zero-shot classification results on 19 benchmark datasets, outperforming other training-free adaptation methods. It also demonstrates strong

Read the complete text