AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data
Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Ioannis Patras · 2025
Read AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data by Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Ioannis Patras (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05665.txt.
Introduction
Zengqun Zhao and four co-authors fair-tune, in 2025 work, a biased face model with made-up data, needing no group tags. Made data can be thin or skewed, and its world gaps the real one. Their path grows the made set with prompts from a context-wise language model, for breadth and bias sway.
Then a pick-led tune moves just the knobs that feel bias most and world shift least. On two face sets the path lifted fairness while use held, past full and part tunes alike. Group tags, oft dear to buy, are not asked for. Made data, well led, can right a model.
Catalog record
| CREATOR | Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Ioannis Patras |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Ioannis Patras Abstract Recent advances in generative models have sparked research on improving model fairness with AI-generated data. However, existing methods often face limitations in the diversity and quality of synthetic data, leading to compromised fairness and overall model accuracy. Moreover, many approaches rely on the availability of demographic group labels, which are often costly to annotate. This paper proposes AIM-Fair, aiming to overcome these limitations and harness the potential of cutting-edge generative models in promoting algorithmic fairness. We investigate a fine-tuning paradigm starting from a biased model initially trained on real-world data without demographic annotations. This model is then fine-tuned using unbiased synthetic data generated by a state-of-the-art diffusion model to improve its fairness. Two key challenges are identified in this fine-tuning paradigm, 1) the low quality of synthetic data, which can still happen even with advanced generative models, and 2) the domain and bias gap between real and synthetic data. To address the limitation of synthetic data quality, we propose Contextual Synthetic Data Generation (CSDG) to generate data using a text-to-image diffusion model (T2I) with prompts generated by a context-aware LLM, ensuring both data diversity and control of bias in synthetic data. To resolve domain and bias shifts, we introduce a novel selective fine-tuning scheme in which only model parameters more sensitive to bias and less sensitive to domain shift are updated. Experiments on CelebA and UTKFace datasets show that our AIM-Fair improves model fairness while maintaining utility, outperforming both fully and partially fine-tuned approaches to model fairness. The code is available at https://github.com/zengqunzhao/AIM-Fair . † † Corresponding author: Ziquan Liu {[email protected]}. Figure 1 : Facial attributes classification (FAC) on the CelebA dataset based on different training strategies, in which Smiling is the target attribute and Gender is the protected attribute. This shows different learning strategies result in variable model utility (Overall Accuracy) vs. model fairness (Worst Group Accuracy and Equalized Odds) on demographic groups. A model trained solely on real data if biased exhibits high accuracy but poor fairness scores. Conversely, models trained on balanced synthetic data show better fairness but poorer accuracy due to a “domain gap” between the real and synthetic data, and a lack of selective model fine-tuning when synthetic data is deployed. Strategies to repair imbalances in real data [ 56 , 9 ] or to supplement real data with synthetic data [ 47 ] marginally increase accuracy but do little to improve fairness. Our method for selective fine-tuning of a pre-trained (biased) model with synthetic data not only preserves model accuracy but also substantially improves fairness, outperforming fully fine-tuning (FFT) in both model utility and fairness. 1 Introduction Recent research has raised significant concerns about fairness and bias in machine learning models [ 30 ] . These models often demonstrate varying performance across different demographic groups, leading to unfair outcomes. To mitigate the spurious correlation caused by learning from imbalanced (biased) data, regularizations were used as optimization objectives [ 1 , 12 , 10 , 23 ] . Methods like distributionally robust optimization (DRO) [ 12 ] optimizes the worst-case performance, while invariant risk minimization (IRM) [ 1 ] learns unbiased representations with invariance to different environments. Influenced by the success of representation learning [ 51 ] , attempts were made to learn a fair feature representation invariant to protected facial attributes [ 6 , 29 , 34 , 54 ] . However, these methods rely on demographic group labelling, often unavailable in practice [ 2 , 4 ] . More recent advances in generative models have utilized generative augmentation for a more balanced training data distribution [ 59 , 9 , 8 , 17 , 40 , 36 ] . These techniques primarily focus on image editing, either on real data [ 40 , 36 ] or synthetic data [ 9 ] , to create less biased training data. Yet, these methods still require additional group annotations of the data they edit. To address this limitation, DiGA [ 59 ] proposed to create an unbiased training set by editing spurious facial attributes to a random degree while keeping a target facial attribute unchanged without knowing each sample’s group label. However, DiGA increases training data size multiple times, leading to substantially higher training costs, and the quality of generated data is limited due to the unknown group labels. Figure 2 : The effects of selective fine-tuning by layer-wise freezing from facial attribute classifications on the CelebA dataset, with Smiling as the target attribute and Male as the protected attribute: When only the fully connected (FC) layer is frozen the model shows improved worst demographic group accuracy and reduced equalized odds, i.e. more fair, but sacrifices some utility (overall accuracy). This indicates increased cross-domain generalisability (real and synthetic). Conversely, freezing only block 2 while fine-tuning the remaining parameters results in high overall accuracy but poorer fairness, i.e. further enhanced domain-bias specificity. When only block 1 is frozen, the model not only maintained equalized odds but also increased utility (overall accuracy) and worst group accuracies. Recent advancements in text-to-image generative models showcase impressive data fidelity [ 43 ] , yet their potential for improving fairness through data expansion has not been fully explored by the fairness community. This raises the question: can AI-generated synthetic data play a crucial role in mitigating biases within machine learning models? This research question is particularly important given the successful application of AI-generated data in fine-tuning large language models (LLMs) [ 11 ] , data augmentation [ 64 ] , and long-tail recognition [ 47 ] , alongside some reported limitations of synthetic data effectiveness [ 49 ] . This work presents a comprehensive empirical investigation into whether fine-tuning on high-quality, balanced generative data from a contemporary text-to-image model can counteract model biases caused by training on imbalanced real data. We identify two key challenges in bias-counter fine-tuning with synthetic data: (1) A data-related challenge arising from linguistic ambiguity of the textual prompt and/or model misrepresentation [ 57 , 48 ] , which results in low-quality and low-diversity generated data. (2) A model learning challenge caused by both a domain shift (synthetic vs. real) and a bias shift (unbiased vs. biased) between the real and the synthetic data. Fine-tuning blindly on the synthetic will result in a model with decreased utility. To address the data quality issue, we propose Contextual Synthetic Data Generation (CSDG). Existing Latent Diffusion Models (LDMs) [ 36 , 3 , 41 , 45 ] often use vision-language models, such as CLIP, for cross-modality alignment. To yield more diverse and better fine-grained image generation, we formulate a contextual synthetic data generation strategy that uses more detailed text descriptions from expansive linguistic expressions of LLMs to condition an LDM with richer, context-driven text. As a result, the generated images cover more scenarios and provide more details, enhancing diversity and mitigating bias in real data. In contrast to other methods that edit or manipulate real data samples, a key strength of our method is that it does not require annotating the Protected Group Attributes of real data. The model learning challenge is due to the existence of