FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

Mahesh Bhosale, Abdul Wasi, Shantam Srivastava, Shifa Latif, Tianyu Luan, Mingchen Gao, David Doermann, Xuan Gong · 2026

Read FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants by Mahesh Bhosale, Abdul Wasi, Shantam Srivastava, Shifa Latif, Tianyu Luan, Mingchen Gao, David Doermann, Xuan Gong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26008.txt.

Introduction

Bhosale and co-authors present FairLLaVA, a fine-tuning method that curbs demographic bias in medical vision-language models. Such models can write uneven diagnostic text for different patient groups. FairLLaVA cuts the mutual information between target attributes in the model's representations. This pushes the representations to ignore demographic traits. The method works as a light plug-in with low-rank adapters. It fits many model designs. On chest X-ray reports and skin image questions, group gaps shrink, the authors report. Clinical scores and text quality both improve.

Catalog record

CREATORMahesh Bhosale, Abdul Wasi, Shantam Srivastava, Shifa Latif, Tianyu Luan, Mingchen Gao, David Doermann, Xuan Gong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66892
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26008

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants Mahesh Bhosale, Abdul Wasi, Shantam Srivastava, Shifa Latif, Tianyu Luan, Mingchen Gao, David Doermann, Xuan Gong Abstract While powerful in image-conditioned generation, multimodal large language models (MLLMs) can display uneven performance across demographic groups, highlighting fairness risks. In safety-critical clinical settings, such disparities risk producing unequal diagnostic narratives and eroding trust in AI-assisted decision-making. While fairness has been studied extensively in vision-only and language-only models, its impact on MLLMs remains largely underexplored. To address these biases, we introduce FairLLaVA, a parameter-efficient fine-tuning method that mitigates group disparities in visual instruction tuning without compromising overall performance. By minimizing the mutual information between target attributes, FairLLaVA regularizes the model’s representations to be demographic-invariant. The method can be incorporated as a lightweight plug-in, maintaining efficiency with low-rank adapter fine-tuning, and provides an architecture-agnostic approach to fair visual instruction following. Extensive experiments on large-scale chest radiology report generation and dermoscopy visual question answering benchmarks show that FairLLaVA consistently reduces inter-group disparities while improving both equity-scaled clinical performance and natural language generation quality across diverse medical imaging modalities. Code can be accessed at https://github.com/bhosalems/FairLLaVA . 1 Introduction Large language models (LLMs) have emerged as general-purpose engines for reasoning and text generation across many domains, driving a wave of foundation model capabilities at unprecedented scale ( 4 ; 39 ; 1 ) . Multi-modal large language models (MLLMs) build on these advances by pairing a language backbone with non-text encoders (e.g., images), enabling image-grounded understanding and generation ( 2 ; 51 ; 40 ; 9 ; 7 ; 28 ; 17 ) . Despite these advances, a growing body of work documents disparities and fairness issues in these models ( 49 ; 8 ; 25 ) . In natural-language domains, societal biases can enter when training datasets are not carefully curated or filtered 48 . For example, models often reproduce gender stereotypes in professions, such as more strongly associating “nurse” with women and “engineer” with men 41 ; 48 ; 42 . However, in safety-critical medical settings, sensitive attributes are rarely explicitly mentioned in clinical reports. In particular, fairness considerations in radiology report generation (RRG)-an automated task that translates medical images into expert-quality textual interpretations 53 ; 56 ; 7 -differ fundamentally from pronoun- or sentiment-based notions in the general language domain. Radiology reports seldom contain explicit demographic markers (e.g., “he/she”), yet models can still exploit latent sensitive information embedded in medical images and acquisition pipelines 16 ; 30 . This renders standard word-level (polarizing-term) probes 14 ; 3 ; 29 ; 48 based on demographic tokens largely inapplicable. Figure 1 : FairLLaVA reduces performance disparities . LLaVA hidden states contain demographic shortcuts (non-zero Mutual Information (MI) between hidden states and demographic attributes) that lead to lower performance for “Female”. FairLLaVA minimizes this MI promoting demographic-invariant representation learning, therefore reducing the performance gap. Standard fairness techniques, such as frequency-based resampling 18 and reweighting 26 ; 13 , aim to mitigate biased performance gaps by treating disparities as mere count imbalances. However, in MLLMs, the gaps are driven by intersectional, cross-attribute dependencies that this assumption overlooks. Ranking-based methods 8 require repeated inference to score and sort the entire training set, incurring substantial computational overhead. Adversarial-classifier methods 41 add a separate pretrained discriminator, which, in our experiments, shows that it triggers catastrophic forgetting of clinical knowledge, evidenced by degraded overall performance (detailed in supplementary). Standard fairness evaluation in discriminative tasks cannot be directly applied to MLLMs either. Typically, fairness can be quantified directly on calibrated scores or binary decisions (e.g., TPR/FPR gaps, demographic parity difference, equalized odds, calibration error), which makes disparity attribution and mitigation comparatively simpler 22 ; 31 ; 47 ; 15 ; 37 ; 45 ; 33 . By contrast, open-ended text generation allows many valid phrasings of the same finding, and clinically important omissions can be just as harmful as incorrect mentions. Fairness, therefore, must be evaluated over the entire generated text, using semantic-level metrics and representations, rather than a single calibrated score or label. Fairness challenges in Medical MLLMs across demographic groups have been documented, with systematic evaluations revealing performance gaps by age and additional disparities by sex and race 49 . Notably, these disparities cannot be explained by sample counts alone: on MIMIC-CXR 24 ; 23 , several MLLMs perform worse for patients labeled as “White” despite this being the largest cohort (detailed in the supplementary material). These gaps can stem from compounding factors: (i) medical images leak sensitive attributes , models can predict self-reported race from radiology images with high AUC, even under corruptions, indicating systematically encoded demographic signal 16 ; (ii) acquisition factors (e.g., projection, device/site mix) correlate with demographics and mediate both predictability and downstream performance 30 ; and (iii) imbalance and prevalence shifts across groups can amplify error gaps 49 . Despite the above-mentioned concerns regarding gaps in fairness disparities across demographic groups 49 ; 25 , fairness and bias mitigation in MLLMs remain underexplored, threatening their trustworthy adoption in healthcare. To this point, we propose FairLLaVA , a fairness-aware, parameter-efficient finetuning strategy applied during visual instruction tuning. As shown in Fig. 1 , FairLLaVA adds a lightweight, architecture-agnostic mutual information regularizer that reduces dependence between multi-modal hidden states and sensitive attributes (age, sex, race), steering the model toward diagnostic cues instead of demographic shortcuts. FairLLaVA strikes a balance by reducing fairness gaps while preserving overall performance unlike traditional reweighting or resampling, which often improve one subgroup at the expense of others. Concretely, we jointly optimize standard instruction following with demographic-invariant representation learning via mutual information minimization, making the method plug-and-play with parameter-efficient finetuning ( Figs. 2 and 1 ). Our contributions are summarized as follows- • We propose FairLLaVA, a debiasing fine-tuning strategy that uses a mutual-information regularizer to remove demographic shortcuts from an MLLM’s hidden representations. Designed for models with enormous parameter counts and prohibitive fine-tuning costs, FairLLaVA is architecture-agnostic and operates with minimal intervention to the base model, incurring only modest computational overhead. • We extend the Equity Scaled Metric (ES-M) to generative models, allowing seamless integration with any language-based evaluation metric, enabling a balanced assessment of both overall performance and disparities across key demographic groups. • Extensive experiments on two large-scale chest radiology report–generation datasets and one dermoscopy QA dataset show that our method outperforms existing medical MLLMs (LLaVA-Rad, MedGemma, Chexagent), general-purpose MLLMs (LLaVA, Qwen, DeepSeek), and prior fairness-oriented baselines in equity scaled metrics. 2 Related Work Classical fairness methods can be divided into frequency-based and feature-based. Frequency-based debiasing methods 26 ; 13 ; 18 attribute fairness gaps to class imbalance and mitigate them via loss re-weighting or oversampling; however, this premise does not need to hold in many settings. Feature-based approaches seek to mitigate biases encoded within the model’s internal representations. 41 addressing the fairness issue uses a pretrained adversarial classifiers to de-bias the pretrained backbone; however, it can lead to catastrophic forgetting if hyper-parameters are not tuned carefully. 31 regularizes image-text similarity distributions to align subgroup and overall scores. Both methods incur extra intervention via additional pretraining or higher computational cost. FairLLaVA, by contrast, handles MLLMs’ large parameter counts and expensive fine-tuning while balancing fairness gaps and overall performance. Fairness in LLMs typically focuses

Read the complete text