Bias at the End of the Score
Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky · 2026
Read Bias at the End of the Score by Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13305.txt.
Introduction
Magid and six colleagues audit reward models used in text-to-image generation for demographic bias. Reward models score outputs, filter datasets, guide training, and screen finished images, yet their fairness as scoring tools was largely untested. The audit finds these models encode demographic bias, so reward-guided tuning tends to sexualize female subjects, reinforce gender and racial stereotypes, and shrink demographic diversity. The authors call for better data collection and training.
Catalog record
| CREATOR | Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59152 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13305
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bias at the End of the Score Salma Abdel Magid, Grace Guo, Esin Tureci, Amaya Dharmasiri, Vikram V. Ramaswamy, Hanspeter Pfister, Olga Russakovsky Abstract Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large-scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects 1 1 1 Throughout this work, we use terms like “female image subject” to reference perceived demographic information of the person pictured within images as we attempt to quantitatively measure stereotypical harms and biases based on perceived identity of synthetic image subjects. We acknowledge that this referring method to sensitive self-identified attributes simplifies complex identities into discrete categories, and thus might not capture nuances of lived experience. See § A for more discussion. , reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring. 1 Introduction Recent advances in text-to-image generative models are driven not only by advancements in diffusion model architecture and optimization methods 28 ; 46 ; 49 ; 52 , but also through use of reward models (RMs): scoring functions trained to measure alignment, fidelity, aesthetics, and human preference, effectively distilling these complex criteria into a single score which represents overall image quality. Reward models are then used for a range of purposes including evaluating the generative model outputs ( text-image alignment ) 33 ; 39 , filtering the generated images 12 ; 29 ; 53 , and finetuning generative model outputs 11 ; 15 ; 7 ; 14 ; 32 . Figure 1 : Reward models, e.g. PickScore 33 , ImageReward 60 , and HPS 58 are trained to estimate overall image quality. When used to optimize the initial noise vector \epsilon^{0} , they can exhibit unintended effects including hypersexualization and change of large structural aspects such as perceived demographics of subjects. We study the such unintended effects of using reward models in the T2I pipeline. Blurred region: feminine-presenting image subject wearing a gray bra and underwear. *See footnote 1 . There are many potential sources that may inject bias into the training of RMs, including generative model and prompt dataset biases that impact the data distribution, human preferences that impact rankings of images, and inductive biases of the model architecture and training procedure that impact calibration of estimated scores. With the increased use of synthetic images for downstream applications, any biases exhibited by RMs have the potential to be compounded exponentially, making it critical to study these further. Given the critical harms associated with demographic bias in AI, we specifically study RM robustness in this area. This targeted investigation may simultaneously offer insights into the nature and origins of broader robustness issues of RMs. True to how RMs are used within the community, we examine biases present with RMs when used for both fine-tuning and evaluations of generative models. First , to illustrate the extent of implications of robustness issues in score measures, we use the ReNO 14 framework, where RMs are used to optimize generative model outputs. We evaluate the impact of score biases on the optimized images and quantify the tendency of demographic attributes to be altered and the emergence of hypersexualization during the optimization. Second , we examine five widely-used RMs across three counterfactual datasets under different fairness and safety criteria for potential disparities in reward scores between gender and/or racial demographics. In addition, we evaluate correlations of reward scores with real-world probability densities and harmful stereotypes. Our findings are: • Reward-model optimization amplifies hypersexualization. Using the ReNO framework, we observe that across all reward and base models evaluated, PickScore 33 produces the largest increase in both NSFW classification rate and skin exposure (SE). Female subjects are disproportionately affected: PickScore increases the NSFW rate by an average of 19\% for female subjects, compared to 7\% for male subjects ( 2.7\times larger increase). • Reward-guided optimization causes demographic convergence. When optimizing image outputs using ReNO with reward models, HPS 58 and ImageReward 60 cause diverse image subjects to converge to White image subjects. ImageReward and HPS show the strongest effect, causing more than 80% of images of Black subjects to be classified as White after optimization. When optimizing with ImageReward and CLIP 48 , images labeled with female also reclassify to male more than 39% and 26% of the time, respectively. • Reward models exhibit demographic scoring biases. When images differ primarily in race or gender presentation, each reward model consistently favors certain demographics. HPS and ImageReward generally rank images of White image subjects highest. VQAScore 39 ranks images of White subjects higher than those of Black subjects when described with positive prompts, and conversely ranks images of Black subjects higher than those of White subjects when described with negative prompts. CLIP generally ranks photos of Black subjects last relative to other racial groups. • Reward models encode real-world frequency priors. When images are paired with prompts containing occupations, reward scores correlate with the share of female employment in each category as reported by the U.S. Bureau of Labor Statistics 47 . This suggests that beyond simply evaluating image quality, RMs assign higher scores to images that conform to the dominant demographic features learned from their design and training procedure. Our findings highlight the shortcomings of current RMs, question their reliability as good measures of image quality, and underscore the need for alternative data collection, training, and optimization procedures for RMs that can result in more robust score metrics. 2 Related Works 2.1 RMs for T2I Models Text and Image Alignment. A number of RMs have been developed to assess the degree of alignment of images with text descriptions. CLIPScore 27 proposes using the CLIP model 48 itself as a RM for text and image alignment. CycleReward 3 uses cycle-consistency as a supervisory signal in order to train a RM for text and image alignment. CycleReward is trained on a generated dataset consisting of comparison pairs scored based on cycle-consistency between many captions to a single image. VQAScore 39 uses a vision-question-answering model to compute the probability of a “Yes” answer to the question“Does this figure show ‘text’?” to produce an alignment score. Aesthetics, Quality, and Style. Aesthetic quality of images and general image quality are two other functions considered for training RMs. LAION Aesthetic predictor 53 is a linear estimator trained on top of CLIP and scores aesthetic quality of images. Importantly, LAION Aesthetic evaluates the image aesthetics independent of the