Personalized Reward Modeling for Text-to-Image Generation

Jeongeun Lee, Ryang Heo, Dongha Lee · 2025

Read Personalized Reward Modeling for Text-to-Image Generation by Jeongeun Lee, Ryang Heo, Dongha Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19458.txt.

Introduction

Jeongeun Lee, Ryang Heo, and Dongha Lee build a reward model that judges text-to-image output by one user's taste at a time. General scoring tools miss how personal visual preferences are, since people read the same prompt in different ways. Their model, PIGReward, invents user-specific judging dimensions on the fly.

With little reference data, a self-bootstrapping step reasons over the few examples to build a rich picture of the user's taste, needing no per-user training. Step-by-step reasoning then scores each image and gives feedback that rewrites prompts toward the user's intent. A per-user benchmark, PIGBench, supports tests showing gains in accuracy and clarity.

Catalog record

CREATORJeongeun Lee, Ryang Heo, Dongha Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19458

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Personalized Reward Modeling for Text-to-Image Generation Jeongeun Lee, Ryang Heo, Dongha Lee Abstract Recent text-to-image (T2I) models generate semantically coherent images from textual prompts, yet evaluating how well they align with individual user preferences remains an open challenge. Conventional evaluation methods, general reward functions or similarity-based metrics, fail to capture the diversity and complexity of personal visual tastes. In this work, we present PIGReward, a personalized reward model that dynamically generates user-conditioned evaluation dimensions and assesses images through CoT reasoning. To address the scarcity of user data, PIGReward adopt a self-bootstrapping strategy that reasons over limited reference data to construct rich user contexts, enabling personalization without user-specific training. Beyond evaluation, PIGReward provides personalized feedback that drives user-specific prompt optimization, improving alignment between generated images and individual intent. We further introduce PIGBench, a per-user preference benchmark capturing diverse visual interpretations of shared prompts. Extensive experiments demonstrate that PIGReward surpasses existing methods in both accuracy and interpretability, establishing a scalable and reasoning-based foundation for personalized T2I evaluation and optimization. Taken together, our findings highlight PIGReward as a robust step toward individually aligned T2I generation. 1 Introduction Recent text-to-image (T2I) generative models 34 ; 36 ; 31 ; 29 have shown remarkable performance in generating high-quality images from natural language prompts. Moving beyond generating generic, one-size-fits-all images, following works explores personalization in T2I generation to better align with individual preferences 63 ; 57 ; 8 . These methods typically adopt a history-driven approach 3 ; 4 ; 15 , leveraging user-specific references (e.g., prior image preferences or interactions) to reflect unique visual tastes. Figure 1 : Each image in the pair is evaluated across multiple dimensions per user. Since two users prioritize different criteria, their overall judgments of which image is better differs. However, evaluating personalization remains a significant challenge under conventional T2I evaluation methods. Most widely-used approaches are based on reward functions fine-tuned on large-scale general human preference datasets 17 ; 58 ; 61 . They reduce human preferences to a single scalar score, failing to consider their inherent complexity. This motivates multi-dimensional evaluation 60 ; 70 ; 23 , which introduces a small set of pre-defined dimensions (e.g., Alignment, Detail). More recently, the strong reasoning capabilities of large vision–language models (LVLMs) have inspired their use as generative reward models 77 ; 54 ; 53 , offering interpretable explanations. Yet they still rely on fixed criteria, applying the same criteria uniformly to all inputs. Such static approaches cannot capture diverse perceptual aspects that vary across individuals. As illustrated in Figure 1 , different users attend to different aspects for the same image pair, leading to contrasting judgments–indicating that fixed, global dimensions are inadequate for personalized evaluation. On the other hand, prior personalized T2I works 42 ; 52 ; 62 commonly employ statistical metrics such as CLIPScore 11 , Fréchet Inception Distance (FID) 12 , and Inception Score (IS) 38 to measure similarity between user references and generated outputs. However, such metrics capture only surface-level recurrence, oversimplifying the intricate nature of human aesthetic preferences underlying in personal history. Some methods instead naïvely instruct LVLMs to assess preferences given reference data, but this lacks the in-depth reasoning to model latent preferences. Assessing visual preference requires more than low-level similarity, also accounting for high-level semantic context. Or they conduct human evaluations, which are inherently costly, time-consuming, and difficult to reproduce at scale. These underscore the need for a more effective framework for assessing user-specific alignment in T2I generation. In this work, we propose PIGReward , a LVLM-based personalized reward model that (1) dynamically generates evaluation dimensions from user contexts, and (2) performs visual chain-of-thought (CoT) reasoning to score along those dimensions with explicit rationales. To support such structured reasoning, we construct a 5K CoT-formatted dataset to train PIGReward to learn to adaptively derive dimensions and perform context-aware evaluation across generated criteria, enabling interpretable assessment that captures diverse and complex user preferences. Additionally, since real-world personalization often suffers from scarce user-level data 35 , we introduce a self-bootstrapping strategy 65 that reasons over limited reference data to build rich user contexts. These contexts allow PIGReward to infer user-conditioned dimensions, enabling personalized assessment without user-specific training. Further, we demonstrate the potential of PIGReward beyond evaluation by leveraging its feedback for personalized prompt optimization. Unlike prior works 22 ; 5 that aim to optimize prompts toward general aesthetic quality, PIGReward incorporates individual context and thus with individual context and thereby provides user-conditioned signal that reflect individual visual taste. These signals identify preferred–rejected prompt variants, which are then used to fine-tune the prompt model via direct preference optimization (DPO), enabling user-aligned generation. To evaluate real-world personalization in T2I generation, we additionally construct PIGBench , a user-level image preference dataset. Existing datasets either focus on subtle visual differences 17 ; 58 ; 23 , lack contrastive feedback 3 , or are constrained to limited visual domains 37 . In contrast, PIGBench is built from open-ended, abstract prompts that allow broad visual interpretation. Each instance consists of four images generated from expanded variants of a base prompt, and participants are asked to rank these images according to their own preferences. For realistic personalization settings, we vary the number of instances provided to each participant, resulting in dynamic reference sizes. In total, PIGBench comprises 75 user records, offering a realistic tested to assess how effectively T2I models infer and adapt to diverse human preferences. 2 Related Work Visual Chain-of-Thought Reasoning. LVLMs 71 ; 20 ; 1 ; 51 have demonstrated strong performance across diverse multimodal reasoning tasks 59 ; 62 ; 18 . As CoT reasoning has significantly improved the performance of LLMs through step-by-step reasoning 56 ; 64 ; 16 ; 19 , recent studies 72 ; 69 ; 73 have extended CoT reasoning to the vision–language domain, showing benefits across diverse visual reasoning tasks, including mathematics 10 and robotic control 73 . However, a key challenge in applying CoT to vision–language models lies in the scarcity of high-quality, CoT-annotated datasets. To address this, recent works 69 ; 53 ; 6 build data generation pipeline to collect task-specific CoT-formatted datasets and distill VLMs 51 ; 1 under supervision. Meanwhile, other works have explored LVLMs as generative reward models 21 ; 78 , leveraging explicit rationales 41 ; 54 ; 59 for scalable and interpretable evaluation. Incorporating CoT has been shown to further enhance the reliability and robustness of reward modeling 69 ; 53 . Personalized Text-to-Image Generation. Unlike generic image generation, personalized T2I adapts outputs to user preferences by modeling intent from personal history (e.g., image interactions) 42 ; 52 ; 63 . PIP 3 proposes a personalized prompt rewriting method conditioned on historical inputs. ViPer 37 and PPD 4 extract user features, expressed either as natural-language profiles or as hidden-state embeddings from LVLMs 20 , and directly inject these features into diffusion models 29 . Recent methods expand to multi-turn settings by giving iterative feedbacks 26 ; 48 ; 67 or with natural language instructions through conversation 24 ; 50 . Personalized Reward Modeling. Preference alignment has been studied for large language models (LLMs), particularly through reinforcement learning from human feedback 45 . Subsequent works use pairwise preference data to align models with human interests 28 ; 7 ; 75

Read the complete text