VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation

Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Dan Zhang, Ming Ding, Xiaohan Zhang, Shiyu Huang, Xiaotao Gu, Minlie Huang, Jie Tang, Yuxiao Dong · 2024

Read VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation by Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Dan Zhang, Ming Ding, Xiaohan Zhang, Shiyu Huang, Xiaotao Gu, Minlie Huang, Jie Tang, Yuxiao Dong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-21059.txt.

Description

Abstract Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases.

Catalog record

CREATORJiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Dan Zhang, Ming Ding, Xiaohan Zhang, Shiyu Huang, Xiaotao Gu, Minlie Huang, Jie Tang, Yuxiao Dong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-21059

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Dan Zhang, Ming Ding, Xiaohan Zhang, Shiyu Huang, Xiaotao Gu, Minlie Huang, Jie Tang, Yuxiao Dong Abstract Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. 1 Tsinghua University 2 Z.AI {xjz22, h-y22}@mails.tsinghua.edu.cn, [email protected] Code — https://github.com/THUDM/VisionReward 1 Introduction Figure 1: Illustration of how VisionReward works for evaluation and optimization of visual generation. a) Evaluation : VisionReward performs comprehensive evaluation through dimension-specific binary visual QA testing, producing human-aligned, fine-grained assessment scores. b) Optimization : VisionReward enable better preference optimization, enhancing multiple key aspects. Visual generative models, including text-to-image ( Ding et al. 2021 ; Ramesh et al. 2021 ; Saharia et al. 2022 ; Rombach et al. 2022 ; Betker et al. 2023 ; Podell et al. 2023 ) and text-to-video ( Hong et al. 2022 ; Ho et al. 2022 ; Villegas et al. 2022 ; Zheng et al. 2024 ; Chen et al. 2024a ; Yang et al. 2024 ) generation, have recently experienced rapid developments. Through large-scale pretraining, these models can effectively translate textual descriptions into photorealistic images or temporally coherent videos. To further align them with human preferences, reinforcement learning from human feedback (RLHF) ( Ouyang et al. 2022 ) —initially introduced in large language models—has recently been adapted to visual generation tasks ( Xu et al. 2023 ; He et al. 2024 ) . A key bottleneck in applying RLHF to visual generation lies in developing effective visual reward models. Recent studies ( Xu et al. 2023 ; Kirstain et al. 2023 ; Wu et al. 2023 ) have explored training reward models to predict human visual preferences, enabling automatic evaluation and preference optimization for visual generative models. For evaluation, reward models function as automated metrics that quantitatively measure the alignment between generated outputs and human preference criteria ( Li et al. 2023 ) . For optimization, reward models identify reliable directions for improving visual generation models. Essentially, they can provide feedback in reinforcement learning or generate preference pairs, thus reducing dependence on human annotation ( Black et al. 2023 ; Fan et al. 2023 ; Clark et al. 2023 ) . Despite recent progress in reward models (RMs) for visual generation, two primary challenges remain: First, lack of interpretability and risk of unexpected bias. Current RMs for visual generation often suffer from limited interpretability. These models inherently involve complex trade-offs among multiple factors, yet their scoring mechanisms lack transparency regarding how such trade-offs are performed. This opacity raises concerns about potential unexpected biases . Though multimodal LLMs like Gemini ( Team et al. 2024 ) and GPT-4o ( Achiam et al. 2023 ) enhance interpretability through explainable rating rationales, their general-purpose architectures usually underperform specialized black-box models in fine-grained assessments ( Chen et al. 2024c ) . This raises a key dilemma: how to design preference prediction method to be interpretable while maintaining accuracy. Second, lack of effective reward models for video generation. The rapid development of text-to-video generative models has intensified the demand for video reward models. Although image reward models can assess individual frame quality, their frame-level nature inherently neglects essential temporal dependencies in video sequences. While VideoScore ( He et al. 2024 ) has pioneered direct video evaluation through learnable metrics, it still suffers from limitations such as insufficient accuracy in preference prediction and optimization in video generation. Contribution. To address these challenges, we propose a general framework VisionReward to build accurate reward models for both image and video generation. VisionReward is trained with two steps: fine-grained visual assessment and interpretable preference learning . First, to capture human visual preferences, we identify nine major dimensions and decouple preferences into 64 fine-grained questions. Second, to ensure interpretable preference learning, we propose to use the classical linear weighting mechanism on the question outcomes. It enables intuitive visualization of each question’s impact. To apply it as a reward model for visual generation models, we propose a multi-dimensional consistent strategy during preference optimization. The goal of this strategy is to mitigate unintended and unquantifiable biases. Specifically, a pair of visual samples is used for preference optimization (e.g., DPO ( Wallace et al. 2024 ) ) only if one sample is consistently preferred over the other across all dimensions. To summarize, we present VisionReward as a general framework for visual preference learning. Empirically, we show that VisionReward makes the following contributions: • We design fine-grained multi-dimensional preference annotation and build the most fine-grained dataset which contain 81K samples and 5M binary annotation, which enable the training of VisionReward. • For visual preference prediction, VisionReward achieves state-of-the-art performance across multiple benchmarks, while maintaining interpretability via hierarchical diagnostic QA and explicit linear weighting. For instance, VisionReward outperforms VideoScore ( He et al. 2024 ) by 17.2% in accuracy on preference prediction. • For visual generation, VisionReward can serve as an effective reward model for preference optimization (e.g., DPO), significantly enhancing the text-to-image and text-to-video models. For example, video generation models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. Dataset Image Video #Samples #Dimensions #Fine-Grained #Annotation ImageReward ( Xu et al. 2023 ) ✓ 9K - - 0.1M Pick-a-Pic ( Kirstain et al. 2023 ) ✓ 38K - - 0.6M HPDv2 ( Wu et al. 2023 ) ✓ 430K - - 0.8M RichHF-18K ( Liang et al. 2024 ) ✓ 18K 4 - 0.1M MPS ( Zhang et al. 2024b ) ✓ 608K 4 - 2.4M VideoScore ( He et al. 2024 ) ✓ 38K 5 - 0.2M VisionReward (Ours) ✓ ✓ 81K 18–20 61–64 5.0M Table 1: Comparison of dataset of VisionReward and other datasets. Figure 2: Illustration of fine-grained multi-dimensional design. (Left) For image: 5 dimensions, 18 sub-dimensions, and 61 binary questions. (Right) For video: 9 dimensions, 20 sub-dimensions, and 64 binary questions. Figure 3: Overall framework of VisionReward. 1) Fine-grained Visual Assessment: fine-tune multimodal LLM to perform binary visual question-answering through hierarchical dimensions. 2) Interpretable Preference Learning: utilize visual QA outputs to predict preferences through linear weighted summation. 3) Multi-Dimensional Preference Optimization: optimization strategy across multiple dimensions. 2 Related Work Reinforcement Learning from Human

Read the complete text