SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models

Jiesong Lian, Ruizhe Zhong, Zixiang Zhou, Xiaoyue Mi, Yixue Hao, Yuan Zhou, Qinglin Lu, Long Hu, Junchi Yan · 2025

Read SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models by Jiesong Lian, Ruizhe Zhong, Zixiang Zhou, Xiaoyue Mi, Yixue Hao, Yuan Zhou, Qinglin Lu, Long Hu, Junchi Yan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22170.txt.

Introduction

Jiesong Lian and eight co-authors tackle reward models, the scorers that teach video generation systems to match human taste. Current scorers learn from pairwise labels collected inside a single prompt, which introduces noise. Their designs are also under-explored, and generators can learn to game them, a failure called reward hacking. The team's system, SoliReward, attacks all three problems at once.

SoliReward collects cheaper, cleaner data through single-item yes-or-no annotations, then builds preference pairs across different prompts. A hierarchical progressive query attention mechanism strengthens feature gathering inside the model. A modified loss handles wins and ties explicitly and spreads scores more evenly among good samples. Benchmarks covering physical plausibility, subject deformity, and semantic alignment show gains in scorer quality and in downstream video training.

Catalog record

CREATORJiesong Lian, Ruizhe Zhong, Zixiang Zhou, Xiaoyue Mi, Yixue Hao, Yuan Zhou, Qinglin Lu, Long Hu, Junchi Yan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22170

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models Jiesong Lian, Ruizhe Zhong, Zixiang Zhou, Xiaoyue Mi, Yixue Hao, Yuan Zhou, Qinglin Lu, Long Hu, Junchi Yan Abstract Post-training alignment of video generation models with human preferences is a critical goal. Developing effective Reward Models (RMs) for this process faces significant methodological hurdles. Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise. Concurrently, the architectural design of VLM-based RMs, particularly their output mechanisms, remains underexplored. Furthermore, RM is susceptible to reward hacking in post-training. To mitigate these limitations, we propose SoliReward, a systematic framework for video RM training. Our framework first sources high-quality, cost-efficient data via single-item binary annotations, then constructs preference pairs using a cross-prompt pairing strategy. Architecturally, we employ a Hierarchical Progressive Query Attention mechanism to enhance feature aggregation. Finally, we introduce a modified BT loss that explicitly accommodates win-tie scenarios. This approach regularizes the RM’s score distribution for positive samples, providing more nuanced preference signals to alleviate over-focus on a small number of top-scoring samples. Our approach is validated on benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, demonstrating improvements in direct RM evaluation metrics and in the efficacy of post-training on video generation models. Code and benchmark will be publicly available. 1 1 footnotetext: \dagger Work done during internship at Tencent Hunyuan. 2 2 footnotetext: * Equal contribution. [email protected], [email protected] 3 3 footnotetext: \ddagger Project leader. 4 4 footnotetext: \lx@sectionsign Corresponding author. [email protected], [email protected] 1 Introduction Recent advancements in video generation models, such as Sora 2 21 , Veo 3 8 , and Seedance 7 , have exhibited remarkable capabilities in synthesizing high-fidelity, temporally coherent visual content. Behind these achievements, post-training alignment techniques, analogous to Reinforcement Learning from Human Feedback (RLHF) 22 in Large Language Models (LLMs), are becoming a key driving force for enhancing model performance and correcting physical implausibility, visual artifacts and instruction following. Currently, methods such as DanceGRPO 36 and LongChat-Video 26 have begun to utilize flow-based GRPO 16 methodologies to align video models. However, the effectiveness of these alignment techniques relies heavily on the capabilities of their core component: the Reward Model (RM). RM is trained to quantify human preferences. However, constructing an RM that can accurately capture complex video qualities faces critical issues. These include data annotation noise and inconsistency, the vulnerability to reward hacking, and the underexplored design space for VLM-based RM architectures. On the data front, the dominant paradigm, pairwise preference learning 17 , trains on video pairs (y_{w},y_{l}) where y_{w} is preferred over y_{l} , typically using the Bradley-Terry (BT) loss 3 . While effective, this “in-prompt” annotation (pairing videos from the same prompt) struggles with pairs of comparable quality, where annotator ambiguity introduces substantial label noise. An alternative, point-wise scoring (e.g., using a five-level scale) 14 , avoids direct comparison but introduces high inter-annotator disagreement due to the subjective and ambiguous nature of intermediate scores. This noise in the preference data fundamentally degrades the RM’s ability to accurately rank video quality. Beyond data noise, RM is highly susceptible to reward hacking in post-training, where the learned proxy objective deviates from intended human preferences 24 . Finally, the architecture used to extract this scalar reward is often insufficiently expressive, leading to a collapse of the reward where scores cluster together. Common methods, such as using the last token’s embedding 38 , relying on the output of a dedicated special token 17 , or using the probability of a “yes/no” token 32 , may fail to capture the full spectrum of information, from low-level artifacts to high-level semantic alignment, encoded across the model’s layers. To mitigate these challenges, we propose a systematic framework towards robust video reward model training, including data annotation, training strategy and model architecture. We reorient data collection, shifting from complex relative comparisons or ambiguous multi-level scores towards simple, low-noise single-item binary annotations (Pass / Fail) targeting specific quality dimensions. To leverage the ranking ability of BT loss with these binary labels, we introduce a cross-prompt pairing strategy , where videos within one pair could belong to different prompts. This strategy generates a large-scale, high-signal preference dataset from simple binary labels, circumventing the noise inherent in ambiguous in-prompt comparisons. Furthermore, we propose the BT with Win-Tie (BT-WT) loss to alleviate the critical issue of reward hacking 24 . By augmenting standard win-lose pairs with ‘win-tie’ pairs (e.g., pairing two ‘Pass’ samples), it explicitly penalizes score variance within the positive set, forcing the RM to map all high-quality samples to a compact manifold and thus mitigating reward spikes that lead to hacking. Finally, to address the architectural limitations identified in extracting a reward signal from a VLM 1 backbone, we move beyond simplistic pooling strategies. We propose the Hierarchical Progressive Query Attention (HPQA) , a novel architecture that explicitly aggregates and refines features from multiple transformer layers, fusing low-level visual fidelity with high-level semantic understanding for a more robust reward signal. Our contributions are as follows: • On the data level, we design a highly efficient and low-noise pipeline combining single-item binary annotation with a novel cross-prompt pairing strategy. • For the training objective, we introduce BT-WT loss to enhance data efficiency and demonstrably mitigate reward hacking by balancing positive sample distributions. • Architecturally, we propose the HPQA , a new reward model architecture that fuses multi-layer features for a more robust scalar reward. • Finally, we establish new benchmarks for evaluating video generation quality based on subject deformity and physical plausibility , and demonstrate through extensive experiments that our combined framework surpasses existing RM training and architectural baselines. Figure 1 : Pipeline of SoliReward, our framework for data annotation and training of video reward models. (a) We introduce a single-item binary annotation method, coupled with a cross-prompt pairing strategy, to mitigate annotation noise. Furthermore, to alleviate reward hacking, we propose the Bradley-Terry with Win-Tie (BT-WT) loss. (b/c) We propose a novel VLM-based Reward Model (VLM-RM) architecture, featuring a Hierarchical Progressive Query Attention (HPQA) adapter. This adapter progressively aggregates multi-level representations from the VLM backbone to compute a robust reward score. 2 Related Works 2.1 Data Construction for Reward Model Training The efficacy of a Reward Model is fundamentally constrained by its training data. Existing annotation paradigms generally fall into three categories: pairwise comparisons, point-wise scoring, and binary annotation. Pairwise Preference Annotation. The dominant paradigm in RLHF 22 and recent video RMs 11 ; 20 involves querying preferences between two generations from the same prompt. This naturally aligns with BT objectives 3 , which excel at modeling relative utility. However, in-prompt annotation suffers from severe decisional ambiguity when candidates exhibit similar quality levels. It forces subjective tie-breaking, injecting noticeable noise. While some efforts introduce “tie” options 17 , defining standardized thresholds for neutrality remains difficult and does not fully resolve the underlying ambiguity. Point-wise Scoring Annotation. Alternatively, discrete multi-level Likert scales (e.g., VideoScore 9 ) could introduce high-variance labeling noise, particularly at subjective decision boundaries. As human preference is intrinsically continuous 37 , forcing rigid discrete labels onto borderline samples (e.g., ‘3’ vs. ‘4’ for similar inputs) discards uncertainty and creates artificial quality gaps. This limitation is underscored by VideoScore 9 , which reports critically low Inter-Annotator

Read the complete text