RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu, Fangzhen Lin, Wenhu Chen, [0.5em] HKUST · 2026
Read RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time by Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu, Fangzhen Lin, Wenhu Chen, [0.5em] HKUST (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11626.txt.
Introduction
Wang and five colleagues show reasoning rewards scale visual output at both training and test time. Teaching reward models to write critiques before scoring turns passive judges into tuning tools. Structured reasons give fine RL rewards at training, while a Generate-Critique-Refine loop turns critiques into prompt fixes at test time without retraining. Their Preference-Anchored Rationalization framework builds reasons from preference data by anchored generation, consistency checks, and teaching. RationalRewards (8B) tops open-source preference scores, rivals Gemini-2.5-Pro, and uses 10 to 20 times less training data than baselines.
Catalog record
| CREATOR | Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu, Fangzhen Lin, Wenhu Chen, [0.5em] HKUST |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79166 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu, Fangzhen Lin, Wenhu Chen, [0.5em] HKUST Abstract Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools—improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate–Critique–Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models—competitive with Gemini-2.5-Pro—while using 10–20× less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit. Models and Code are available at Project Page . Figure 1: Train-Time RL and Test-Time PromptTuning (PT) with RationalRewards on text and image-to-image generation benchmarks. (Left) Comparison on image editing benchmarks. RL with RationalRewards outperforms prior open-source generators. Crucially, we find that test-time PT with RationalRewards alone can surpass expensive RL. (Right) Breakdown results on text-to-image benchmark UniGenBench++. 1 introduction As visual generation advances toward photorealistic, instruction-following outputs ( 9 ; 25 ; 42 ; 8 ) , reward models that evaluate these outputs have become the binding constraint on further progress. Yet most reward models remain scalar black boxes: they compress multi-dimensional human judgments—perceptual quality, instruction faithfulness, physical plausibility, text rendering—into a single unexplained number ( 46 ; 45 ; 21 ; 40 ; 12 ) . This discards the structured reasoning underlying human preference, leaving generators to exploit shortcut correlations rather than learn principled evaluation criteria ( 18 ) . This paper asks: can reward models be made to reason —and can their structured critiques not only evaluate but actively improve visual generation? Figure 2: RationalRewards is a reasoning-based reward model that produces structured rationales before assigning scores, enabling dual-space optimization for image generation. (a) As a reward model, it improves RL-based fine-tuning of generators over scalar baselines; (b) as a test-time optimizer, its Generate–Critique–Refine loop matches or surpasses RL-based optimization on multiple benchmarks without parameter updates. We introduce RationalRewards , a reasoning-based reward model that generates structured, multi-dimensional critiques before deriving scores. We argue that this shift from scalar outputs to structured reasoning transforms the reward model from a passive evaluator into a versatile optimization interface for visual generation. By producing explicit reasoning, RationalRewards unlocks optimization in two complementary spaces: • Parameter Space: Multi-dimensional structured rationales provide semantically grounded, dense feedback for reinforcement learning—replacing opaque scalar gradients prone to reward hacking (Fig. 3 ), with explanations of what to improve and why . • Prompt Space: Beyond serving as a reward signal, RationalRewards functions as a post-generation prompt optimizer . It critiques a generated image, identifies concrete deficiencies, and translates them into targeted prompt revisions in a Generate–Critique–Refine loop. Unlike prompt enhancers that rewrite inputs blindly before synthesis ( 38 ) , this approach is post-hoc and reactive, trading test-time compute for improved fidelity without parameter updates ( 30 ; 33 ) . Realizing this vision requires a reward model that produces high-quality structured rationales ( 24 ; 10 ; 54 ; 34 ) , yet human rationale annotations are prohibitively expensive at scale. We observe, however, that pairwise preference data is widely available from online AIGC platforms. Leveraging this, we propose Preference-Anchored Rationalization (PARROT) , a variational training framework that treats rationales as latent variables and derives an evidence lower bound (ELBO) on observed preferences. The terms of this ELBO map directly onto a simple, scalable pipeline: (1) a teacher VLM generates candidate rationales anchored to known preference labels, (2) a consistency filter rejects hallucinations and retains rationales that are genuinely predictive, and (3) a student model is trained to produce rationales without seeing the answer. This tight theory–practice correspondence (Fig. 4 ) converts existing preference datasets into high-quality reasoning supervision using 10–20 \times less data than comparable scalar reward baselines. Figure 3: RL (LoRA) training on Qwen-Image using scalar rewards encounter reward hacking (bottom row): as training reward continues to grow, generation quality starts to degenerate, because black box rewards mislead visual generators with biases. In contrast, RationalRewards (top row) sustains generation quality with stable reward growth. See Fig. 10 and 11 for more details. Key results. Instantiated via PARROT on Qwen3-VL-Instruct-8B backbone, RationalRewards achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro (Table 1 ). As an RL reward, it consistently improves generators beyond scalar baselines across both text-to-image and image editing tasks (Tables 2 – 3 ). Most interestingly, RationalRewards ’s Generate–Critique–Refine loop—requiring no parameter updates—matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured critiques can unlock latent generator capabilities that suboptimal prompts fail to elicit. We envision that RationalRewards empower more than four compelling use cases demonstrated in Fig. 8 . 2 Method We introduce Preference-Anchored Rationalization (PARROT) , a framework that trains reward models to produce explicit, multi-dimensional rationales before scores ( 54 ; 34 ) . Assessment dimensions—text faithfulness, physical/visual quality, text rendering, and (for editing) image faithfulness—follow the taxonomy of 12 , chosen for coverage of the primary failure modes in current generators. Since ground-truth rationales are prohibitively expensive to annotate at scale, we formulate rationales as latent variables inferred from pairwise preference data via a variational objective. The resulting ELBO (Eq. 1 ) decomposes into three terms, each corresponding to a concrete pipeline phase (Fig. 4 ): (1) generate rationales anchored to known preferences, (2) filter for predictive consistency, and (3) distill into a student model. Readers primarily interested in the practical pipeline may consult Fig. 4 and return to the derivation for justification. Takeaway 1: Why Reasoning Rewards Resist Reward Hacking. Scalar reward models are susceptible to reward hacking because they compress evaluation into a single number that can be inflated by exploiting biases without genuine quality improvement. As shown in Fig. 3 , RL with scalar rewards exhibits reward increases while generation quality visibly degrades. RationalRewards mitigates this through an implicit regularization: the model must produce coherent, multi-dimensional reasoning before emitting scores, structurally grounding evaluation in interpretable criteria. Reward inflation without corresponding visual evidence becomes difficult when the model must justify why each dimension merits its score. Empirically, RationalRewards maintains monotonic correspondence between reward and quality throughout training (Fig. 3 , Fig. 9 ). We envision that RationalRewards will facilitate broader and more reliable research on diffusion RL methods. 2.1 Variational Framework: The Hindsight-Foresight Decomposition Let x=(I_{A},I_{B},c) denote a comparison tuple comprising two generated images and a conditioning user request c (which includes text instructions and, for editing tasks, a source image). Let y\in\{A\succ B,B\succ A\} denote the ground-truth human preference. Unlike reward models that model