Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Changyao Tian, Shengyuan Ding, Yuhang Zang, Junchi Yan, Xue Yang · 2026

Read Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation by Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Changyao Tian, Shengyuan Ding, Yuhang Zang, Junchi Yan, Xue Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12247.txt.

Introduction

Zhao and co-authors build FIRM, a frame for reward models that grade image edits and text-to-image output. Current critics often hallucinate and hand out noisy scores, which misleads RL training. Tailored data pipelines score edits on execution and consistency, and generation on how well instructions are followed. Two large datasets train a specialist reward model for each job. A Base-and-Bonus plan then feeds these critics into the RL pipeline. The end models beat general ones on fidelity in broad tests.

Catalog record

CREATORXiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Changyao Tian, Shengyuan Ding, Yuhang Zang, Junchi Yan, Xue Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54437
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation Xiangyu Zhao, Peiyuan Zhang, Junming Lin, Tianhao Liang, Yuchen Duan, Changyao Tian, Shengyuan Ding, Yuhang Zang, Junchi Yan, Xue Yang Abstract Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM ( F aithful I mage R eward M odeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing methods. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel ”Base-and-Bonus” reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code will be publicly available. \setlogoheight 14mm \setlogospacing 5mm \setsjtublue \settitlerulethickness 3pt \abstractboxon \setabstractframecolor gray \setabstractbgcolor gray!10 \setlogotolineshift 5mm \settoprulethickness 2.5pt \setbottomrulethickness 1.5pt 1]Shanghai Jiao Tong University 2]Wuhan University 3]BUPT 4]CUHK 5]Fudan University 6]Shanghai AI Laboratory \contribution [*]Equal contribution \contribution [†]Corresponding Author \checkdata [Project Page] https://firm-reward.github.io/ \checkdata [Code] https://github.com/VisionXLab/FIRM-Reward \checkdata [Hugging Face] https://huggingface.co/collections/VisionXLab/firm-reward 1 Introduction The rapid advancement of diffusion models and autoregressive models has revolutionized both text-to-image (T2I) generation [ 9 , 2 , 54 , 32 ] and image editing [ 25 , 45 , 55 ] . Recently, Reinforcement Learning (RL) has emerged as a prevailing paradigm, relying heavily on reward models (or ”critics”) to provide optimization signals. In this RL-driven alignment process, the quality of the generated or edited images is fundamentally bound by the accuracy and reliability of the critics. Despite the widespread adoption of RL, current pipelines face a critical bottleneck: the unreliability of the critics. While recent Multimodal Large Language Models (MLLMs) [ 29 , 15 , 1 , 35 , 40 ] have demonstrated impressive general capabilities, they frequently struggle when employed as zero-shot reward models for fine-grained image editing and generation tasks. These models inherently suffer from severe hallucinations, object neglect, and a lack of precise spatial reasoning, leading to unreasonable and noisy reward scores. Figure 1 : Comparison of image editing results across different methods. “w. FIRM-Edit-8B” indicates that FIRM-Edit-8B is adopted as the reward model during RL process. To address these critical limitations, we present FIRM ( F aithful I mage R eward M odeling), a comprehensive framework designed to train robust, task-specific reward models that serve as trustworthy critics for both image editing and generation. FIRM introduces tailored data construction pipelines to synthesize high-quality reward data. For image editing, we observe a crucial counter-intuitive phenomenon: while MLLMs struggle to directly judge whether an edited image perfectly follows instructions while preserving consistency, they excel at identifying the differences between two images. Therefore, our editing pipeline adopts a ”difference-first” approach. We leverage an MLLM to caption the exact visual differences between the source and edited images, and subsequently feed this textual difference to an MLLM to reliably deduce the final scores for both execution and consistency. For T2I generation, we propose a checklist-based prompting strategy: an LLM first extracts key checking points from the user prompt, which are then appended to the MLLM’s scoring prompt. This explicitly guides the MLLM to verify fine-grained details, significantly reducing hallucinations. By utilizing the pipelines and existing open-source datasets, we construct two high-quality reward datasets: FIRM-Edit-370K and FIRM-Gen-293K. Subsequently, we employ these datasets to train our reward models, FIRM-Edit-8B and FIRM-Gen-8B, which are initialized from the Qwen3-VL-8B-Instruct model. Figure 2 : Comparison of T2I generation results across different methods. “w. FIRM-Gen-8B” indicates that FIRM-Gen-8B is adopted as the reward model during RL process. To rigorously validate our critics, we also construct a human-annotated benchmark FIRM-Bench for both generation and editing tasks. By sourcing prompts from diverse existing benchmarks and images from various models while strictly controlling the ground-truth score distribution, we demonstrate that our trained reward models achieve remarkable alignment with human preference, vastly outperforming existing open-source MLLMs. Building upon these trustworthy critics, we further perform RL to optimize generative models. A well-known challenge in RL is that naively maximizing multiple, often competing rewards frequently triggers optimization collapse or severe reward hacking. To mitigate this, we extensively explore ”Base-and-Bonus” reward weighting strategies and identify a synergistic reward formulation termed Consistency-Modulated Execution (CME). This strategy successfully balances the trade-off and prevents reward hacking in image editing. Similarly, for T2I generation, we propose Quality-Modulated Alignment (QMA) to balance instruction following and image quality. Guided by our robust critics and formulations, we successfully train the FIRM-Qwen-Edit and FIRM-SD3.5 models. Extensive experiments demonstrate that our models yield substantial performance gains across both generation and editing paradigms, as illustrated in figure 1 and figure 2 . In summary, the main contributions of this work are four-fold: 1. We propose the FIRM Framework, which consists of two specialized data construction pipelines (FIRM-Gen and FIRM-Edit) to train robust reward models. We design a novel ”difference-first” (MLLM-to-LLM) pipeline for image editing and a checklist-guided approach for T2I generation. These pipelines yield the high-quality reward datasets FIRM-Edit-370K and FIRM-Gen-293K, along with their corresponding reward models, FIRM-Edit-8B and FIRM-Gen-8B. 2. We construct a comprehensive, fully human-annotated benchmark FIRM-Bench for both image editing and generation. Evaluations confirm that our reward models achieve superior alignment with human compared to existing models. 3. We propose a novel ”Base-and-Bonus” reward fusion strategy in RL process, encompassing Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. This formulation effectively mitigates reward hacking and ensures balanced performance across various scenes. 4. Through extensive RL experiments yielding FIRM-Qwen-Edit and FIRM-SD3.5 models, we demonstrate that FIRM reward models and reward strategies lead to substantial and consistent performance enhancements in both faithful image editing and precise image generation, validating the efficacy of our overall approach. 2 Related Works 2.1 Recent development of Image Editing and Generation Text-to-image (T2I) generation has witnessed a fundamental paradigm shift from early adversarial methods like GANs [ 14 ] , and variational approaches like VAEs [ 21 ] , to diffusion-based architectures [ 17 , 28 , 31 ] . Consequently, flow-based models [ 11 , 23 ] have emerged as an efficient alternative, offering accelerated sampling without compromising synthesis quality. Autoregressive models [ 51 , 37 , 48 ] have also gained attention by treating image synthesis as a sequence modeling task. Building upon the capabilities of generation, image editing has rapidly evolved. Initial diffusion-based models [ 16 , 38 ]

Read the complete text