HP-Edit: A Human-Preference Post-Training Framework for Image Editing

Fan Li, Chonghuinan Wang, Lina Lei, Yuping Qiu, Jiaqi Xu, Jiaxiu Jiang, Xinran Qin, Zhikai Chen, Fenglong Song, Zhixin Wang, Renjing Pei, Wangmeng Zuo · 2026

Read HP-Edit: A Human-Preference Post-Training Framework for Image Editing by Fan Li, Chonghuinan Wang, Lina Lei, Yuping Qiu, Jiaqi Xu, Jiaxiu Jiang, Xinran Qin, Zhikai Chen, Fenglong Song, Zhixin Wang, Renjing Pei, Wangmeng Zuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19406.txt.

Introduction

Li and colleagues propose HP-Edit, a human-preference post-training tool for image editing, with RealPref-50K, a dataset across eight common tasks. Common editing relies on generative diffusion models, and methods like Diffusion-DPO and Flow-GRPO have improved generation quality. Yet applying human-feedback reinforcement learning to diffusion-based editing remains largely unexplored, for lack of scalable preference datasets and tools for diverse needs. HP-Edit uses little human-preference scoring data and a pretrained visual large language model to build preference-aligned editing.

Catalog record

CREATORFan Li, Chonghuinan Wang, Lina Lei, Yuping Qiu, Jiaqi Xu, Jiaxiu Jiang, Xinran Qin, Zhikai Chen, Fenglong Song, Zhixin Wang, Renjing Pei, Wangmeng Zuo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37069
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19406

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HP-Edit: A Human-Preference Post-Training Framework for Image Editing Fan Li, Chonghuinan Wang, Lina Lei, Yuping Qiu, Jiaqi Xu, Jiaxiu Jiang, Xinran Qin, Zhikai Chen, Fenglong Song, Zhixin Wang, Renjing Pei, Wangmeng Zuo Abstract Common image editing tasks typically adopt powerful generative diffusion models as the leading paradigm for real-world content editing. Meanwhile, although reinforcement learning (RL) methods such as Diffusion-DPO and Flow-GRPO have further improved generation quality, efficiently applying Reinforcement Learning from Human Feedback (RLHF) to diffusion-based editing remains largely unexplored, due to a lack of scalable human-preference datasets and frameworks tailored to diverse editing needs. To fill this gap, we propose HP-Edit, a post-training framework for H uman P reference-aligned Edit ing, and introduce RealPref-50K, a real-world dataset across eight common tasks and balancing common object editing. Specifically, HP-Edit leverages a small amount of human-preference scoring data and a pretrained visual large language model (VLM) to develop HP-Scorer—an automatic, human preference-aligned evaluator. We then use HP-Scorer both to efficiently build a scalable preference dataset and to serve as the reward function for post-training the editing model. We also introduce RealPref-Bench, a benchmark for evaluating real-world editing performance. Extensive experiments demonstrate that our approach significantly enhances models such as Qwen-Image-Edit-2509, aligning their outputs more closely with human preference. † † footnotetext: * Equal Contribution, † Project Leader, 🖂 Corresponding Author 1 Introduction Text-to-image (T2I) generation and image-to-image (I2I) editing have become foundational technologies for content creation across industries, ranging from digital design and product marketing to real-world scene customization. Diffusion models have emerged as the de facto standard due to their high-quality, controllable outputs 10 ; 35 ; 6 ; 41 . For I2I editing, state-of-the-art models 15 ; 48 typically build on pretrained T2I backbones via Supervised Fine-Tuning (SFT), leveraging large-scale I2I datasets to acquire editing capabilities. However, two critical limitations plague SFT-based approaches: first, the mixed sources of SFT data (e.g., cartoons, synthetic images) often misalign with real-world human preferences; second, constructing preference-aligned editing datasets requires expensive manual annotation, making scalable alignment impractical. Reinforcement learning (RL) 43 has proven to be highly effective in enhancing the reasoning and alignment capabilities of large language models (LLMs) 7 ; 13 , and recent works such as Diffusion-DPO 45 , Flow-GRPO 22 , and Dance-GRPO 53 have shown promise in improving the quality of T2I generation through post-training. Despite this progress, RL-driven human-preference alignment for I2I editing remains underexplored. Unlike open-ended T2I synthesis, I2I editing demands both task accuracy (e.g., faithfully removing an object) and preference alignment (e.g., natural-looking results). This dual objective calls for frameworks that combine efficient preference data construction—without prohibitive annotation costs—with task-aware reward models tailored to diverse editing sub-tasks. Furthermore, existing editing research lacks a real-world, object-balanced benchmark, hindering accurate evaluation of preference-aligned editing. Our work addresses these gaps with three key contributions. First, we present HP-Edit, a post-training framework for H uman P reference Edit ing that unifies a Visual Large Language Model (VLM)-based scorer, HP-Scorer, an efficient hard-case-focused dataset construction pipeline, and task-aware RL post-training to align models with human preferences while preserving editing accuracy. Second, we introduce RealPref-50K, a real-world-oriented dataset of over 50K cases across eight common sub-tasks—addition, removal, background replacement, object swapping, color change, bokeh (background defocus), relighting, and style transfer—covering common MS-COCO 20 objects (e.g., person, car, cake ) with a balanced distribution to better reflect practical needs. Third, we release RealPref-bench, a benchmark for evaluating preference-aligned editing using real-world images and manually verified preference instructions, enabling rigorous model evaluation. In HP-Edit, we first collect a small number of triples (input image, edited output, instruction) per editing sub-task and annotate each triple with human-preference scores ranging from 0 to 5 (see the supplementary material for the scoring criteria). The proposed HP-Scorer is built on a pretrained VLM (e.g., Qwen2.5-VL 2 ) with a progressively optimized task-aware scoring prompt to approximate human judgments. We then run inference on existing editing datasets using the pretrained editing model and use the VLM-based HP-Scorer to filter hard cases that best capture preference signals, constructing a scalable RL training dataset. Finally, we employ the HP-Scorer as a task-aware reward model and apply RL post-training to the pretrained editing model. Extensive experiments on FLUX.1-Kontext-dev and Qwen-Image-Edit-2509 demonstrate that HP-Edit achieves significant improvements in both human-preference ratings and visual quality, compared to strong pretrained baselines. Figure 2 : The overview of the proposed framework, HP-Edit which consists of three stages: the task-aware HP-Scorer for human preference scoring human preference, the data pipeline of human preference data and the human preference RL post-training. 2 Related Work 2.1 Diffusion Models In recent years, diffusion models 38 ; 10 ; 39 ; 40 ; 31 have achieved remarkable progress in generative tasks. Early diffusion models are driven by a stochastic differential equation (SDE) and optimize denoising-based score matching objectives 10 ; 40 ; 39 , commonly minimizing the MSE between the predicted noise and the ground-truth perturbation. Recently, Rectified Flow 25 and Flow Matching 21 adopt a deterministic probability flow ODE to reduce dependence on noise assumptions and improve stability and scalability. Diffusion models have expanded beyond generic image synthesis to encompass text-to-image and video generation 17 ; 12 ; 37 ; 54 ; 14 ; 16 ; 33 ; 42 ; 47 ; 49 , as well as a wide range of image restoration and editing. 2.2 Image Editing Research on image editing centers on controllability, local fidelity, and tight coupling with strong base generators. Data-driven fine-tuning and scalable pipelines 4 ; 55 ; 17 improve editing realism, while architectural and adaptation techniques 12 ; 37 ; 54 ; 56 enhance fine-grained control and efficiency. Unified frameworks 52 ; 50 ; 8 ; 30 ; 24 ; 5 ; 29 ; 19 merge generation and editing into a unified stack. FLUX.1-Kontext 15 adds high-consistency reference conditioning for rapid, reference-guided edits. Qwen-Image 48 leverages large-scale curriculum training for precise edits and complex text rendering. 2.3 Learning from Human Feedback In generative vision, aligning diffusion or flow models with human preferences has shifted from reliance on automatic quantitative proxies to explicit preference-based optimization. Direct Preference Objectives (DPO) 34 improves human-preference alignment by increasing the log-probability of preferred samples. Subsequent works refine this paradigm with robust weighting and score-matching-based formulations 45 ; 44 ; 57 , and extend it beyond single-image to multi-image, video, and editing tasks 26 ; 23 ; 51 ; 11 . In parallel, the integration of Group Relative Policy Optimization (GRPO) 36 into modern flow models converts deterministic probability-flow ODEs into marginally consistent SDEs via efficiency-aware heuristics, enabling few-step online preference alignment 21 ; 25 ; 22 . Subsequent GRPO variants expand applicability to diffusion/flow formulations, supporting text-to-image, text-to-video, and image-to-video tasks 53 ; 9 ; 46 ; 28 . 3 Preliminaries 3.1 Flow Matching Flow matching 21 ; 25 serves as a generative modeling paradigm for training continuous normalizing flows by aligning modeled velocity fields with those derived from data interpolations. In the rectified flow 25 formulation, a sample \mathbf{x}_{t} at time t\in[0,1] is defined as a linear interpolation between a base sample \mathbf{x}_{0}\sim p_{0} (typically Gaussian) and a data sample \mathbf{x}_{1}\sim p_{1} : \mathbf{x}_{t}=(1-t)\mathbf{x}_{0}+t\mathbf{x}_{1}. (1) The model learns a neural velocity field \mathbf{v}_{\theta}(\mathbf{x}_{t},t) to approximate the target

Read the complete text