EditCaption: Human-Aligned Instruction Synthesis for Image Editing via Supervised Fine-Tuning and Direct Preference Optimization Conference: Conference on Empirical Methods in Natural Language Processing; November; Peking

Xiangyuan Wang 1,5* Honghao Cai 2,5* Yunhao Bai 1,5 Tianze Zhou 3,5 Haohua Chen 4,5 Yao Hu 5 Xu Tang 5 Yibo Chen 5 Wei Zhu 5† · 2026

Read EditCaption: Human-Aligned Instruction Synthesis for Image Editing via Supervised Fine-Tuning and Direct Preference Optimization Conference: Conference on Empirical Methods in Natural Language Processing; November; Peking by Xiangyuan Wang 1,5* Honghao Cai 2,5* Yunhao Bai 1,5 Tianze Zhou 3,5 Haohua Chen 4,5 Yao Hu 5 Xu Tang 5 Yibo Chen 5 Wei Zhu 5† (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08213.txt.

Introduction

Wang and 11 colleagues present EditCaption for image editing instructions. Good source-target pairs with exact editing instructions matter for instruction-guided editing, yet building triplets at scale stays costly. Recent pipelines use vision-language models to synthesize editing instructions, but strong models still struggle to describe visual changes between pairs. They show three repeated failure types: mixed-up orientation, unclear viewpoint, and missing details. Human review of 400 pairs found several open-source baselines with critical-error rates above 47 percent, leaving many synthesized instructions unfit for training.

Catalog record

CREATORXiangyuan Wang 1,5* Honghao Cai 2,5* Yunhao Bai 1,5 Tianze Zhou 3,5 Haohua Chen 4,5 Yao Hu 5 Xu Tang 5 Yibo Chen 5 Wei Zhu 5†
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57549
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08213

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EditCaption: Human-Aligned Instruction Synthesis for Image Editing via Supervised Fine-Tuning and Direct Preference Optimization Conference: Conference on Empirical Methods in Natural Language Processing; November; Peking Xiangyuan Wang 1,5* Honghao Cai 2,5* Yunhao Bai 1,5 Tianze Zhou 3,5 Haohua Chen 4,5 Yao Hu 5 Xu Tang 5 Yibo Chen 5 Wei Zhu 5† Abstract . High-quality training triplets—source-target image pairs paired with precise editing instructions—are a critical bottleneck for scaling instruction-guided image editing models. While Vision-Language Models (VLMs) have been widely adopted for automated instruction synthesis, we identify three systematic failure modes in their outputs when applied to image-pair reasoning: orientation inconsistency (e.g., left/right confusion), viewpoint ambiguity, and lack of fine-grained attribute description. Human evaluation confirms that over 47% of instructions generated by strong baseline VLMs contain critical errors that render them unusable for downstream training. To address this, we propose EditCaption , a scalable two-stage post-training pipeline for VLM-based editing instruction synthesis. In the first stage, we construct a 100K supervised fine-tuning (SFT) dataset by combining automatic annotation from GLM with EditScore-based quality filtering and human refinement, targeting spatial, directional, and attribute-level accuracy. In the second stage, we build 10K human-annotated preference pairs that explicitly capture the three failure modes, and apply Direct Preference Optimization (DPO) to further align the model beyond what SFT alone achieves. Extensive experiments on three benchmarks—an in-house Eval-400 set, ByteMorph-Bench, and HQ-Edit—demonstrate that our fine-tuned Qwen3-VL models substantially outperform all open-source baselines. Our 235B model achieves a weighted score of 4.712 on Eval-400, surpassing Gemini-3-Pro (4.706) and surpassing GPT-4.1 (4.220) and Kimi-K2.5 (4.111). On the challenging ByteMorph-Bench, it scores 4.588, exceeding both Gemini-3-Pro (4.522) and GPT-4.1 (3.412). Human evaluation further shows that critical error rates drop from 47.75% to 23% after our two-stage training, with the correct rate improving from 41.75% to 66%. Our work provides a practical and effective approach for scalable, human-aligned instruction synthesis to support large-scale image editing data construction. 1. Introduction Instruction-guided image editing has emerged as a central paradigm in controllable visual generation, enabling users to specify complex transformations in natural language ( Brooks et al. 2023 ; Fu et al. 2023 ) . The practical ceiling of such systems is, however, not primarily determined by the expressiveness of the underlying generative model, but by the quality and scale of the training triplets: paired source and target images accompanied by a precise editing instruction. Constructing such triplets at scale is prohibitively expensive via manual annotation, motivating a line of work that delegates instruction synthesis to large Vision-Language Models ( Liu et al. 2023 ; Bai et al. 2023 ) . This delegation, however, rests on an assumption that does not hold in practice: that VLMs capable of rich semantic captioning can equally well describe the transformation between two images. Through systematic empirical analysis, we identify three dominant failure modes in existing VLM-generated instructions. First, orientation inconsistency: models frequently reverse directional relationships, e.g., describing a leftward displacement as rightward. Second, viewpoint ambiguity: models fail to correctly characterize perspective shifts or camera-motion changes between the source and target. Third, lack of fine-grained detail: models produce generic descriptions that omit critical attribute-level changes such as color, texture, or object-level modifications. These failure modes are not rare edge cases—our human evaluation on 400 image pairs shows that over 47% of instructions generated by strong baseline VLMs contain critical errors (P0) that render them unusable for downstream editing model training. The root cause is systematic. VLMs are trained to maximize token likelihood over fluent natural language, a distribution that is largely insensitive to the precise relational and spatial structure encoded in image pairs. No amount of scale or prompt engineering reliably resolves this misalignment; what is needed is training data and optimization objectives specifically designed to target these failure modes. To address this, we develop EditCaption, a two-stage post-training pipeline for VLM-based instruction synthesis. In the first stage, we construct a large-scale supervised fine-tuning (SFT) dataset of 100K image pairs with human-refined instructions. Starting from automatic annotations generated by GLM, we use EditScore ( Luo et al. 2025 ) to measure instruction-image consistency and filter misaligned samples, and then engage human annotators to correct spatial, directional, and attribute-level errors. In the second stage, we construct 10K preference pairs by sampling model outputs and having human annotators provide corrected alternatives targeting the three identified failure modes. We then apply Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) to further align the model with human judgment, achieving improvements beyond what SFT alone can deliver. We validate our approach on three benchmarks: an in-house evaluation set of 400 diverse image pairs, ByteMorph-Bench (focusing on non-rigid motion and camera transformations), and HQ-Edit (high-resolution images with detailed editing instructions). Across all benchmarks, our fine-tuned Qwen3-VL models substantially outperform all open-source baselines—including Qwen3.5-397B-A17B, Kimi-K2.5, and GLM-4.5V—and approach the performance of closed-source model such as Gemini-3-Pro and GPT-4.1, despite being significantly smaller in scale. In summary, this paper makes three contributions: • We conduct a systematic empirical analysis of failure modes in VLM-based editing instruction synthesis, identifying orientation inconsistency, viewpoint ambiguity, and lack of fine-grained detail as the three dominant error categories, and providing quantitative evidence via large-scale human evaluation. • We introduce a scalable two-stage data construction and training pipeline—comprising 100K human-refined SFT data and 10K human-annotated preference pairs. • We demonstrate through comprehensive experiments that our approach achieves state-of-the-art performance among open-source models across multiple benchmarks, and that the SFT+DPO combination consistently outperforms either stage alone, validating our two-stage design. 2. Related Work 2.1. Instruction-Guided Image Editing Text-driven image editing has advanced rapidly with the rise of large-scale diffusion models ( Rombach et al. 2022 ) . Prompt-to-Prompt ( Hertz et al. 2023 ) demonstrates that cross-attention maps in a frozen diffusion model can be manipulated to achieve precise, word-level edits without retraining. Imagic ( Kawar et al. 2023 ) extends this to real images by optimizing a text embedding while fine-tuning the model. InstructPix2Pix ( Brooks et al. 2023 ) takes a fundamentally different approach, learning to follow free-form editing instructions by training on a large synthetic dataset of (image, instruction, edited image) triplets generated by combining GPT-3 and Stable Diffusion. MagicBrush ( Zhang et al. 2023 ) subsequently provides a manually annotated counterpart to address the quality gap in synthetic training data. Recent benchmarks such as HQ-Edit ( Hui et al. 2024 ) and ByteMorph-Bench ( Chang et al. 2025 ) further highlight the importance of instruction quality and diversity, particularly for spatially complex and non-rigid transformations. Despite these advances, all such systems remain fundamentally constrained by the quality of their training triplets—a bottleneck our work directly targets by developing a scalable, human-aligned pipeline for instruction synthesis. 2.2. Vision-Language Models for Data Synthesis The capacity of VLMs to bridge visual perception and language generation has made them natural candidates for automated annotation. BLIP-2 ( Li et al. 2023 ) introduces a lightweight querying transformer that bootstraps image-text alignment from frozen unimodal encoders. Building on this, InstructBLIP ( Dai et al. 2023 ) and LLaVA ( Liu et al. 2023 ) demonstrate that instruction-tuned VLMs can produce detailed, context-sensitive image descriptions. Qwen-VL ( Bai et al. 2023 ) further extends VLM capability to grounding and localization tasks.

Read the complete text