Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers
Yiqing Shi, Yiren Song, Mike Zheng Shou · 2025
Read Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers by Yiqing Shi, Yiren Song, Mike Zheng Shou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18673.txt.
Introduction
Yiqing Shi, Yiren Song, and Mike Zheng Shou turn image editing diffusion models into dense perceivers of depth, surface tilt, and matte edges. Most dense perception tools rest on text to image generators built for loose creation. Editing models, by contrast, must stay true to a given image pixel by pixel, a far better base for measures that must match the input.
Edit2Perceive fine tunes the full FLUX.1 Kontext editing model with a pixel space consistency loss that holds structure across denoising steps. One step, fixed inference then runs fast, trained on small sets. Broad tests show top class results across the three tasks. The work recasts generation skill as measurement skill.
Catalog record
| CREATOR | Yiqing Shi, Yiren Song, Mike Zheng Shou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18673
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers Yiqing Shi, Yiren Song, Mike Zheng Shou Abstract Recent advances in diffusion transformers have shown remarkable generalization in visual synthesis, yet most dense perception methods still rely on text-to-image (T2I) generators designed for stochastic generation. We revisit this paradigm and show that image editing diffusion models are inherently image-to-image consistent, providing a more suitable foundation for dense perception task. We introduce Edit2Perceive , a unified diffusion framework that adapts editing models for depth, normal, and matting. Built upon the FLUX.1 Kontext architecture, our approach employs full-parameter fine-tuning and a pixel-space consistency loss to enforce structure-preserving refinement across intermediate denoising states. Moreover, our single-step deterministic inference yields up to faster runtime while training on relatively small datasets. Extensive experiments demonstrate comprehensive state-of-the-art results across all three tasks, revealing the strong potential of editing-oriented diffusion transformers for geometry-aware perception. Code is released at https://github.com/showlab/Edit2Perceive . We present Edit2Perceive , a unified framework for diverse dense prediction tasks. Our model achieves state-of-the-art performance across zero-shot Monocular Depth Estimation, Surface Normal Estimation, and Interactive Matting, consistently outperforming previous methods. The bar charts on the right compare the average ranking score of our method against prior work on each task; lower is better. The green dot ( \bullet ) on the input images indicates the positive user prompt for interactive matting. † † footnotetext: * Equal contribution. † † footnotetext: \dagger Corresponding author. 1 Introduction Dense perception tasks, such as Monocular Depth Estimation, Surface Normal Estimation, and Interactive Matting, are central to understanding 3D structure and object segmentation from 2D images. These tasks are particularly challenging because they require models to accurately predict pixel-level geometric or optical attributes. Monocular Depth Estimation, for instance, necessitates recovering 3D scene geometry from a single image, which is inherently ill-posed. Similarly, Surface Normal Estimation demands precise angular predictions, while Interactive Matting involves the segmenting the foreground from the background in an image given interactive input, which is highly sensitive to edge details. Solving these ill-posed problems depends on how much prior knowledge a model possesses about the visual world. Recent progress has been largely driven by leveraging rich visual priors from large-scale text-to-image (T2I) diffusion models, such as Stable Diffusion 36 , which have shown impressive success when adapted for perception tasks 13 . However, we argue that this paradigm is inherently limited by a representation mismatch. T2I models are trained to synthesize diverse visual concepts based on unstructured textual prompts, which excel at semantic composition (“concept-to-pixel” mapping) but lack the capacity to reason about the structural relationships within an image. As a result, they are misaligned with the deterministic and geometry-aware nature of dense perception. We revisit this assumption and propose that image-to-image (I2I) diffusion models, especially context-based editors like FLUX.1 Kontext 16 , offer a more natural foundation for dense perception. Unlike T2I models, these models are trained to generate semantically coherent edits conditioned on an existing image. This pretraining objective implicitly requires parsing the input into a structured scene representation—capturing objects, surfaces, and their interrelations. Such structured priors is essential for dense perception. Building upon this insight, we introduce Edit2Perceive , a unified framework that adapts a powerful I2I diffusion model into a dense perceiver. We first convert the stochastic denoising process into a pseudo-deterministic path by fixing the random seed, ensuring a unique and reproducible input-output mapping.To further strengthen geometric fidelity, we design a pixel-space consistency loss that enforces fine-grained geometric fidelity and enhances structure preservation in the final output. Additionally, to meet the high numerical stability required for Monocular Depth Estimation task, we theoretically derive that a square-root mapping minimizes the Absolute Reletive (AbsRel) error caused by the preprocess. Finally, benefiting from the inherent mechanism of flow matching, Edit2Perceive enables single-step deterministic inference, combining accuracy with efficiency. Extensive experiments across three major dense perception tasks (Monocular Depth Estimation, Surface Normal Estimation, and Interactive Matting) show that our method achieves state-of-the-art performance while using limited train data. It achieved impressive results in the wild (Fig. Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers ). Our main contributions are summarized as follows: • We demonstrate that image editing diffusion models, rather than text-to-image generators, provide a better inductive bias for deterministic dense perception, effectively bridging generative modeling and geometric reasoning. • We propose Edit2Perceive , a unified diffusion transformer fine-tuned across depth, normal, and matting tasks, enhanced with a pixel-consistency loss and an analytically derived optimal normalization for depth. • We achieve state-of-the-art performance with efficient single-step inference, showing that editing-oriented diffusion models can serve as a new class of perception-oriented foundation models. Figure 1 : Overview of the Editor2Perceive Framework. We adapt the FLUX.1 Kontext editor for dense perception. An input image x and text prompt p condition for the target image y . In the forward process, target token will be adding noise z_{t} and then concat with text condition token and image condition token. The DiT backbone is trained to predict the velocity of the flow from a noise vector z_{0} to the target latent z_{1} . Our training objective combines the latent-space flow matching loss ( \mathcal{L}_{\text{FM}} ) with a pixel-space consistency loss ( \mathcal{L}_{\text{Cons}} ) for enhanced geometric fidelity. 2 Related Work 2.1 Image Generation and Editing Models Large-scale diffusion models have become the cornerstone of visual content creation. Text-to-image (T2I) systems such as Stable Diffusion 36 and FLUX.1 17 demonstrate strong semantic understanding through large-scale image–text pre-training. Meanwhile, research attention is shifting toward more controllable generation 27 ; 29 ; 26 ; 28 ; 41 ; 55 ; 57 ; 4 . Unlike T2I models that generate scenes from abstract text, I2I models aim to make precise modifications to existing visual content. Recent editing systems such as Step1X-Edit 23 and FLUX.1 Kontext 16 learn complex correspondences between image structure and editing instructions, achieving high structural consistency. With the transition to Diffusion Transformers 32 , controllable generation has advanced rapidly. Models such as EasyControl 56 exemplify MM-DiT–based conditioning and have inspired a broad line of DiT-driven methods 43 ; 8 ; 12 ; 46 ; 42 ; 40 ; 9 ; 25 ; 38 ; 39 . 2.2 Dense Perception Tasks Dense perception tasks, including monocular depth estimation, surface normal estimation, and interactive matting, are fundamental to computer vision. Traditional approaches can be broadly categorized into two paths. The first is data-driven, involving training specialized models on massive annotated datasets. From MiDaS 34 , which pioneered cross-dataset generalization, to the Depth Anything series 50 ; 51 , which leverages large-scale unlabeled data for pre-training, this path improves performance by scaling up data and model size. However, this approach is computationally intensive, and the resulting models are often designed for a single task, limiting their ability to generalize to new ones. The second path is task-driven, focusing on designing sophisticated inductive biases for specific tasks, such as the geometric constraints in DSINE 1 for normal estimation. While efficient, these designs are often difficult to transfer to other perception tasks. Both paths highlight the need for a powerful and general-purpose visual prior, which is precisely what large-scale generative models can offer. 2.3 Diffusion Models for Dense Perception To leverage the powerful priors of