Concept Lancet: Image Editing with Compositional Representation Transplant
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Hancheng Min, Chris Callison-Burch, René Vidal · 2025
Read Concept Lancet: Image Editing with Compositional Representation Transplant by Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Hancheng Min, Chris Callison-Burch, René Vidal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02828.txt.
Introduction
Editing a photo with a text-driven model often breaks what should stay: push the change too hard and the scene warps, too soft and nothing happens. Jinqi Luo and co-authors offer a fix in a 2025 paper that needs no retraining. Their method first reads what concepts a picture holds, by breaking its inner code into a sparse mix of known visual ideas.
A large bank of concept codes, built for this work, supplies the parts for that reading. Once the mix is known, the edit swaps, adds, or drops the right concept with a measured dose. On several editing systems the add-on lifted both how well edits land and how much of the scene survives. The dose is set per image by the reading, not by trial and error, which saves costly guesswork.
Catalog record
| CREATOR | Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Hancheng Min, Chris Callison-Burch, René Vidal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concept Lancet: Image Editing with Compositional Representation Transplant Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Hancheng Min, Chris Callison-Burch, René Vidal Abstract Diffusion models are widely used for image editing tasks. Existing editing methods often design a representation manipulation procedure by curating an edit direction in the text embedding or score space. However, such a procedure faces a key challenge: overestimating the edit strength harms visual consistency while underestimating it fails the editing task. Notably, each source image may require a different editing strength, and it is costly to search for an appropriate strength via trial-and-error. To address this challenge, we propose Co ncept Lan cet (CoLan), a zero-shot plug-and-play framework for principled representation manipulation in diffusion-based image editing. At inference time, we decompose the source input in the latent (text embedding or diffusion score) space as a sparse linear combination of the representations of the collected visual concepts. This allows us to accurately estimate the presence of concepts in each image, which informs the edit. Based on the editing task (replace/add/remove), we perform a customized concept transplant process to impose the corresponding editing direction. To sufficiently model the concept space, we curate a conceptual representation dataset, CoLan-150K, which contains diverse descriptions and scenarios of visual terms and phrases for the latent dictionary. Experiments on multiple diffusion-based image editing baselines show that methods equipped with CoLan achieve state-of-the-art performance in editing effectiveness and consistency preservation. More project information is available at https://peterljq.github.io/project/colan . 1 Introduction Figure 1 : Given a source image and the editing task, our proposed CoLan generates a concept dictionary and performs sparse decomposition in the latent space to precisely transplant the target concept. How can we edit a given image following a specified conceptual guidance, say Cat \rightarrow Dog or Sketch \rightarrow Painting ? This problem was briefly discussed in the early work of Image Analogies [ 12 ] in the 2000s, widely studied in the era of generative adversarial networks [ 19 , 70 ] , and recently revolutionized by diffusion models [ 14 , 48 ] . Not only do diffusion models shine in producing realistic images [ 46 , 42 ] , but they also allow for conditioning on multimodal guidance such as text prompts [ 45 , 21 ] and regional masks [ 1 , 68 , 16 ] , making it handy for image editing. In this paper, we consider the task of utilizing diffusion models for image editing that impose desired concepts based on user prompts. Specifically, given a source image and its source caption, our task is to modify the content, appearance, or pattern of the image based on a given target prompt. To accomplish it with diffusion models, a basic idea is to first use the score predictor to perform the noising process (e.g., DDIM Inversion [ 49 ] ) and then follow the denoising process conditioned on the target concept. Such conditioning in the diffusion-based backbones typically happens in a structured latent space (score or text embedding space; to be detailed in § 2 ), where one moves a source latent vector towards an edit direction (i.e., the shift from the source concept to the target concept). A fruitful series of works have contributed to enhancing the inversion process [ 51 , 18 , 27 , 36 , 17 ] , innovating attention controls [ 11 , 40 , 65 , 4 ] , and tuning the backbone with conditional instructions [ 39 , 3 , 9 , 71 ] . Despite the remarkable progress, an often overlooked issue is the magnitude of the concept representation to be imposed, that is, determining how far to move along the edit direction. Prior works [ 40 , 65 ] typically assume a certain amount of editing without adjusting the magnitude based on the contexts or existing concepts of the given image. Such a heuristic choice can be problematic as we show in Figure 2 : adding too much of the edit may overwhelm the image with the target concept (Image (4)), while too little may fail to fully remove the source concept (Image (2)). Hence, since the presence of the source concept varies in every image, proper estimation of the edit magnitude is necessary for accurate editing. To address the aforementioned issue, we draw inspiration from low-dimensional [ 28 , 22 ] and compositional [ 59 , 25 ] structures in latent representations of diffusion models. In this paper, we propose Concept Lancet (CoLan), a zero-shot plug-and-play framework to interpret and manipulate sparse representations of concepts for diffusion-based image editing. Our intuition is to sufficiently model the latent spaces to analyze how much each concept is present in the source representation, which allows for accurate transplant from the source concept to the target one in the proper magnitude. We state our contributions and workflows as follows: • (§ 3.1 ) To allow for such an analysis, one needs a dictionary of directions representing diverse concepts. Existing dictionaries are limited since they either contain concepts without clear visual meanings (e.g., “hardship”), lack phrases (e.g., “made of wood”), or have only a limited number of concepts. Thus, we collect a conceptual representation dataset, CoLan-150K, of diverse descriptions for visual concepts and compute a dictionary of concept vectors to the latent (text embedding or score) space. • (§ 3.2 ) At inference time, we propose to decompose the source latent vector as a linear combination of the collected visual concepts to inform the edit. To mitigate the optimization inefficiency with an overcomplete dictionary, we instruct a vision-language model (VLM) to parse image-prompt tuples into a representative list of visual concepts as dictionary atoms. For common editing tasks of replacing, we switch the source concept vector in the decomposition with our target concept vector and synthesize the edited image with the backbone. The task of adding or removing concepts can be recasted as special cases of concept replacing, detailed in the method section. • (§ 4 ) We conduct quantitative comparisons on multiple diffusion-based image editing baselines and qualitative evaluations on the visual synthesis. Methods equipped with CoLan achieve state-of-the-art performance in editing effectiveness and consistency preservation. Notably, the plug-and-play design of our method provides flexibility in the choice of backbones and latent spaces. 2 Preliminaries in Diffusion-Based Editing This section briefly discusses diffusion-based image editing and how it involves a representation manipulation process in either the text embedding or the score space. DDIM Inversion. Diffusion model samples a new image {\boldsymbol{z}}_{0} by gradually denoising a standard Gaussian {\boldsymbol{z}}_{T} through the following reverse-time conditional denoising process: \displaystyle{\boldsymbol{z}}_{t-1} \displaystyle=\sqrt{\alpha_{t-1}}\cdot\left\lparen\frac{{\boldsymbol{z}}_{t}-\sqrt{1-\alpha_{t}}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}({\boldsymbol{z}}_{t},t,{\boldsymbol{c}})}{\sqrt{\alpha_{t}}}\right\rparen \displaystyle\quad+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\cdot\boldsymbol{\epsilon}_{\boldsymbol{\theta}}({\boldsymbol{z}}_{t},t,{\boldsymbol{c}}) \displaystyle\quad+\sigma_{t}\boldsymbol{\epsilon}_{t},\ \text{ with }\boldsymbol{\epsilon}_{t}\sim\mathcal{N}(0,{\boldsymbol{I}})\,, (1) where {\boldsymbol{z}}_{t} is the denoised image at time t (t=0,\dots,T) , {\boldsymbol{c}} is the text embedding of the caption of the image to be sampled, and \boldsymbol{\epsilon}_{\boldsymbol{\theta}}({\boldsymbol{z}}_{t},t,{\boldsymbol{c}}) models the score function [ 50 ] for noise prediction. With the choice of \sigma_{t}=0 , the denoising process in ( 1 ) enables DDIM inversion [ 49 ] . Specifically, one replaces the forward process with the iterative calling of \boldsymbol{\epsilon}_{\boldsymbol{\theta}}(\cdot,t,{\boldsymbol{c}}) to predict the forward noise for the source {\boldsymbol{z}}_{0} . The forward process often stores anchor information (e.g., attention map) [ 11 , 40 , 65 , 4 ]