MARBLE: Material Recomposition and Blending in CLIP-Space
Ta Ying Cheng*, Prafull Sharma, Mark Boss, Varun Jampani · 2025
Read MARBLE: Material Recomposition and Blending in CLIP-Space by Ta Ying Cheng*, Prafull Sharma, Mark Boss, Varun Jampani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05313.txt.
Introduction
Ta-Ying Cheng and co-authors present a way to repaint the material of an object in a photo using another photo as the sample, in a 2025 paper. A chair could borrow the brushed metal of a kettle, or a table could take on marble. Their method, MARBLE, works inside the representation space of a vision-language model, where materials can be described as directions. Those directions steer a standard image generator.
The team first found which block inside the generator's denoising network controls what material an object appears to be made of. Blending two sample materials means walking between their directions in that space. A small add-on network gives dial control over single traits such as roughness, shine, transparency, and glow. Tests showed clean blends, several edits in one pass, and success on paintings as well as photos.
Catalog record
| CREATOR | Ta Ying Cheng*, Prafull Sharma, Mark Boss, Varun Jampani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05313
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MARBLE: Material Recomposition and Blending in CLIP-Space Ta Ying Cheng*, Prafull Sharma, Mark Boss, Varun Jampani Abstract Editing materials of objects in images based on exemplar images is an active area of research in computer vision and graphics. We propose MARBLE, a method for performing material blending and recomposing fine-grained material properties by finding material embeddings in CLIP-space and using that to control pre-trained text-to-image models. We improve exemplar-based material editing by finding a block in the denoising UNet responsible for material attribution. Given two material exemplar-images, we find directions in the CLIP-space for blending the materials. Further, we can achieve parametric control over fine-grained material attributes such as roughness, metallic, transparency, and glow using a shallow network to predict the direction for the desired material attribute change. We perform qualitative and quantitative analysis to demonstrate the efficacy of our proposed method. We also present the ability of our method to perform multiple edits in a single forward pass and applicability to painting. Project Page: https://marblecontrol.github.io/ Figure 1 : Overview. We present MARBLE, a method for performing various material editing in images such as material blending (top row) and parametric control of material properties (bottom row) leveraging CLIP-space and pre-trained generative models. Given two material exemplar images, we can achieve a controllable blend of materials on the object by blending the material representation in CLIP-space. For parametric material attribute control, we learn a shallow network using synthetic data to predict the direction in CLIP-space for changing specific material properties such as metallic. † † * Work was done during internship at Stability AI. 1 Introduction Editing object materials such as diffuse albedo, roughness, etc. in images is instrumental for graphics and vision applications such as game design, advertising, and visual content creation. Performing material editing in a single image using traditional graphics techniques requires understanding of several object and environment properties such as object geometry, its material properties as well as environment illumination, making it a highly challenging task. Previous approaches for material editing use crude approximations of object geometry and environment maps Khan et al. 2006 , resulting in non-photorealistic results limited to finite material editing options. In this work, we tackle the problem of material transfer and recompose the material properties given a single image by directly leveraging the implicit knowledge of object and environment properties present in the pre-trained image diffusion models Rombach et al. 2021 . This circumvents the need for explicit estimation of these properties, which is challenging given a single image. Recent works such as Alchemist Sharma et al. 2023a and ZeST Cheng et al. 2025 demonstrate the use of diffusion models for material editing in images. ZeST Cheng et al. 2025 proposes a zero-shot technique for exemplar-based material transfer, where the object material from an exemplar image is transferred to the target object in the input image. However, this approach is limited to high-level material changes and does not perform fine-grained control of material properties. On the other hand, Alchemist Sharma et al. 2023a proposes a supervised fine-tuning of Stable Diffusion Rombach et al. 2021 for fine-grained material control such as roughness, transparency etc. in images. However, such a fine-tuning of the diffusion model has the potential to overfit to the synthetic data used for training, thereby destroying the valuable object prior knowledge in these models. In contrast, we propose a technique that can perform versatile material editing with material transfer as well as fine-grained control, while also retaining the base diffusion model priors. In particular, we propose to keep the image diffusion model intact and perform material editing via CLIP Radford et al. 2021 image features that are injected into the diffusion model. A key contribution of this work is to demonstrate that a surprising amount of material editing is possible with the manipulation of the CLIP-Space features. Our technique called MARBLE (Material Recomposition and Blending in CLIP-space) enables versatile material editing tasks ranging from performing coarse material transfer using an example image or blending materials from multiple objects (Figure MARBLE: Material Recomposition and Blending in CLIP-Space top row) to fine-grained material control of properties such as metallic, transparency, etc. (Figure MARBLE: Material Recomposition and Blending in CLIP-Space bottom row). It is far from trivial to achieve such diverse material editing tasks using only CLIP image features as CLIP captures all the object properties such as semantics, geometry etc., not just the material properties. We build our method using ZeST architecture Cheng et al. 2025 for exemplar-based material transfer with some modifications. ZeST uses IP-Adapter Ye et al. 2023 that injects CLIP features into the diffusion model, along with a color-agnostic inpainting technique for material transfer from an exemplar image to the target object image. With systematic experiments, we find a U-Net block in the Stable Diffusion that responds to the object materials. Following this insight, we propose to inject CLIP features into this specific U-Net block resulting in better material transfer. This modified architecture acts as the base for two variants of material editing. First, we show that this technique can also be used for material blending between two or more exemplar images. For the fine-grained control of the material properties such as increasing or decreasing transparency, we propose to learn lightweight MLPs, using a small synthetic dataset, that predicts material editing directions for each of the individual material properties in the CLIP-space. As a result, we can achieve fine-grained control of the material properties by moving the CLIP features along these editing directions. We provide extensive experimental analysis and results on a wide range of applications, combining a series of coarse and fine-grained material edits. Results on both synthetic and real-world images demonstrate highly plausible material editing using MARBLE for material transfer as well as fine-grained control. We compare MARBLE against other image/material editing approaches when a baseline is available, of which both our quantitative and qualitative analysis shows superiority in performance. As we keep the based diffusion model intact, we find that the learned editing directions using the shader-based synthetic dataset can generalize to various image styles, including anime and paintings. Overall, MARBLE has several favorable properties for material editing in images: • Wide Range of Novel Editing Controls. To the best of our knowledge, MARBLE is the first approach to offer parametric control, exemplar-based guidance, and blending of materials all within one general framework. • Operates only in CLIP-Space. The minimal tuning nature of our approach brings maximal flexibility in model selection and performing multiple edits in one go. • Robustness in Various Styles. MARBLE can not only generate and edit materials of realistic images but can also be incorporated with various painting and artwork styles, bringing much flexibility to graphic designers. 2 Related Work Controlled Image Editing with Diffusion Models. Recent advances in text and class-conditional image generation using diffusion models enable photorealistic image generation Dhariwal and Nichol 2021 ; Ho et al. 2020 ; Ho et al. 2022 ; Ho and Salimans 2022 ; Song and Ermon 2019 ; Karras et al. 2022 ; Kang et al. 2023 ; Saharia et al. 2022 ; Ramesh et al. 2022 ; Rombach et al. 2021 ; Nichol et al. 2021 . These models act as a base model