3D PixBrush: Image-Guided Local Texture Synthesis
Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka · 2025
Read 3D PixBrush: Image-Guided Local Texture Synthesis by Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03731.txt.
Introduction
Dale Decatur, Itai Lang, Kfir Aberman, and Rana Hanocka present 3D PixBrush, a tool that paints a chosen object from a photo onto the right spot of a 3D mesh. Editing texture on one region of a model usually demands hand-drawn masks. PixBrush predicts both parts itself: a mask saying where the object belongs, and the texture to fill it. The placement reads the whole scene, so the object lands in a spot that makes global sense, while the mask hugs the local geometry.
The trick is a modified score distillation method that mixes the predicted mask with the reference image as it guides generation. No scribbles or boxes from the user are needed at any point. Across many meshes and photos, the edits stayed faithful to the reference. The authors offer the method for fast, local, image-driven model styling.
Catalog record
| CREATOR | Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3D PixBrush: Image-Guided Local Texture Synthesis Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka Abstract We present 3D PixBrush, a method for performing image-driven edits of local regions on 3D meshes. 3D PixBrush predicts a localization mask and a synthesized texture that faithfully portray the object in the reference image. Our predicted localizations are both globally coherent and locally precise. Globally - our method contextualizes the object in the reference image and automatically positions it onto the input mesh. Locally - our method produces masks that conform to the geometry of the reference image. Notably, our method does not require any user input (in the form of scribbles or bounding boxes) to achieve accurate localizations. Instead, our method predicts a localization mask on the 3D mesh from scratch. To achieve this, we propose a modification to the score distillation sampling technique which incorporates both the predicted localization and the reference image, referred to as localization-modulated image guidance. We demonstrate the effectiveness of our proposed technique on a wide variety of meshes and images. \begin{overpic}[figures/lowres_images/teaser.jpg] \put(8.0,-1.0){{\color[rgb]{0,0,0}sunglasses}} \put(34.0,-1.0){{\color[rgb]{0,0,0}mask}} \put(60.0,-1.0){{\color[rgb]{0,0,0}hat}} \put(85.0,-1.0){{\color[rgb]{0,0,0}shirt}} \end{overpic} Figure 1 : 3D PixBrush produces localized textures on meshes driven by a reference image. We use a text prompt to initialize the localization and refine it to match the reference image. 3D PixBrush predicts a localization mask (yellow) and a texture that capture how the object in the reference image could plausibly be synthesized on the shape. 1 Introduction When 3D artists manually texture local regions on meshes, they often use a reference image as guidance. However, it’s challenging to create automated tools that mimic this approach: it requires using a reference image to both accurately localize a fine-grained segmentation and texture that region so that it adheres to the style of the reference image. A promising strategy is to leverage the recent emergence of text-to-image (T2I) models [ 48 , 18 , 51 ] for guiding local texturing of 3D shapes, which has been shown to work well when using text prompts as input [ 9 , 52 , 69 ] . While intuitive, such text prompts lack precision regarding the attribute’s appearance and structure. For example in Fig. 1 , it is difficult to describe the pattern in the neon ‘shirt’ result or the shape of the segmentation in the blue ‘mask’ result. Recent advances in T2I models have enabled images to act as prompts, providing a new level of control by guiding image generation with visual references [ 62 , 65 ] . Recently, EASI-Tex [ 42 ] presented a technique for globally stylizing 3D shapes using images. However, EASI-Tex cannot make local stylizations. Other works such TIP-Editor [ 70 ] and Focal Dreamer [ 28 ] perform local edits to 3D representations, but rely on manual user input to specify the localization region. In this work, we propose a technique to enable using a reference image to synthesize a localized texture ( i.e . , predict an explicit localization mask and a texture) on a 3D shape. This task requires understanding the global context of the reference image, i.e . where to position the reference image on the shape. Further, it requires local fine-grained understanding, i.e . synthesizing textures in 3D that adhere to the structure and style of the reference image. Additionally, our task is particularly challenging as explicitly blending local textures onto bare meshes prominently showcases any artifacts in the underlying segmentation mask. We present 3D PixBrush, a method for using a reference image to prescribe a local texture to a 3D mesh. Our method can capture the object in the reference image and faithfully synthesize it on the 3D shape. Notably, our method accurately localizes a segmentation region on the surface of the 3D shape which is driven by the object in the reference image, as well as a text prompt. This enables our method to produce localized textures that capture the shape of the object in the reference image, beyond what can be generically localized with text alone (see Fig. 4 ). Moreover, our method synthesizes textures that both captures the style of the reference image, and conform to the predicted localization region. As far as we are aware, no method on any 3D representation ( e.g . meshes, nerfs, gaussian splats) can use a reference image to both predict an explicit fine-grained segmentation in 3D and create a corresponding local texture contained to that region. Figure 2 : Localization Modulated Guidance. Our method (right) uses text, image, and a learned (continually updating) mask to achieve a localized texture which adheres to the reference image. Using only text as input (left), yields a localized result that does not capture all the details of the reference image. Using text and image as input without a continually updating mask (middle), results in textures that resemble the reference image but are not properly localized. Figure 3 : Gallery. We perform image-driven texturing of local regions on 3D meshes, using a reference image to both localize the relevant region on the mesh and simultaneously learn a texture that captures the style of the reference image. Our predicted local textures respect the global understanding of the shape and capture the fine-grained details of the reference image. A key innovation of our work is the ability to predict a localization mask which accurately contextualizes the reference image onto the mesh surface (yellow insets in Fig. 1 ). Our predicted segmentation masks are both globally coherent and locally precise . Globally - our predicted masks are automatically positioned appropriately on the mesh surface ( e.g . the eye glasses appear in the eye region in Fig. 1 ). Locally - our predicted masks contain the precise geometric structure of the reference image ( e.g . the round eye glass shape in Fig. 1 ). Notably, our method is able to predict such segmentation masks without any user assistance ( e.g . in the form of scribbles or bounding boxes). In tandem, we predict textures that, within the localized region of our predicted mask, accurately capture the content of the reference image (note the coloring of the sunglasses in Fig. 1 ). \begin{overpic}[figures/lowres_images/loc_no_img_grad.jpg] \put(4.0,-5.0){{\color[rgb]{0,0,0}Ref. image}} \put(43.0,-5.0){{\color[rgb]{0,0,0}Ours}} \put(73.0,-5.0){{\color[rgb]{0,0,0}Text only: `vest'}} \end{overpic} Figure 4 : Importance of Image Guidance on Localization . Using a reference image of a blue vest (left), 3D PixBrush (Ours) produces an accurate localization that precisely captures specific structural details such as the sharp v neck and points at the bottom in the blue vest. If we produce a localization using only text guidance, the localization has no influence from the reference vest image and produces a generic vest as that misses many of the features in the reference vest. We synthesize a localization mask and texture directly on the input mesh using a modified version of score distillation sampling (SDS) [ 43 , 59 ] . A straightforward application of SDS to optimize localized textures using only text guidance does not take into account the reference image (see Fig. 2 , left). Notably, text does not capture the intricate details, textures, and spatial configurations depicted by an image. To incorporate a reference image as guidance, we leverage an image conditioned IP-Adapter model [ 62 ] . However,