CasTex: Cascaded Text-to-Texture Synthesis via Explicit Texture Maps and Physically-Based Shading
Mishan Aliev, Dmitry Baranchuk, Kirill Struminsky · 2025
Read CasTex: Cascaded Text-to-Texture Synthesis via Explicit Texture Maps and Physically-Based Shading by Mishan Aliev, Dmitry Baranchuk, Kirill Struminsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06856.txt.
Introduction
Mishan Aliev, Dmitry Baranchuk, and Kirill Struminsky build a tool that paints surface textures on 3D models from text prompts. Good textures must look right under many lights, which means storing how each point reflects light. A common trick trains the texture with gradients from an image model, but with latent models it often leaves smears and artifacts. Their 2025 method, CasTex, takes a more direct route.
CasTex runs a chain of diffusion models, each refining the texture maps in plain form rather than hidden code. That choice removes the need for a smoothing stand-in used by older tools. On public test sets, the textures beat the best older methods by a clear margin. The authors say direct maps plus staged models give cleaner, more true-to-life surfaces.
Catalog record
| CREATOR | Mishan Aliev, Dmitry Baranchuk, Kirill Struminsky |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06856
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CasTex: Cascaded Text-to-Texture Synthesis via Explicit Texture Maps and Physically-Based Shading Mishan Aliev, Dmitry Baranchuk, Kirill Struminsky Abstract This work investigates text-to-texture synthesis using diffusion models to generate physically-based texture maps. We aim to achieve realistic model appearances under varying lighting conditions. A prominent solution for the task is score distillation sampling. It allows recovering a complex texture using gradient guidance given a differentiable rasterization and shading pipeline. However, in practice, the aforementioned solution in conjunction with the widespread latent diffusion models produces severe visual artifacts and requires additional regularization such as implicit texture parameterization. As a more direct alternative, we propose an approach using cascaded diffusion models for texture synthesis (CasTex). In our setup, score distillation sampling yields high-quality textures out-of-the box. In particular, we were able to omit implicit texture parameterization in favor of an explicit parameterization to improve the procedure. In the experiments, we show that our approach significantly outperforms state-of-the-art optimization-based solutions on public texture synthesis benchmarks. Project page: https://thecrazymage.github.io/CasTex/ . 1 Introduction The creation of 3D assets is a central component of modern computer graphics, with demand driven by industries such as film production, industrial design, and computer gaming. High-quality assets should simultaneously meet aesthetic requirements and technical production requirements (e.g., clean mesh topology and limited polygon counts). As a result, 3D modeling remains a labor-intensive creative task that requires multiple expert skills and is difficult to automate due to the scarcity of high-quality data and the challenge of formalizing target criteria . Meanwhile, the rapid progress of generative models in the visual domain has already transformed many creative domains and is now extending into 3D graphics. Advances in 3D reconstruction [ 23 , 1 , 15 ] and text-to-image foundation models [ 31 , 30 ] have enabled early propotypes for fully automated, end-to-end 3d asset generation [ 27 , 21 , 18 ] . These systems leverage the prior knowledge acquired by the diffusion model to generate visually appealing 3D content. However, the resulting assets come in a volumetric representation and are often incompatible with modern graphics engines. Even when volumetric outputs are converted into meshes, the geometry tends to be excessively detailed and may contain high-level structural errors (e.g., multiple faces as in the Janus effect). The limitations mentioned above may be critical when technical requirements must be met. Texture synthesis offers a more targeted and practical path toward automating parts of the 3D asset pipeline by generating realistic surface appearance for a given shape. In this work, we assume that the geometry is provided by an artist and focus on synthesizing physically based textures [ 4 ] , which allow for expressive material modeling and realistic rendering under diverse lighting conditions. Early texture synthesis methods relied on back-projecting synthetic images onto 3D surfaces. However, such approaches are incompatible with PBR materials because RGB images capture only a limited subset of material properties through RGB colors. As an alternative, we consider an optimization-based framework that enables back-propagation through a differentiable renderer to recover full material parameters. In particular, we build on a line of recent work exploring Score Distillation Sampling (SDS) [ 27 ] for optimization-based generation. Our goal is to develop a zero-shot texture-synthesis solution that avoids reliance on auxiliary 3D data and foundation model fine-tuning. However, we observe that the texture synthesis setup unveils the limitations of Score Distillation Sampling applied with modern diffusion models. Consistent with [ 36 ] , we find that SDS often drives optimization toward images that drift outside the natural image manifold. We expand upon this observation and attribute the effect to inherent properties of latent diffusion models. Motivated by this analysis, we explore cascaded diffusion models, which operate in RGB space rather than latent space. Our method follows the cascaded-model structure: we first generate a coarse texture using SDS guided by a low-resolution model, then refine it using SDS with a super-resolution model. In the following, we summarize our main contributions. • We study the limitations of Score Distillation Sampling when applied to latent diffusion models. We show that the corresponding optimization procedure tends to converge to images outside the natural image domain. • Based on these insights, we introduce CasTex, a texture-synthesis approach using Score Distillation Sampling with cascaded pixel-space diffusion models. The proposed method mimics the architecture of a diffusion model and acts in two stages. The first stage generates a coarse texture, while the second stage improves the overall quality of the texture obtained in the first stage. • We demonstrate that CasTex outperforms state-of-the-art optimization-based texture synthesis solutions on the Objaverse dataset. Score Distillation Sampling in pixel space allows omitting additional regularizers required in previous works and enables efficient, high-resolution texture generation. Additionally, we report the scaling performance of our method with the diffusion model size, the impact of lighting conditions used during training, and the role of physically-based textures. 2 Related Work Generating high-quality textures for 3D objects is a long-standing challenge in computer graphics and machine learning. Early approaches are based on category-specific generative models, such as generative adversarial networks (GANs) like Texture Fields [ 26 ] , SPSG [ 10 ] , LTG [ 37 ] , and Texturify [ 34 ] . These methods often struggle with limited generalization and required extensive manual adjustments. Recent advances in large-scale text-to-image diffusion models [ 30 , 31 ] have transformed texture synthesis, allowing more flexible and realistic results. However, different approaches vary in how they leverage diffusion models to incorporate 3D information and ensure texture consistency. Diffusion-based texture generation. A significant part of recent works formulates texture generation as an iterative inpainting process using depth-conditioned diffusion models. Text2Tex [ 6 ] , TEXTure [ 29 ] , and Paint3D [ 38 ] generate textures by synthesizing single-view images and progressively refining the texture using different object views. Although these zero-shot approaches allow for training-free generation, they often struggle with seams at visibility boundaries and suffer from baked-in lightning artifacts, leading to global inconsistencies. To mitigate these issues, some methods introduce additional refinement steps: Text2Tex employs an inpainting diffusion model for post-processing, while Paint3D refines textures directly in UV space. TexDreamer [ 19 ] follows a different approach by directly generating a UV texture map using a fine-tuned diffusion model, avoiding the need for iterative view synthesis and improving consistency across the surface. An alternative strategy to address inconsistency is to introduce additional structures to enforce coherence between views. SyncMVD [ 20 ] tackles this by generating each view independently while employing a synchronization module that aligns outputs in every denoising step of the diffusion model. Similarly, TexFusion [ 5 ] suggest a module that aggregates multiview object predictions into a unified latent texture representation, iteratively refining it throughout the diffusion process to ensure consistency across the object. Another approach, inspired by MVDream [ 32 ] , leverages consistent multiview image generation as an initialization step for texture synthesis. Meta 3D TextureGen [ 2 ] follows this idea by generating four consistent views, which are projected onto a 3D surface to form an initial texture for subsequent refinement in UV space. MVPaint [ 8 ] extends this pipeline by introducing a spatially-aware 3D inpainting stage between projection and UV