MatLat : Material Latent Space for PBR Texture Generation
Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung · 2025
Read MatLat : Material Latent Space for PBR Texture Generation by Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17302.txt.
Introduction
Kyeongmin Yeo and three co-authors present MatLat, a system that paints realistic surface textures onto 3D models. Game and film artists need textures that state how a surface reflects light, absorbs it, and feels rough or smooth. Training data for such full material textures is scarce, so the team borrows strength from image models trained on ordinary photos.
MatLat learns a compact material space by fine-tuning the image model's encoder instead of freezing it, so extra material channels fit without warping the space. A patch rule keeps each code tied to its matching spot on the surface, which keeps views consistent. Tests against older methods show sharper, more faithful textures, and removing any single piece hurt the result.
Catalog record
| CREATOR | Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MatLat : Material Latent Space for PBR Texture Generation Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung Abstract We propose a generative framework for producing high-quality PBR textures on a given 3D mesh. As large-scale PBR texture datasets are scarce, our approach focuses on effectively leveraging the embedding space and diffusion priors of pretrained latent image generative models while learning a material latent space, MatLat , through targeted fine-tuning. Unlike prior methods that freeze the embedding network and thus lead to distribution shifts when encoding additional PBR channels and hinder subsequent diffusion training, we fine-tune the pretrained VAE so that new material channels can be incorporated with minimal latent distribution deviation. We further show that correspondence-aware attention alone is insufficient for cross-view consistency unless the latent-to-image mapping preserves locality. To enforce this locality, we introduce a regularization in the VAE fine-tuning that crops latent patches, decodes them, and aligns the corresponding image regions to maintain strong pixel-latent spatial correspondence. Ablation studies and comparison with previous baselines demonstrate that our framework improves PBR texture fidelity and that each component is critical for achieving state-of-the-art performance. Project page is at https://matlat-proj.github.io . Figure 1 : PBR Textures Generated by MatLat . Our method produces PBR textures that accurately represent rough materials (left), metallic surfaces (middle), and complex mixed materials (right). 1 Introduction Generative modeling has recently driven remarkable progress in 3D asset creation. Modern methods can synthesize geometry Jun and Nichol 2023 ; Poole et al. 2023 ; Chen et al. 2023b ; Yi et al. 2024 , textures and materials Chen et al. 2023a ; Richardson et al. 2023 ; Kim et al. 2024 ; Yeo et al. 2025 , motions and animations Tevet et al. 2023 ; Hassan et al. 2023 ; Petrovich et al. 2023 , and even complete assets and scenes Zhang et al. 2024a ; Höllein et al. 2023 ; Li et al. 2024a . Among these, a crucial requirement for production-ready assets is Physically Based Rendering (PBR) textures, which comprise albedo (diffuse color), roughness, and metallic channels. These channels enable physically accurate relighting under arbitrary illumination and are essential components of modern graphics pipelines. Recent works on PBR texture generation Chen et al. 2023a ; Richardson et al. 2023 ; Wang et al. 2024 ; Chen et al. 2025 advance beyond baked appearances by predicting material maps that enable relightable assets. However, progress remains constrained by the scarcity of large-scale, high-quality datasets with PBR channels, as the existing public dataset Deitke et al. 2023b ; Deitke et al. 2023a is limited to only tens of thousands of assets. A natural direction to overcome this challenge is to leverage pretrained image generative models, which provide strong priors learned from large-scale RGB image datasets Schuhmann et al. 2022 . Techniques such as Score Distillation Sampling (SDS) Poole et al. 2023 and its variants Wang et al. 2023 ; Lin et al. 2023 have been explored for this purpose, but they often fail to produce high-fidelity PBR material images and exhibit saturation artifacts. In practice, the most effective strategy has been to generate multi-view images with pretrained 2D models and then project them onto a mesh Kim et al. 2024 ; Yeo et al. 2025 ; Zhu et al. 2024 ; Huang et al. 2024 ; Wei et al. 2025 ; Zhu et al. 2025 ; Fei et al. 2025 ; He et al. 2025 ; Chen et al. 2025 . However, this multi-view generation approach still faces two major challenges for PBR texture synthesis. First, directly leveraging diffusion models trained on latent RGB images for PBR material generation is non-trivial, as PBR textures include additional channels (roughness and metallic). Simply mapping these channels to the pretrained encoder leads to a substantial domain gap. Second, preserving multi-view consistency is challenging, and its failure can lead to blurring and visual artifacts in overlapping regions when the images are unprojected onto the mesh surface. To this end, we introduce a novel framework that addresses both challenges by learning a Mat erial Lat ent space, referred to as MatLat , which substantially improves the quality of generated PBR textures. Specifically, we propose a two-stage pipeline: we first learn the material latent space by fine-tuning the pretrained VAE using PBR texture images, resulting in Material VAE ( MatVAE ), and then fine-tune a diffusion model to generate multi-view material images in the adapted latent space. Within this framework, we introduce the following two main components addressing the aforementioned challenges. The first component is a latent-space adaptation module in MatVAE that extends the pretrained latent space to incorporate additional PBR channels—roughness and metallic—while preserving the pretrained prior. Most previous works Zhu et al. 2024 ; Huang et al. 2024 ; Wei et al. 2025 ; Zhu et al. 2025 ; Fei et al. 2025 ; He et al. 2025 ; Chen et al. 2025 adopt a naïve approach that freezes the pretrained encoder and incorporates the additional PBR channels through zero-channel padding. While straightforward, this causes a severe distributional mismatch between the encoded PBR materials and the pretrained latent space, which leads to suboptimal performance during subsequent diffusion model fine-tuning. In contrast, our approach fine-tunes the pretrained encoder to effectively incorporate the additional channels of PBR material images while applying a distributional regularization that constrains deviations from the original latent distribution. This controlled adaptation effectively preserves the pretrained priors, stabilizes subsequent diffusion fine-tuning, and thereby better leverages these priors to produce higher-fidelity PBR textures. While similar latent-space extensions have been explored for transparent Zhang and Agrawala 2024 or RGB-D image synthesis Krishnan et al. 2025 , this work is the first to adapt this idea to PBR texture generation, and we further identify crucial components that make such adaptation effective in practice. The second component is locality regularization in MatVAE , which establishes latent–image spatial alignment. This alignment is subsequently leveraged by correspondence-aware attention (CAA) Tang et al. 2023 within the diffusion model. Previous work on multi-view generation Tang et al. 2023 has shown that CAA effectively enforces correspondence across views in the pretrained RGB latent space when the latent-image mapping preserves spatial locality. While such locality largely holds for pretrained RGB encoder He et al. 2025 , it often breaks for additional material channels, leading to multi-view inconsistent images. To address this, we introduce locality regularization , applied during MatVAE fine-tuning, which explicitly enforces spatial alignment between latent tokens and image pixels. Specifically, we crop latent patches, decode them, and align the corresponding image regions with an \ell_{2} reconstruction loss, encouraging pixel–latent spatial locality. In our experiments, we evaluate the quality of the generated PBR textures against comprehensive baselines, including models trained from scratch, SDS-based optimization methods, and multi-view diffusion models. Quantitative results with comprehensive evaluation metrics show that MatLat outperforms baselines trained from scratch, which suffer from suboptimal quality due to limited PBR supervision, and SDS-based methods, which tend to produce over-saturated appearances and exhibit prohibitive runtimes. Notably, our method also outperforms multi-view diffusion baselines on most quantitative metrics, achieving new state-of-the-art performance. Additionally, we present systematic ablation studies that validate the effectiveness of each component of our framework, including the proposed latent-space adaptation in MatVAE , correspondence-aware attention, and locality regularization.