Large Material Gaussian Model for Relightable 3D Generation
Jingrui Ye, Lingting Zhu, Runze Zhang, Zeyu Hu, Yingda Yin, Lanjiong Li, Lequan Yu, Qingmin Liao · 2025
Read Large Material Gaussian Model for Relightable 3D Generation by Jingrui Ye, Lingting Zhu, Runze Zhang, Zeyu Hu, Yingda Yin, Lanjiong Li, Lequan Yu, Qingmin Liao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22112.txt.
Introduction
Jingrui Ye and seven co-authors present a model that generates three-dimensional objects together with their surface materials. Industry demand for 3D assets is growing, and recent large reconstruction models can build shapes quickly from generated views. Those models, though, output colored surfaces with lighting already baked in, so the assets cannot be relit in a new scene. Real use needs the material itself: base color, roughness, and metalness.
The team's Large Material Gaussian Model starts by fine-tuning a multi-view diffusion model that produces material images, guided by depth and surface-direction maps. From those views, a Gaussian representation carries each material channel alongside the geometry, in step with a two-dimensional splatting scheme. The rebuilt point cloud can then be rendered under fresh lighting maps for dynamic relighting. In experiments, the generated materials looked better than baselines and modeled materials more faithfully.
Catalog record
| CREATOR | Jingrui Ye, Lingting Zhu, Runze Zhang, Zeyu Hu, Yingda Yin, Lanjiong Li, Lequan Yu, Qingmin Liao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22112
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Material Gaussian Model for Relightable 3D Generation Jingrui Ye, Lingting Zhu, Runze Zhang, Zeyu Hu, Yingda Yin, Lanjiong Li, Lequan Yu, Qingmin Liao Abstract The increasing demand for 3D assets across various industries necessitates efficient and automated methods for 3D content creation. Leveraging 3D Gaussian Splatting, recent large reconstruction models (LRMs) have demonstrated the ability to efficiently achieve high-quality 3D rendering by integrating multiview diffusion for generation and scalable transformers for reconstruction. However, existing models fail to produce the material properties of assets, which is crucial for realistic rendering in diverse lighting environments. In this paper, we introduce the Large Material Gaussian Model (MGM) , a novel framework designed to generate high-quality 3D content with Physically Based Rendering (PBR) materials, i.e., albedo, roughness, and metallic properties, rather than merely producing RGB textures with uncontrolled light baking. Specifically, we first fine-tune a new multiview material diffusion model conditioned on input depth and normal maps. Utilizing the generated multiview PBR images, we explore a Gaussian material representation that not only aligns with 2D Gaussian Splatting but also models each channel of the PBR materials. The reconstructed point clouds can then be rendered to acquire PBR attributes, enabling dynamic relighting by applying various ambient light maps. Extensive experiments demonstrate that the materials produced by our method not only exhibit greater visual appeal compared to baseline methods but also enhance material modeling, thereby enabling practical downstream rendering applications. I Introduction Automatic 3D content creation holds immense potential across various domains, including digital gaming, virtual reality, and filmmaking. Recently, 3D generative models achieve significant advancements, and they have remarkably reduced huge manual labor of professional 3D artists and democratized creativity in 3D asset generation for non-experts. Previous optimization-based methods [ 1 , 2 , 3 , 4 , 5 , 6 ] in 3D generation have focused predominantly on score distillation sampling (SDS) with 2D diffusion model priors, which is often hindered by expensive optimization, the Janus problem, and inconsistency between different views. Recent advancements have significantly shortened inference time by leveraging large reconstruction model (LRM), which usually take multiview images as input. Some approaches [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] utilize transformers [ 15 ] to learn the triplane-based [ 16 ] neural radiance fields (NeRF) [ 17 ] , but fail to reconstruct detailed geometry and texture due to the low-resolution nature and inferior post-conversion. More recently, several works [ 10 , 18 , 14 , 11 , 19 , 20 ] employ 3D Gaussian Splatting [ 21 ] or its variant [ 22 ] as 3D representation and facilitate high-resolution training with expressiveness, combined with attention mechanism to generate Gaussians from multiview images. Despite these advancements, such approaches still struggle to control light shading and do not account for the material properties of the 3D objects, e.g., Physically Based Rendering (PBR) materials. Consequently, the reconstructed 3D objects often exhibit uncontrolled light baking in textures, resulting in a lack of realistic rendering effects and an inability to adapt appearances under varying ambient lighting conditions. This significantly limits their applicability in downstream applications. To address these limitations, as far as we know, we are the first to propose a novel method for generating 3D Gaussians with PBR materials, which are adaptable for relighting during Gaussians rendering. Our approach consists of two main components: a generative model and a reconstruction model, starting from a text prompt and finally producing 3D rendering incorporating PBR. Initially, we train a text-to-PBR multiview diffusion model on 3D material datasets, built upon multiview diffusion models, e.g., MVDream [ 23 ] . Subsequently, a large reconstruction model based on 2D Gaussian Splatting [ 22 ] is employed to reconstruct the sparse views of the generated PBR images following the designs and architectures of recent Gaussian-based reconstruction models [ 10 , 14 ] . To ensure global consistency across both the generation and reconstruction stages, we incorporate geometry data, i.e., depth and normal maps, to serve as proxy information. The integration offers benefits in three folds: 1) We can seamlessly adopt geometry generation models, e.g., [ 24 , 25 , 26 , 12 , 27 ] for additional geometry guidance and mitigate multiview inconsistency; 2) The assistance of geometry guidance models, e.g., text-to-geometry models and text-to-depth/normal models [ 6 ] , can be optionally removed when untextured 3D models are available to produce rendering geometry maps, aligning more closely with practical usages; 3) Since the PBR images inherently lack lighting cues and differ significantly from the distribution of shaded images, relying solely on them for training radiance fields produces Gaussian points with inaccurate spatial details, indicating the geometry guidance is necessary in reconstruction. In practice, we design a geometry-conditioned multiview generation model with ControlNet [ 28 ] in the generation stage, and then introduce geometry features injection and utilize geometric guidance to produce additional supervision in reconstruction. Incorporating efficient rasterization, our method achieves high-resolution renderings along with albedo, roughness, and metallic maps, enabling dynamic relighting effects under various ambient lighting conditions. Extensive qualitative and quantitative experiments validate the effectiveness of our method, demonstrating that it matches or exceeds state-of-the-art Gaussian Splatting based generation quality while achieving realistic rendering results under diverse lighting conditions. Our contributions are summarized as follows: 1. We pioneer to propose a novel framework to generate high-resolution Gaussians with material properties from text prompts, capable of being dynamically relighted through physically based rendering. 2. We introduce principled models tailored to relightable Gaussians, including a controllable multiview PBR model and a unified material Gaussian reconstruction model. 3. Extensive experiments demonstrate that our method excels in both appearance and material quality with efficiency. II Related Work II-A Large Reconstruction Model Recent advancements in NeRF [ 17 ] and 3DGS [ 21 ] enable high-quality novel view synthesis capabilities. Pioneered by the large reconstruction model (LRM) [ 7 ] , recent works [ 9 , 8 , 13 , 12 ] demonstrate that image tokens can be directly mapped to 3D representations, typically triplane-NeRF, in a feed-forward manner via a scalable transformer-based architecture [ 15 ] with large-scale 3D training data [ 29 , 30 , 31 ] . Among them, Instant3D [ 8 ] integrates LRM with multiview diffusion models [ 32 , 23 , 33 , 34 , 35 ] , using four views of generated images for better quality. To avoid inefficient volume rendering and limited triplane resolution, some concurrent works [ 11 , 10 , 36 ] follow Instant3D and introduce 3DGS or its variants [ 22 ] into sparse-view LRM variants for more efficient rendering. Specifically, LGM [ 10 ] combines 3D Gaussians from different views using a convolution-based asymmetric U-Net [ 37 ] , along with other Gaussian reconstruction models like [ 19 , 18 , 38 , 39 ] . GRM [ 11 ] and GS-LRM [ 36 ] use pixel-aligned Gaussian [ 40 , 41 ] with a pure transformer-based reconstruction model. LaRa [ 14 ] adopts 2DGS [ 22 ] representations and models scenes as Gaussian volumes and group attention layers for better quality and faster convergence. II-B Optimization-based 3D Generation In 3D generation, due to the scarcity of 3D data,