PRM: Photometric Stereo based Large Reconstruction Model

Wenhang Ge, Jiantao Lin, Guibao Shen, Jiawei Feng, Tao Hu, Xinli Xu, Ying-Cong Chen · 2024

Read PRM: Photometric Stereo based Large Reconstruction Model by Wenhang Ge, Jiantao Lin, Guibao Shen, Jiawei Feng, Tao Hu, Xinli Xu, Ying-Cong Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07371.txt.

Introduction

Wenhang Ge and co-authors build a reconstruction model that reads shine to recover shape in a 2024 paper. Photometric stereo infers fine surface detail from how light plays across an object. Earlier large reconstruction models fed on plainly lit images, which starve them of exactly those cues.

PRM renders its own training views under shifting materials and lights, using a real-time physics-based renderer with mesh rasterization. The mesh itself is explicit, so light physics can grade the geometry directly and handle glossy color with care. Rich lighting variety also hardens the model against odd-looking inputs at test time. Benchmarks put PRM ahead of rival models on fine detail. The authors present photometric cues, long a specialist tool, as fuel that big reconstruction models were missing.

Catalog record

CREATORWenhang Ge, Jiantao Lin, Guibao Shen, Jiawei Feng, Tao Hu, Xinli Xu, Ying-Cong Chen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07371

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRM: Photometric Stereo based Large Reconstruction Model Wenhang Ge, Jiantao Lin, Guibao Shen, Jiawei Feng, Tao Hu, Xinli Xu, Ying-Cong Chen Abstract We propose PRM, a novel photometric stereo based large reconstruction model to reconstruct high-quality meshes with fine-grained local details. Unlike previous large reconstruction models that prepare images under fixed and simple lighting as both input and supervision, PRM renders photometric stereo images by varying materials and lighting for the purposes, which not only improves the precise local details by providing rich photometric cues but also increases the model’s robustness to variations in the appearance of input images. To offer enhanced flexibility of images rendering, we incorporate a real-time physically-based rendering (PBR) method and mesh rasterization for online images rendering. Moreover, in employing an explicit mesh as our 3D representation, PRM ensures the application of differentiable PBR, which supports the utilization of multiple photometric supervisions and better models the specular color for high-quality geometry optimization. Our PRM leverages photometric stereo images to achieve high-quality reconstructions with fine-grained local details, even amidst sophisticated image appearances. Extensive experiments demonstrate that PRM significantly outperforms other models. Project page: https://wenhangge.github.io/PRM/ . 1 Introduction Recent advancements in generative models ( Song et al., 2020 ; Ho et al., 2020 ) have spurred notable progress in 2D content creation, driven by fast growth in data volumes. In contrast, the development in 3D field remains encumbered due to limited 3D assets, which are essential for diverse applications including game modeling ( Gregory, 2018 ) , computer animation ( Parent, 2012 ; Lasseter, 1987 ) , and virtual reality ( Schuemie et al., 2001 ) . Traditional approaches to generating 3D assets have utilized optimization-based techniques from multi-view posed images ( Wang et al., 2021 ; Yariv et al., 2021 ; Yariv et al., 2020 ) or have harnessed SDS-based distillation methods from 2D diffusion models ( Liang et al., 2023 ; Lin et al., 2023 ; Poole et al., 2022 ) . Despite their effectiveness, these methods often require increased computational costs, making them less suitable for rapid deployment in real world. Feed-forward 3D generative models ( Hong et al., 2023 ; Hu et al., 2024 ; Zhang et al., 2024 ) have been developed to address the limitations of per-scene optimization by training a generalizable model on large-scale 3D assets. Notably, the Large Reconstruction Model (LRM) ( Hong et al., 2023 ) has demonstrated promising results, exhibiting exceptional reconstruction speeds. The subsequent LRM series ( Hong et al., 2023 ; Xu et al., 2023 ; Wang et al., 2024 ; Xu et al., 2024a ; Tang et al., 2024 ) utilizes a Transformer-based architecture to encode either single or multi-view images, and decoding them into 3D representations, such as triplanes ( Chan et al., 2022 ) , Flexicubes ( Shen et al., 2023 ) or 3D Gaussians ( Tang et al., 2024 ) . These 3D representations enable differentiable rendering from arbitrary viewpoints, which is crucial for calculating multi-view reconstruction loss for optimization. While LRM series demonstrate effectiveness and efficiency in globally coherent 3D assets reconstruction, they encounter limitations in accurately capturing fine-grained local details. This challenge stems from their dependence on images rendered under fixed and simple lighting conditions, which provide inadequate photometric information for detailed surface reconstruction. Furthermore, LRM series are sensitive to variations in the appearance of conditioned images, particularly when dealing with surfaces exhibiting glossy characteristics. As a result, LRM series tend to entangle the texture and geometry, leading to wrong geometry reconstruction. Figure 1: Top left: PRM is capable of reconstructing high-quality meshes with fine-grained local details even under complex image appearances, such as specular highlights and dark appearances. Right: We demonstrate a scene comprising diverse 3D objects generated by our models. Bottom left: A zoomed-in visualization of the scene highlights these details more clearly. To address the above mentioned challenges, we introduce PRM, a photometric stereo based large reconstruction model. This model is adept at capturing fine-grained local details and ensures robustness against the complex appearances of input images. We achieve these objectives by leveraging photometric stereo images ( Hernandez et al., 2008 ) . Specifically, we render photometric stereo images by varying camera pose, materials (i.e., metallic and roughness), and lighting for both input and supervision. However, rendering these images is not trivial since there are infinite possible combinations of camera pose, materials and lighting. In the recent LRM series, images are typically rendered offline using Blender’s Cycles engine ( Hess, 2013 ) . While this approach produces high-quality, noise-free images, it requires numerous samples of lighting directions, significantly increasing the time cost and making it expensive to maximize the training sample distribution. To address this issue, we incorporate a real-time, physically based rendering technique known as split-sum approximation ( Karis & Games, 2013 ) , along with mesh rasterization for online rendering. This approach offers greater flexibility compared to traditional offline methods. We discuss two corresponding training strategies in the Appendix, thanks to the flexible rendering. Photometric stereo images offer two distinct advantages. First, photometric stereo images furnish additional photometric cues, thereby enhancing the capacity to recover fine-grained local details. Second, the PRM model demonstrates remarkable robustness to variations in the appearances of input images. For instance, it is capable of accurately reconstructing the geometry of images with glossy surfaces. Moreover, by utilizing mesh as our 3D representation, we are capable of utilizing differentiable PBR to produce intermediate shading variables such as albedo, specular light, and diffuse light maps, along with geometric cues like normals and depth. These variables provide multiple supervisions, including photometric supervision and geometric supervision for high-quality geometry reconstruction. Furthermore, PBR can better disentangle the specular component, making the geometry also be correctly recovered when the supervision images are characterized with glossy surfaces. To summarize, our contributions are listed as follows. • We introduce PRM, a model that is capable of reconstructing geometry with fine-grained local details and robust to variations in the appearance of input image by utilizing photometric stereo images as input and supervision. • To the best of our knowledge, we are the first to integrate split-sum approximation and mesh rasterization to render images online for LRM, offering significantly greater flexibility. • By utilizing mesh as the 3D representation, we ensure differentiable PBR for predictive rendering. This approach is advantageous for modeling reflective components and enables the incorporation of multiple supervisions for high-quality geometry reconstruction, significantly outperforming other models. 2 Related Work 2.1 Feed-forward 3D Generative Models Large-scale 3D assets ( Deitke et al., 2023 ) facilitate the training of generalizable reconstruction models. Recent works have focused on generating 3D objects using feed-forward models ( Hong et al., 2023 ; Xu et al., 2023 ; Wang et al., 2024 ; Xu et al., 2024b ; Li et al., 2023 ; Hu et al., 2024 ; Tang et al., 2024 ; Zhang et al., 2024 ) , demonstrating impressive results in terms of speed and quality. Specifically, Clay ( Zhang et al., 2024 ) utilizes occupancy for direct supervision. X-ray ( Hu et al., 2024 ) explores novel 3D representations by converting a 3D object into a series of surface frames at different layers. The LRM series ( Hong et al., 2023

Read the complete text