SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation
Zhenyu Jin, Wenjie Li, Zhanyu Ma, Heng Guo · 2025
Read SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation by Zhenyu Jin, Wenjie Li, Zhanyu Ma, Heng Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17316.txt.
Introduction
Zhenyu Jin, Wenjie Li, Zhanyu Ma, and Heng Guo present SpecGen, a method for generating spectral reflectance data from ordinary images, in a two thousand twenty-five paper. Photorealistic rendering needs to know how materials reflect light at each wavelength. Measured spectral data of that kind is scarce.
SpecGen starts from a single RGB photograph of a sphere. It generates the material's full spectral reflectance function. A network design called Spectral-Spatial Tri-plane Aggregation models responses across wavelengths and light directions. Training cleverly reuses plentiful RGB reflectance data to compensate for the shortage. Experiments showed accurate reconstruction from limited spectral data. The method beat earlier approaches on hyperspectral image reconstruction by a clear margin.
Catalog record
| CREATOR | Zhenyu Jin, Wenjie Li, Zhanyu Ma, Heng Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation Zhenyu Jin, Wenjie Li, Zhanyu Ma, Heng Guo Abstract Synthesizing spectral images across different wavelengths is essential for photorealistic rendering. Unlike conventional spectral uplifting methods that convert RGB images into spectral ones, we introduce SpecGen, a novel method that generates spectral bidirectional reflectance distribution functions (BRDFs) from a single RGB image of a sphere. This enables spectral image rendering under arbitrary illuminations and shapes covered by the corresponding material. A key challenge in spectral BRDF generation is the scarcity of measured spectral BRDF data. To address this, we propose the Spectral-Spatial Tri-plane Aggregation (SSTA) network, which models reflectance responses across wavelengths and incident-outgoing directions, allowing the training strategy to leverage abundant RGB BRDF data to enhance spectral BRDF generation. Experiments show that our method accurately reconstructs spectral BRDFs from limited spectral data and surpasses state-of-the-art methods in hyperspectral image reconstruction, achieving an improvement of 8 dB in PSNR. Codes and data will be released upon acceptance. \begin{overpic}[PDF/teaser.pdf] \end{overpic} Fig. 1 : From a single RGB image of a sphere ( left ), our method recovers the spectral BRDF of the corresponding material, allowing the spectral image rendering of arbitrary illuminations and shapes covered by the material under different wavelengths ( right ). See Fig. 5 for more spectral renderings at more diverse wavelengths. I Introduction Real light is composed of a spectrum of wavelengths, and photorealistic image synthesis requires capturing the wavelength-dependent reflectance properties of materials under varying illumination and viewing conditions. Spectral bidirectional reflectance distribution functions (BRDFs) constitute a crucial ingredient in this process, enabling the synthesis of hyperspectral images for remote sensing [ 1 , 2 ] , material analysis [ 3 ] , and virtual reality [ 4 ] , as well as for classification tasks [ 5 ] . However, acquiring high-quality spectral BRDF data is challenging, as the measurement process requires scanning a four-dimensional domain at high resolution—a tedious and time-consuming task [ 6 ] , leading existing spectral BRDF datasets to be scarce and limited in scale. To bypass the need for extensive spectral measurements, it is desired to develop a spectral BRDF generation method that can produce photorealistic spectral BRDFs from a single image input. Existing methods have attempted to address BRDF generation, but they primarily focus on RGB BRDFs. For example, Gokbudak et al. [ 7 ] proposes a hypernetwork to reconstruct measured RGB BRDF from sparse samples. ControlMat [ 8 ] and MatFuse [ 9 ] leverage diffusion models to generate spatially-varying BRDFs, including albedo, roughness, metallic, and normal maps, from a single flash RGB photo. These methods, though effective in RGB BRDF generation, cannot be directly extended to spectral BRDFs due to differences in BRDF representation and the lack of spectral BRDF data. On the other hand, another line of research explores spectral uplifting [ 10 , 11 , 12 , 13 ] , which generates spectral images from RGB inputs. These methods mainly focus on producing spectral images, making extracting the underlying spectral BRDFs from the RGB inputs challenging. As a result, they cannot transfer the appearance of spectral material to different shapes and illuminations, a capability that is crucial for downstream image editing tasks. In summary, despite significant progress in spectral image generation and RGB BRDF generation, spectral BRDF generation remains an open problem. The key challenge of spectral BRDF generation lies in the scarcity of spectral BRDF data. Currently, only 51 isotropic spectral BRDFs are publicly available [ 6 ] , which makes it difficult to train high-quality, generalizable models. In contrast, measured RGB BRDF datasets, such as MERL [ 14 ] , are relatively easy to collect and are available at a much larger scale. Our key insight is to design a compatible spectral BRDF representation, along with a training strategy that leverages large-scale RGB BRDF data to facilitate spectral BRDF generation. In this paper, we propose SpecGen, which, to the best of our knowledge, is the first network to generate spectral BRDFs from a single RGB image. The core of SpecGen is the Spectral-Spatial Tri-plane Aggregation (SSTA) module, which facilitates the mapping between RGB and spectral BRDFs. Specifically, spectral BRDFs can be represented as a 4D function of ({\theta}_{h},{\theta}_{d},{\varphi}_{d},\lambda) , where the spatial attributes ({\theta}_{h},{\theta}_{d},{\varphi}_{d}) —related to incident and outgoing directions—are shared between spectral and RGB BRDFs, while the spectral attribute \lambda (wavelength) is unique to spectral BRDFs. Inspired by the 4D function decomposition in dynamic neural radiance fields (x,y,z,t) , we decompose the spatial and spectral attributes into two tri-planes, encoding their corresponding latent features separately. Building upon this tri-plane representation, we design a training strategy that leverages large-scale RGB BRDF datasets to supervise the spatial tri-planes, thereby alleviating the data scarcity issue in the spectral domain. Furthermore, to effectively integrate RGB and spectral features, we propose an Adaptive Feature Fusion (AFF) module, which replaces traditional feature fusion methods such as the Hadamard product [ 15 ] . The AFF module enhances the model’s representational capability by adaptively combining multi-plane features. As shown in Fig. 1 , these dedicated designs enable our method to accurately generate spectral BRDFs and to facilitate downstream spectral rendering tasks. To summarize, our main contributions are as follows: • SpecGen is, to the best of our knowledge, the first spectral BRDF generation framework capable of generating photorealistic spectral BRDFs from a single RGB image. • We introduce the SSTA module, which alleviates the scarcity of spectral BRDF data by leveraging large-scale RGB BRDF datasets through a novel tri-plane representation of spectral and spatial BRDF attributes. • We present the AFF module, which dynamically selects and fuses RGB and spectral features to enhance the model’s representational capacity. Extensive experiments demonstrate that our method can reconstruct and generate high-quality spectral BRDFs, benefiting downstream tasks such as hyperspectral image rendering. II Related work As shown in Table I , we summarize existing methods for BRDF generation and spectral image reconstruction from RGB images. Since our method is based on neural implicit representations, we also briefly review relevant works on spectral BRDF generation and neural implicit representations. II-A BRDF Generation BRDF reconstruction and generation play a key role in advancing computer graphics and material analysis. Early methods, such as those based on CNNs [ 16 , 17 ] , extract spatially varying BRDFs (SVBRDFs) from images, but struggle with generalization and require separate training for each material. Later works utilize U-Net architectures [ 18 , 19 ] , which improve generalization performance but often fail to preserve fine details, particularly in highlight regions. To address this limitation, GAN-based methods [ 20 , 21 , 22 ] have gained popularity. For instance, SurfaceNet [ 20 ] and MaterialGAN [ 21 ] utilize GAN structures to generate realistic SVBRDFs from a single image. TileGen [ 22 ] extends MaterialGAN by training category-specific models, while PhotoMat [ 23 ] builds upon prior works by incorporating real-world datasets to further enhance visual quality. With the rise of diffusion models [ 24 ] , recent work has explored diffusion-based approaches for BRDF generation, such as ControlMat [ 8 ] and MatFuse [ 9 ] . These methods address the low-quality artifacts observed in GAN-based approaches, but suffer from slower