BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video
Yu Hong, Yize Wu, Zhehao Shen, Chengcheng Guo, Yuheng Jiang, Yingliang Zhang, Qiang Hu, Jingyi Yu, Lan Xu · 2025
Read BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video by Yu Hong, Yize Wu, Zhehao Shen, Chengcheng Guo, Yuheng Jiang, Yingliang Zhang, Qiang Hu, Jingyi Yu, Lan Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-08297.txt.
Introduction
Yu Hong and co-authors present BEAM, a pipeline that turns multi-camera footage of a moving person into a 3D video asset that can be relit, in a 2025 paper. Volumetric video lets viewers move around a captured scene, but most captures are stuck with the lighting of the day. Their pipeline rebuilds shape and surface traits using Gaussian splats, a fast rendering primitive.
Steps recover the performer's geometry over time, then add surface finish, shading, and color with the help of a diffusion model and a custom ray tracer. The finished asset drops into standard graphics pipelines for real-time or film-quality rendering. The authors aim at more flexible storytelling and shared virtual presence.
Catalog record
| CREATOR | Yu Hong, Yize Wu, Zhehao Shen, Chengcheng Guo, Yuheng Jiang, Yingliang Zhang, Qiang Hu, Jingyi Yu, Lan Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-08297
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video Yu Hong, Yize Wu, Zhehao Shen, Chengcheng Guo, Yuheng Jiang, Yingliang Zhang, Qiang Hu, Jingyi Yu, Lan Xu Abstract . Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based human performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization. Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755233 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Rendering † † cc-license: by Figure 1. We present BEAM, a novel pipeline that bridges 4D Gaussians with accurate physically-based rendering to produce relightable volumetric videos, delivering immersive and realistic experiences on platforms such as VR. 1. Introduction Volumetric video captures dynamic 3D scenes from multiple angles, allowing interactive viewing from any perspective. This technology is crucial for creating immersive experiences in virtual and augmented reality, enhancing storytelling, education, cultural preservation, and telepresence with lifelike, interactive content. However, traditional volumetric video is often limited by fixed lighting conditions captured during recording, which can clash with dynamic or virtual environments, reducing realism and flexibility. Relightable volumetric video overcomes this limitation by enabling post-capture relighting. This allows for seamless integration into dynamic lighting environments and offers creative control over visual aesthetics. The prevailing workflow ( Collet et al., 2015 ; Dou et al., 2016 ; Lawrence et al., 2024 ; Garon et al., 2016 ) for producing relightable volumetric videos in the industry still relies on tracked mesh sequences and texture videos, which can be seamlessly integrated into standard CG pipelines to support relighting under various lighting conditions. However, the intricate reconstruction process often introduces artifacts such as holes and noise, and the quality of relighting remains constrained, frequently resulting in visible imperfections. Neural advancements ( Srinivasan et al., 2021 ; Zhang et al., 2021 ; Yao et al., 2022 ) focus on enabling relighting capabilities using neural factorization within implicit MLPs representations. However, these approaches often face challenges in balancing training efficiency, rendering speed, and output quality, ultimately failing to deliver satisfactory results. Recently, 3D Gaussian Splatting (3DGS) ( Kerbl et al., 2023 ) , an efficient point-based representation, has achieved photo-realistic rendering at unprecedented frame rates. While dynamic variants ( Jiang et al., 2024b ; Li et al., 2024a ; Wang et al., 2024 ) can produce high-quality volumetric videos, they fail to produce the detailed geometry necessary for essential operations like relighting. Although efforts ( Gao et al., 2025 ; Shi et al., 2023 ; Liang et al., 2024 ; Jiang et al., 2024d ) have been made to integrate physically-based rendering into the 3DGS pipeline, these methods are often computationally expensive and limited to static scenarios. These limitations severely restrict their applicability in industrial workflows, hindering the efficient production of 4D content. In this paper, we introduce BEAM, a novel pipeline that bridges 4D Gaussians with accurate physically-based rendering (PBR) for producing relightable volumetric videos from multi-view RGB footage. Our key idea is to robustly recover detailed geometry of human performances and decouple the PBR properties (e.g., ambient occlusion, roughness, and base color) using a carefully selected suite of techniques, i.e., rasterization, performance tracking, and ray tracing, all within a Gaussian-based paradigm. As a result, BEAM enables lifelike dynamic scenes that can be seamlessly and CG-friendly integrated into various platforms under diverse lighting (see Fig. 1 ). We first recover detailed and spatial-temporally consistent geometries from multi-view video input, which organically combines the Gaussian-based performance tracking ( Jiang et al., 2024b ) with the geometry-aware Gaussian rasterization ( Zhang et al., 2024a ) . While the former excels at motion tracking and the latter at static geometry recovery, we unify them in a coarse-to-fine optimization framework. Specifically, we employ coarse joint Gaussians to track non-rigid motion and dense skin Gaussians to preserve intricate geometry details. We adopt a robust optimization process that integrates normal consistency, photometric consistency, and temporal regularization to enhance geometric accuracy and smoothness. This enables accurate depth and normal recovery from the dense Gaussians using the geometry-aware rasterizer ( Zhang et al., 2024a ) , providing a robust foundation for material decomposition and relighting. We further decouple the dense 4D Gaussians to recover detailed material properties, enabling high-quality physically-based rendering grounded in the rendering equation ( Kajiya, 1986 ) and simplified Disney BRDF ( Burley and Studios, 2012 ) . Assuming human-centric scenes with negligible metallic components, we focus on accurately associating roughness, ambient occlusion (AO), and base color properties with the Gaussians, ensuring realistic and adaptable rendering under diverse lighting conditions. To achieve this, we adopt a step-by-step approach to disentangle these properties. Specifically, we first generate a roughness texture using the material diffusion module in previous work ( Zhang et al., 2024b ) with multi-view conditioning, which is associated with the dense Gaussians through UV projection. Then, for the AO and base color, we adopt a 2D-to-3D strategy, where these attributes are estimated in the input views to bake 2D material maps, and then optimized into the corresponding dense Gaussians in the 3D space. This strategy effectively reduces noise and smooths the disentanglement to improve relighting quality. For further 2D AO and base color decomposition, the lighting environment during capturing can be estimated using an off-the-shelf tool ( New House Internet Services B.V., 2025 ) , while the geometry attributes and roughness are obtained in previous stages. Thus, by carefully re-examining and simplifying the rendering equation ( Kajiya, 1986 ) , we identify a critical insight: both 2D AO and base color can be accurately derived by accumulated visibility information for specific points along specific directions during ray tracing. We tailor the Gaussian-based ray tracer ( Moenne-Loccoz et al., 2024 ) to compute such visibility, with a novel alpha blending strategy based on the dense Gaussians. This strategy efficiently captures visibility information, forming the foundation for estimating AO and base color maps in the input viewpoints. Figure 2. We propose a novel BEAM method to produce Gaussian sequences. We first use tracking results of joint Gaussians and a normal regularizer to obtain our 4D Gaussians with consistent geometry. Then we infer roughness using a