3D Scene-Camera Representation with Joint Camera Photometric Optimization

Weichen Dai Kangcheng Ma, Jiaxin Wang, Kecen Pan, Yuhang Ming, Hua Zhang, Wanzeng Kong · 2025

Read 3D Scene-Camera Representation with Joint Camera Photometric Optimization by Weichen Dai Kangcheng Ma, Jiaxin Wang, Kecen Pan, Yuhang Ming, Hua Zhang, Wanzeng Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20979.txt.

Introduction

Weichen Dai and co-authors build scene maps that own up to the camera's own light flaws, in a study from 2025. Pics from sundry eyes bear bends of light: dim rims, hue casts, and more. A scene build that trusts raw pics bakes those flaws in as if they were walls and goods.

Their form learns two maps at once: the scene's glow field and the camera's light law, in and out. A depth rein keeps the scene from soak up light junk as form. The pair then trains as one, so each map stays in its lane. Builds came out clean where past builds smeared the flaws. A full map, they pitch, holds the eye as well as the seen.

Catalog record

CREATORWeichen Dai Kangcheng Ma, Jiaxin Wang, Kecen Pan, Yuhang Ming, Hua Zhang, Wanzeng Kong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20979

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D Scene-Camera Representation with Joint Camera Photometric Optimization Weichen Dai Kangcheng Ma, Jiaxin Wang, Kecen Pan, Yuhang Ming, Hua Zhang, Wanzeng Kong Abstract Representing scenes from multi-view images is a crucial task in computer vision with extensive applications. However, inherent photometric distortions in the camera imaging can significantly degrade image quality. Without accounting for these distortions, the 3D scene representation may inadvertently incorporate erroneous information unrelated to the scene, diminishing the quality of the representation. In this paper, we propose a novel 3D scene-camera representation with joint camera photometric optimization. By introducing internal and external photometric model, we propose a full photometric model and corresponding camera representation. Based on simultaneously optimizing the parameters of the camera representation, the proposed method effectively separates scene-unrelated information from the 3D scene representation. Additionally, during the optimization of the photometric parameters, we introduce a depth regularization to prevent the 3D scene representation from fitting scene-unrelated information. By incorporating the camera model as part of the mapping process, the proposed method constructs a complete map that includes both the scene radiance field and the camera photometric model. Experimental results demonstrate that the proposed method can achieve high-quality 3D scene representations, even under conditions of imaging degradation, such as vignetting and dirt. Figure 1 : Comparison of different methods using the images with photometric distortions. The input images contain severe photometric distortions. Our method effectively separates these distortions (The left image is affected by water droplets, and the right image by vignetting), leading to a more accurate reconstruction with higher fidelity. In each group, the bottom-left image visualizes the photometric distortion modeled by our camera model, the right-side images show results from different methods, with corresponding model names and PSNR values. 1 Introduction Representing the scene from multi-view 2D images is one of the crucial tasks in computer vision [ 40 , 33 ] , with extensive applications in visual effects [ 44 ] , mixed reality [ 25 ] , and robotics [ 46 , 42 ] . Recently, radiance fields—such as NeRF [ 33 ] , 3DGS [ 18 ] , and their extensions [ 39 , 22 ] —have achieved remarkable success in capturing scene and view-dependent effects. Numerous subsequent methods [ 26 , 34 , 38 , 29 , 14 ] have advanced the performance of radiance fields in various dimensions. These methods are designed to operate on, and are evaluated with, data captured by commodity cameras without taking advantage of knowing – or even being able to influence – the full image formation pipeline, overlooking the variations in light attenuation across different sensor positions due to distortion and contamination introduced by actual camera [ 21 , 15 ] . Typically, in addition to camera geometric model [ 29 ] , these methods recover the radiance fields without considering the photometric modeling of the camera itself. When photometric distortions are induced by the factors such as lens smudges, lenses with high distortion, and non-uniform response, the imaging quality of the camera is compromised [ 27 ] , which degrades the performance of the radiance fields, as shown in Fig. 1 . Fixed photometric distortions related to the camera in images may not significantly impact classic Structure-from-Motion (SfM) methods [ 40 ] , which primarily rely on modern keypoint detectors and descriptors [ 30 ] that are robust or invariant to arbitrary monotonic brightness changes. However, while radiance fields take light variance across different views into account and do not fully assume photometric consistency, the model capacity to accurately fit both scene radiance information and exclude the camera photometric distortions is limited. This limitation create challenges when fitting photometric values with adverse distortions, leading to erroneous geometric estimation and introducing incorrect radiance values. Therefore, radiance field methods should consider not only the geometric parameters [ 29 ] of the camera but also its photometric characteristics. Utilizing image restoration [ 45 , 28 , 27 ] techniques to address the adverse photometric distortions may present another approach. However, most existing methods focus primarily on single-frame images, relying on prior knowledge to identify defect areas and enhance them. Unfortunately, the supplementary scene information by the image restoration often leads to multi-view images that do not maintain geometric consistency, thereby undermining the foundational principles of radiance field representation. In this paper, we propose a 3D scene-camera representation method with joint camera photometric optimization that incorporates a comprehensive camera photometric model. The proposed method addresses both internal and external sources of camera photometric distortion, including sensor non-uniform response, lens vignetting, and imperfections such as smudges, thereby considering a full imaging formation. For external sources of camera photometric distortion, we refer to the depth-of-field(DoF) model [ 16 ] to reduce the impact of defocus. Based on the photometric model, a camera photometric representation method is designed using a shallow MLP to effectively capture and store camera photometric information. We optimize the camera photometric parameters and the radiance field of 3D Gaussians alternately, fixing one component while optimizing the other. To prevent convergence to local minima and to handle the limited constraints on photometric parameters in sparse-view settings, we further introduce a depth regularization. This regularization applies Gaussian constraints to the opacity distribution during rendering to optimize camera parameters, ensuring that the camera representation does not capture extraneous information. The main contributions of this work are as follows: • We propose a 3D scene-camera representation method to estimate scene radiance fields and the camera photometric parameters, considering a full imaging formation. • We propose a photometric representation that integrates defocus effects, modeling camera photometric parameters to enable the synthesis of high-quality images free from camera photometric distortions. • By modeling camera photometric representation, the proposed method facilitates the generation of high-fidelity images, demonstrating robustness across both a custom dataset -designed to capture diverse photometric distortions from multiple camera types - and real-world public datasets. 2 Related Work 2.1 3D Scene Representation Various representations, including point clouds [ 36 ] , meshes [ 37 ] , and radiance fields [ 11 ] , are employed to reconstruct 3D scenes from 2D images for a range of tasks, such as Novel View Synthesis (NVS) [ 17 , 23 ] , SLAM [ 5 ] , and Structure-from-Motion (SfM) [ 40 ] . Recently, neural radiance field (NeRF) methods [ 34 , 3 , 35 , 33 ] and 3DGS [ 18 ] have demonstrated significant advances in learning scene radiance fields, achieving high-fidelity representations. For camera geometric parameters, to reduce reliance on prior geometric camera parameters, many existing methods aim to minimize dependence on SfM by integrating pose estimation [ 29 , 4 , 12 ] , similar to SLAM [ 5 ] . However, most of these methods focus exclusively on geometric parameters. Without considering the photometric parameters within the full image formation pipeline, these methods may suffer from reduced performance in scenarios where images are affected by photometric degradation, such as lens attenuation (vignetting) and lens smudges or scratch. This issue is particularly relevant, as data used for radiance field reconstruction is often captured by consumer-grade cameras, which typically have limited control over camera conditions in real-world scenarios. Existing radiance field representation methods have preliminarily considered photometric factors involved

Read the complete text