Exploring Dynamic Novel View Synthesis Technologies for Cinematography

Adrian Azzarelli, Nantheera Anantrasirichai, David R Bull · 2024

Read Exploring Dynamic Novel View Synthesis Technologies for Cinematography by Adrian Azzarelli, Nantheera Anantrasirichai, David R Bull (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17532.txt.

Introduction

Filmmakers can now reshoot a moving scene from angles never captured, thanks to methods that rebuild scenes in 3D from footage. Adrian Azzarelli and co-authors survey those dynamic view synthesis tools in this 2024 paper, with an eye to cinema use. Two families dominate: neural radiance fields and Gaussian splatting, both able to model real scenes in motion.

The prize is shots that sets or budgets would block: smooth floating camera moves, virtual re-shoots, and slow-motion passes. The paper compares models to ease the choice between them for a given production. A short montage filmed with several models shows what each does well. The authors frame the tools as new grammar for camera work, not just effects.

Catalog record

CREATORAdrian Azzarelli, Nantheera Anantrasirichai, David R Bull
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17532

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Dynamic Novel View Synthesis Technologies for Cinematography Adrian Azzarelli, Nantheera Anantrasirichai, David R Bull Abstract . Novel view synthesis (NVS) has shown significant promise for applications in cinematographic production, particularly through the exploitation of Neural Radiance Fields (NeRF) and Gaussian Splatting (GS). These methods model real 3D scenes, enabling the creation of new shots that are challenging to capture in the real world due to set topology or expensive equipment requirement. This innovation also offers cinematographic advantages such as smooth camera movements, virtual re-shoots, slow-motion effects, etc. This paper explores dynamic NVS with the aim of facilitating the model selection process. We showcase its potential through a short montage filmed using various NVS models. 1. Introduction Novel view synthesis (NVS) creates new images and videos of a scene from viewpoints that are not captured in the original data by utilizing a virtual representation of a real 3D scene. The recent success of learning-based techniques like NeRF and GS, has enabled the production of high-quality volumetric scenes from existing image and video data. This offers significant advantages in cinematography: (1) enabling smooth camera movement without requiring gimbals or dollies; (2) supporting virtual re-shoots, thus eliminating the need for expensive in-person re-shoots; (3) producing slow-motion effects without requiring special equipment; (4) altering scene geometry and aesthetics without manual intervention, reducing the burden on visual effects and compositing artists ( Yuan et al., 2022 ; Wang et al., 2022 ; Sun et al., 2022 ; Song et al., 2023b ; Kuang et al., 2023 ; Bao et al., 2023 ; Chen et al., 2023 ; He et al., 2024 ) ; and (5) introducing/correcting camera-based artefacts like de-blurring ( Wang et al., 2023d ) and lens distortions ( Kim et al., 2024 ; Xian et al., 2023 ; Moreau et al., 2022 ) . More creative uses have also been explored, like cinematic style transfer ( Wang et al., 2023a ) , the discovery of new cinematographic shots ( Skartados et al., 2024 ) and modeling live sport for action-replay ( Lewin et al., 2023 ) . In Figure 1 , we demonstrate this by training a static NeRF toy scene using only 36 images to synthesize a fly-through shot that would otherwise require special equipment. Figure 1 . Fly-through shot on a toy-scene The main figure is an 3-D vector illustration of a toy train surrounded by images used for training. A synthesized camera path and a novel rendering camera of a fly-through shot are shown in green and red color, respectively. The bottom subfigure shows several frames of the rendered fly-through shot. Considering the primary goal in cinematography is to capture scenes or assets in motion, our focus is on dynamic NVS research. We explore these new technologies and evaluate their meaning and effectiveness in relation to cinematographic content acquisition. We specifically explore: (1) Dynamic Representations (2) Dynamic scenes vs Animate-able human assets (3) Data Acquisition Subsequently, we film a short montage captured using various static and dynamic NVS models to showcase their potential in cinematographic production. 2. Technical Background Figure 2 illustrates the main steps involved in generating a neural 3D representation to achieve NVS. A collected image dataset first requires calibration to determine the position and rotation of each image with respect to the world coordinate space. Then, for each view in the training set we sample the in-frustum volumetric space and pass the position and viewing direction of each volume to a neural net that estimates the color and density of each volume-sample. Finally, we render the in-frustum volumes with respect to depth and backpropagate a loss by evaluating the predicted images against the ground truth images. Figure 2 . An illustration of the classical NVS pipeline. The learnable parts of the pipeline are highlighted in red The image contains four columns. The first column illustrates the data collection and camera calibration process, where images of a real set of collected and the position and orientation of each image is determined. The second column illustrates the volume sampling process where volume segments along a 3-D ray are sampled leading to determining the origin and viewing direction of each sample. The third column illustrates the inference step where origin and direction are passed into an MLP and a color and density value is estimated. The fourth column consists of rendering the ray-colors for each pixel-ray and propagating the loss (comparing the predicted pixel to a ground truth value) to the camera calibration, volume sampling, inference and rendering steps. The predominant challenge is how best to represent the volumetric space and achieve view-dependent effects, e.g. reflections from non-Lambertian materials. Addressing this, the two contributions that have shaped the NVS landscape are NeRF and GS, which are discussed below and accompanied by Figure 3 . (a) Neural Radiance Fields sample in-frustum volumes to estimate a color and density value for each sample projected along a pixel-ray (b) Gaussian Splatting uses a point cloud to estimate the color, density and covariance matrix (scale and rotation) of each 3D Gaussian point, which are rendered via tile-splatting Figure 3 . An illustration of the original NeRF and GS representations Two subfigures are shown. The first subfigure illustrates the NeRF approach, where a ray is projected from a pixel belonging to a specific camera/view and the global position of the volume segments along the ray are used to estimate the color of the pixel. The second subfigure illustrates the GS approach, where a distribution of gaussian points appear in front of a camera view and for each point the color, density, scale and rotation are predicted. 2.1. Neural Radiance Fields Mildenhall et al. (2021) propose the NeRF representation using an implicit coordinate multilayer perceptron (MLP) \Theta(\cdot) , defined as follows. \Theta(\mathbf{r_{i}},\mathbf{d}_{a,b})\rightarrow(\mathbf{c}_{i},\sigma_{i}), takes the position \mathbf{r}_{i}=[x,y,z]_{i} along a ray \mathbf{r} , and viewing direction \mathbf{d}_{a,b} of a volume i to predict the color \mathbf{c}_{i} and density \sigma_{i} . As each ray corresponds to a pixel (a,b) in an image \mathbf{H}(\mathbf{r}) with a focal point at \mathbf{o} , the relationship between \mathbf{r_{i}} and \mathbf{d}_{a,b} can be written as \mathbf{r}_{i}=\mathbf{o}+t_{i}\mathbf{d}_{a,b}. This enables learning view-dependent effects. Consequently, an image can be rendered with (1) \hat{\mathbf{H}}(\mathbf{r})=\sum^{t_{far}}_{i={t_{near}}}(1-\exp(-\sigma_{i}\delta_{i}))\mathbf{c_{i}}\exp({\sum^{i-1}_{j=t_{near}}-\sigma_{j}\delta_{j}}), where \delta_{i} is the width of each ray-volume sample. 2.2. Gaussian Splatting Kerbl et al. (2023) propose the GS representation, where a differentiable point cloud is used to model the position of 3D Gaussian volumes, \mathbf{x}_{i}\in\mathbf{x} , in 3D space using, (2) G(\mathbf{x})=exp(-\frac{1}{2}(\mathbf{x})^{T}\Sigma^{-1}(\mathbf{x})), where (3) \Sigma^{\prime}=JW\Sigma_{i}W^{T}J^{T} is a 3\times 3 covariance matrix dependent on the viewing transform W and the Jacobian of the affine projection transform J . In GS, \Sigma^{\prime} is simplified into a 2\times 2 matrix by ignoring the third row and column. For each point \mathbf{x}_{i} , a rotation R_{i} and scale S_{i} are approximated with an MLP allowing for \Sigma_{i}=R_{i}S_{i}S_{i}^{T}R_{i}^{T} . The color \mathbf{c}_{i} and density \sigma_{i} of each sample are determined using a separate MLP. This forms the basis of the GS representation, which is rendered with a differentiable tile-splatting strategy where an image, with properties W and J , is split into 16\times 16 tiles, the Gaussians G(\mathbf{x}) are ordered with respect to depth and finally rendered using the equation: (4) \hat{\mathbf{H}}(\mathbf{x})=\sum_{i\in N}\mathbf{c}_{i}\alpha_{i}\prod^{i-1}_{j=1}(1-\alpha_{j}), where \alpha_{i} is given

Read the complete text