Twinner: Shining Light on Digital Twins in a Few Snaps

aaaaaaaaaaaa Jesus Zarzar, Tom Monnier, Roman Shapovalov aaaaaaaaaaaa, Andrea Vedaldi, David Novotny · 2025

Read Twinner: Shining Light on Digital Twins in a Few Snaps by aaaaaaaaaaaa Jesus Zarzar, Tom Monnier, Roman Shapovalov aaaaaaaaaaaa, Andrea Vedaldi, David Novotny (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08382.txt.

Introduction

Jesus Zarzar and co-authors present a model that rebuilds a full digital twin of an object from just a few photos, in a paper from 2025. The catalog creator field is padded with stray letter strings, but the head byline is clean. The model recovers shape, surface material, and the lighting of the scene together.

A memory saving grid transformer keeps the computation in check. Since real training examples with known materials are scarce, the team trained on a large synthetic set of textured objects under varied light. A physics based shading step then tuned the model on real photos. On a standard real world test, quality beat fast rival methods and matched much slower per scene approaches.

Catalog record

CREATORaaaaaaaaaaaa Jesus Zarzar, Tom Monnier, Roman Shapovalov aaaaaaaaaaaa, Andrea Vedaldi, David Novotny
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08382

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Twinner: Shining Light on Digital Twins in a Few Snaps aaaaaaaaaaaa Jesus Zarzar, Tom Monnier, Roman Shapovalov aaaaaaaaaaaa, Andrea Vedaldi, David Novotny Abstract We present the first large reconstruction model, Twinner, capable of recovering a scene’s illumination as well as an object’s geometry and material properties from only a few posed images. Twinner is based on the Large Reconstruction Model and innovates in three key ways: 1) We introduce a memory-efficient voxel-grid transformer whose memory scales only quadratically with the size of the voxel grid. 2) To deal with scarcity of high-quality ground-truth PBR-shaded models, we introduce a large fully-synthetic dataset of procedurally-generated PBR-textured objects lit with varied illumination. 3) To narrow the synthetic-to-real gap, we finetune the model on real life datasets by means of a differentiable physically-based shading model, eschewing the need for ground-truth illumination or material properties which are challenging to obtain in real life. We demonstrate the efficacy of our model on the real life StanfordORB benchmark where, given few input views, we achieve reconstruction quality significantly superior to existing feed-forward reconstruction networks, and comparable to significantly slower per-scene optimization methods. Figure 1 : We present Twinner , the first large reconstruction model capable of predicting a scene’s illumination as well as an object’s geometry and material properties from a few posed images. This enables tasks such as realistic relighting of objects in novel scenes in a few seconds by replacing costly per-scene optimizations with a single forward pass. 1 Introduction The creation of digital twins, i.e . , the conversion of real-world objects into virtual 3D assets, is an important problem in virtual and augmented reality, gaming, conservation, education, advertisement, and general 3D content creation. In the last few years, new techniques such as NeRF [ 31 ] and 3D Gaussian Splatting [ 24 ] , along with their their feed-forward counterparts [ 18 , 63 , 49 , 60 ] , have significantly improved the quality, speed, and generality of digital twinning. However, most of theses techniques still bake illumination into the scene’s appearance as radiance. As a consequence, the resulting assets cannot be relit and look out-of-place when visualized under different lighting conditions. To address this limitation, one must also reconstruct the field of Bidirectional Reflectance Distribution Functions (BRDF), which describe the optical properties of the materials. Estimating the BRDF directly is notoriously challenging due to its high dimensionality [ 3 ] and is usually done by means of 3D scanners that take thousands of images controlling for viewpoint and illumination [ 65 , 42 , 35 , 7 , 6 , 5 , 4 ] . This is often impractical, so authors have considered “casual” reconstruction setups, where the object is captured by a conventional camera in uncontrolled conditions. Examples include NVDiffRec [ 34 ] , NeRD [ 8 ] and NeRFactor [ 59 ] , which simultaneously estimate the illumination, in the form of an environment map, and a simplified BRDF, parameterized using Physically-Based Rendering (PBR) with albedo, roughness and metallicity [ 10 ] . These quantities are recovered by minimizing the photometric reconstruction error, comparing the output of a differentiable PBR renderer to the input images. However, this optimization is slow and ill-posed, and requires the introduction of handcrafted priors for regularization. In this paper, we propose a new method, Twinner , a feed-forward network that outputs shape, PBR textures, and environment illumination. This network is fast, performs well, requires only a small number of images as input, and learns empirically the necessary reconstruction priors. Siddiqui et al. [ 45 ] recently introduced a similar predictor, MetalLRM, as an extension of the LightplaneLRM [ 11 ] . MetalLRM requires supervision with a large dataset of 3D assets with high-quality PBR textures which, unfortunately, are not abundant in existing synthetic datasets like Objaverse [ 14 ] . Moreover, even if a large synthetic dataset were available, the synthetic-to-real gap between the artist-created training assets and the target domain of real input images makes MetalLRM unsuitable for digital twinning. With our new Twinner, we address the scarcity of high-quality training PBR assets and the train/test domain shift. First, we ask whether real images in datasets like CO3Dv2 [ 40 ] can be used to supervise the model. Although these datasets lack PBR information, they are abundant and match the target distribution of real images we aim to digitally twin. In order to use this data, we propose to minimise the photometric error similar to optimization-based models, but with the key difference that the 3D shape, PBR materials, and environment light are all predicted by a large feed-forward transformer [ 19 , 45 ] . During training, each scene is rendered with a differentiable PBR shader that accepts the predicted environment light and the predicted PBR-textured shape. The latter then admits indirect PBR and illumination supervision via photometric losses between the renders and the corresponding real ground-truth images. In addition, we consider a second approach to tackle the data scarcity where, inspired by Zeroverse [ 58 ] , we procedurally generate PBR-textured objects lit with a known set of environment maps. While the resulting objects are not realistic, their key advantage is that they come with clean ground-truth PBR and illumination information, which can be used to supervise the corresponding predictors directly instead of indirectly via the photometric loss. Besides addressing data scarcity, we also improve the architecture of the reconstruction model. We believe in the virtues of a simple and direct voxel-grid representation of the object [ 47 ] , but note that predicting a number of tokens cubic in the voxel grid resolution is intractable with a transformer. Reducing the number of predicted tokens is the primary reason why recent models [ 19 , 45 ] factor the voxel grid as a triplane. The question, then, is how to retain the benefits of the direct voxel grid representation and still make computations feasible. We solve this problem by introducing the novel tricolumn representation, where each feature vector (column) in a triplane is obtained by stacking the corresponding cells in the underlying voxel grid. In this manner, the number of tokens becomes quadratic in the voxel grid size, while each voxel is still represented explicitly. We further augment the network to predict a representation of the environment map, capturing illumination. We demonstrate the efficacy of Twinner model on the StanfordORB benchmark [ 26 ] , consisting of casual captures of real-life objects. We demonstrate reconstruction quality superior to existing feed-forward predictors and comparable to optimization-based methods, while using only a very small number of input images of the object and operating much faster. We also obtain state-of-the-art reconstruction of the environment maps, achieving an improvement of 28% in the predicted illumination’s angular error over existing methods. Figure 2 : Overview of Twinner. The input views are processed together with their foreground masks and camera poses by an image tokenizer. The resulting tokens are processed by a Diffusion Transformer (DiT) model to predict a 3D volumetric representation of the scene, and a second DiT model to predict the scene’s illumination. The 3D representation is then rendered to obtain images of the scene’s material properties, normals, and opacity from the target’s point of view.

Read the complete text