Digital Twin Generation from Visual Data: A Survey
Andrew Melnik, Benjamin Alt, Giang Nguyen, Artur Wilkowski, Maciej Stefańczyk, Qirui Wu, Sinan Harms, Helge Rhodin, Manolis Savva, Michael Beetz · 2025
Read Digital Twin Generation from Visual Data: A Survey by Andrew Melnik, Benjamin Alt, Giang Nguyen, Artur Wilkowski, Maciej Stefańczyk, Qirui Wu, Sinan Harms, Helge Rhodin, Manolis Savva, Michael Beetz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13159.txt.
Introduction
Andrew Melnik and nine colleagues survey how digital twins are now built from visual data, in a 2025 paper. A digital twin is a virtual three-dimensional replica of a real object or place. Twins serve robotics, media production, design, and construction, and cameras are their main source.
The survey weighs the main generation methods. Three-dimensional Gaussian splatting builds twins from photos with striking realism. Generative inpainting fills gaps, semantic segmentation labels parts, and foundation models supply broad visual understanding. Each route has clear strengths and limits. Standing challenges include occlusions, changing light, and scaling to large scenes. The authors map trends and open gaps for the field. Their overview is meant as an entry point for researchers who want to build twins from images and video.
Catalog record
| CREATOR | Andrew Melnik, Benjamin Alt, Giang Nguyen, Artur Wilkowski, Maciej Stefańczyk, Qirui Wu, Sinan Harms, Helge Rhodin, Manolis Savva, Michael Beetz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13159
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Digital Twin Generation from Visual Data: A Survey Andrew Melnik, Benjamin Alt, Giang Nguyen, Artur Wilkowski, Maciej Stefańczyk, Qirui Wu, Sinan Harms, Helge Rhodin, Manolis Savva, Michael Beetz Abstract This survey explores recent developments in generating digital twins from videos. Such digital twins can be used for robotics application, media content creation, or design and construction works. We analyze various approaches, including 3D Gaussian Splatting, generative in-painting, semantic segmentation, and foundation models highlighting their advantages and limitations. Additionally, we discuss challenges such as occlusions, lighting variations, and scalability, as well as potential future research directions. This survey aims to provide a comprehensive overview of state-of-the-art methodologies and their implications for real-world applications. Awesome list: https://github.com/ndrwmlnk/awesome-digital-twins Fig. 1: Generating Indoor Digital Twins from Visual Data. I Introduction Digital Twins (DT) 1 1 1 https://www.nvidia.com/en-us/glossary/digital-twin/ are virtual representations of products, processes, and facilities that enterprises use to design, simulate, and operate their physical counterparts. In the scope of this survey, we define Digital Twins as 3D models enriched with processes and semantic connections, similar to those in computer game environments and industrial CAD models. These models can range in accuracy from approximated 3D representations, like Digital Cousins [ 1 ] to highly detailed virtual replicas of real-world environments known as Digital Twins . Our focus will be on methodologies and understanding required to improve the accuracy of these 3D reconstructions and their associated processes using visual data, advancing from digital approximations to true Digital Twins. Recent advances in automatization of Digital Twin reconstruction from visual data provide opportunities for various domains, including robotics [ 2 , 3 ] , architecture, facility management, industrial fabrication, gaming, video stream compression, and virtual reality. Traditionally, the creation of Digital Twins has relied on specialized hardware such as LiDAR scanners or hand-crafted CAD models. Recent advances in computer vision and machine learning, such as 3D Gaussian Splatting (3DGS), NeRF, and diffusion models, have opened new possibilities for simplifying this process using widely accessible video data. Videos, captured with ubiquitous devices such as smartphones, offer a rich source of spatial and visual information that can be leveraged for Digital Twins generation. Another source of visual data is short videos generated by Video Diffusion Models (VDM) [ 4 ] , which often suffer from a lack of physical plausibility. These limitations of VDMs can be mitigated by generating and animating Digital Twins based on the synthesized video content. Moving forward, the next frontier in 3D representation lies in enriching these DT models by integrating both the physical [ 5 ] and semantic [ 6 ] properties of objects. Moreover, by analyzing demonstrations, such as a person interacting with a chest of drawers or a light switch, models can begin to absorb deeper contextual properties beyond mere geometry, like articulation of parts, hierarchical structure and connectivity, interaction properties, rules, associated processes, material properties, audio patterns, etc. This survey focuses on the generation of Digital Twins (Figure 1 ) from images and videos, exploring the methods, challenges, and innovations that enable this transformative process. We aim to address key questions, including how to accurately reconstruct indoor environments, integrate semantic information, and ensure scalability across diverse use cases. The goal is to present approaches that not only enhance the fidelity of digital twins but also reduce the barriers to their adoption. We also highlight practical applications and outline future directions to advance the field further. II Shape reconstruction II-A Mesh, CAD, 3DGS Digital shape representations are usually based on mesh models, CAD models, and 3D Gaussian Splatting (3DGS) - each having distinct advantages. Mesh representation is using interconnected polygons (triangles or quads), making it efficient for real-time rendering, gaming, and 3D printing, though it lacks precise parametric control. CAD representation employs mathematical curves (NURBS, B-Rep) to create highly accurate, editable, and feature-based models, making it ideal for engineering, manufacturing, and design applications. 3D Gaussian Splatting represents objects using point-based Gaussian functions with attributes like color, opacity, and density. This method excels in photorealistic, view-dependent rendering and is particularly useful for neural scene representations and volumetric graphics. Althoughugh mesh and CAD models are more structured and interpretable, 3DGS offers a continuous and memory-efficient alternative for immersive applications (see Table I ). The growing number of new methods for automated 3DGS reconstruction is establishing 3DGS as one of the key representations for approximating 3D scenes from visual input. TABLE I: Comparison of 3DGS , Mesh , and CAD models Feature 3D Gaussian Splatting Mesh Model CAD Model Representation Gaussian splats (point-based with density, color, and opacity) Polygonal (triangles/quads) Mathematical (NURBS, B-Rep) Precision Approximate (view-dependent, continuous representation) Approximate Highly precise Modifiability Hard to modify (requires re-optimization) Hard to modify Easy to modify (parametric) Usage Real-time rendering, neural scene representation, volumetric rendering 3D graphics, gaming, animation, 3D printing Engineering, manufacturing, design File Size Usually compact (but can grow with complexity) Usually smaller Larger due to parametric data Rendering Direct rendering via splatting (fast and efficient) Rasterization or ray tracing Mostly used for manufacturing, converted to mesh for visualization Conversion Difficult to convert to mesh or CAD Can be converted to CAD (but loses precision) Can be meshed or voxelized II-B Video to intermediate 3D representations COLMAP & Point Clouds COLMAP [ 7 ] is a general-purpose Structure-from-Motion (SfM) pipeline. SfM [ 8 ] , is a computer vision technique used to reconstruct a 3D representation of a scene from a set of 2D images taken from different viewpoints (Figure 2 b). The process involves detecting and matching features across images, reconstructing the motion of the camera, and using triangulation to recover the 3D structure of a scene [ 7 ] . The key part of SfM is determining both 3D geometry of the scene and the camera’s position and orientation relative to that geometry. SfM typically begins with feature detection, where key points such as edges, corners, and texture detected in each images using matching algorithms, ranging from classical techniques such as SIFT [ 9 ] to more recent deep learning-based approaches like SuperGlue [ 10 ] , R2D2 [ 11 ] , and LoFTR [ 12 ] , which utilize neural networks or transformer architectures to achieve SOTA performance. The corresponding features are then matched between the images, and by analyzing how key points move between the images, the algorithm determines the camera positions and orientations and camera intrinsic parameters. This allows for the reconstruction of 3D points using triangulation. To refine the points and camera parameters, SfM uses bundle adjustments [ 13 ] , reducing projection errors for higher accuracy. The result of this process is a sparse 3D point cloud. A point cloud is a collection of discrete points in 3D space that represents the surface geometry of an object or a scene. Each point has (X,Y,Z) coordinates, defining its position in space. Additional attributes such as color, intensity, and normal vectors can be stored with each point to enhance visualization and processing. These point clouds serve as the foundational data structure for many 3D modeling techniques, including mesh generation and virtual scene reconstruction. In some cases SfM is extended with Multi-View Stereo (MVS) to generate a dense point cloud that captures more details of the scene [ 14 ] . Fig.