Comprehensive Machine Learning Benchmarking for Fringe Projection Profilometry with Photorealistic Synthetic Data
Anush Lakshman S, Adam Haroon, Beiwen Li · 2026
Read Comprehensive Machine Learning Benchmarking for Fringe Projection Profilometry with Photorealistic Synthetic Data by Anush Lakshman S, Adam Haroon, Beiwen Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08900.txt.
Introduction
Anush Lakshman S, Adam Haroon, and Beiwen Li build a test set for a 3D scan method that uses striped light. The method shines stripe lines on a thing and reads how they bend to get its shape. Learn tools for it lack large, rich train sets. Real scans are slow to make, so progress has limped.
The team makes a fake set with true to life light and shade in a sim tool. It holds a vast set of stripe shots and depth maps for scores of things. They also give a full test plan so labs can grade tools the same way. A reader in 3D sight gets both data and a scale. New tools can now be trained and ranked with less guess work.
Catalog record
| CREATOR | Anush Lakshman S, Adam Haroon, Beiwen Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08900
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comprehensive Machine Learning Benchmarking for Fringe Projection Profilometry with Photorealistic Synthetic Data Anush Lakshman S, Adam Haroon, Beiwen Li Abstract Machine learning approaches for fringe projection profilometry (FPP) are hindered by the lack of large, diverse datasets and comprehensive benchmarking protocols. This paper introduces the first open-source, photorealistic synthetic dataset for FPP, generated using NVIDIA Isaac Sim, comprising 15,600 fringe images and 300 depth reconstructions across 50 diverse objects. We apply this dataset to single-shot fringe projection profilometry, where models predict 3D depth maps directly from individual fringe images without temporal phase shifting. Through systematic ablation studies, we establish optimal learning configurations for long-range (1.5-2.1 m) depth prediction. We first compare three data normalization strategies (raw, global normalized, individual normalized), finding that individual normalization, which decouples object shape from absolute scale, achieves 9.1 × \times better object reconstruction (16.20 mm vs 148.07 mm MAE) than raw depth. Our experiments also revealed that removing background fringe patterns catastrophically degrades performance by 2.8-7.3 × \times across all normalizations, demonstrating that background fringes provide essential spatial phase reference rather than acting as noise. We then evaluate six loss functions (L1, MaskedL1, HybridL1, RMSE, MaskedRMSE, HybridRMSE), identifying Hybrid L1 loss ( α \alpha =0.7) as optimal, improving object error by 10% over baseline RMSE (14.54 mm vs 16.20 mm MAE). Using the optimal configuration (individual normalization + Hybrid L1 α \alpha =0.7), we benchmark four architectures: UNet achieves best performance (14.54 mm object MAE, 17.88 mm object RMSE), outperforming Hformer by 52%, ResUNet by 60%, and Pix2Pix by 91%. Critically, Pix2Pix’s counterintuitive failure reveals fundamental misalignment between adversarial training and metrological objectives. Even the best model’s 14.54 mm error represents 18% of object with an 80 mm depth range, far from FPP’s sub-millimeter capabilities. The modest 1.9 × \times performance gap between best and worst architectures confirms that the fundamental limitation is information deficit, not model design: single fringe images lack sufficient information for accurate depth recovery without explicit phase information. This work provides standardized evaluation protocols, a robust benchmark dataset, and systematic evidence that direct fringe-to-depth mapping is fundamentally limited, motivating hybrid approaches combining traditional phase-based FPP with learned refinement. Our dataset is available at https://huggingface.co/datasets/aharoon/fpp-ml-bench , and our code is available at https://github.com/AnushLak/fpp-ml-bench . a]Department of Mechanical Engineering, Iowa State University, Ames, Iowa, USA b]College of Engineering, University of Georgia, Athens, Georgia, USA *]These authors contributed equally. keywords Fringe projection profilometry, machine learning, synthetic data, deep learning, 3D reconstruction, structured light, NVIDIA Isaac Sim, benchmarking † † authorinfo: Send correspondence to Adam Haroon: E-mail: [email protected] 1 Introduction and Related Work Fringe projection profilometry (FPP) has emerged as a critical non-destructive technology in robotic scanning [ 8 ] , manufacturing inspection [ 11 ] , and 3D printing optimization [ 12 ] , offering high-precision surface measurements with submillimeter accuracy [ 31 , 6 ] . While traditional FPP uses multi-step phase-shifting algorithms requiring sequential pattern capture, machine and deep learning (ML and DL) offers possibilities for single-shot reconstruction enabling real-time applications [ 34 , 30 , 32 , 33 , 23 , 9 , 1 ] . These approaches have shown promise in phase unwrapping [ 24 ] , fringe denoising [ 27 ] , and depth regression [ 34 ] . However, progress is hindered by three fundamental challenges. First, unlike computer vision benchmarks such as ImageNet [ 4 ] or COCO [ 14 ] , FPP lacks large-scale datasets with perfect ground truth and standardized evaluation protocols. Moreover, obtaining perfect 3D ground truth geometry for model training remains challenging as measurement systems introduce their own errors. To address these issues, synthetic data generation through virtual twins has proven powerful for optical metrology [ 16 , 3 ] using Blender [ 32 ] , Unity [ 22 ] , or MATLAB [ 29 ] . But these systems either require pre-calibrated physical setups or provide simplified optical models that do not reflect the complexities of real-world systems, such as over-saturation, occlusion, and subsurface scattering. From a machine learning perspective, the field lacks systematic ablation studies to establish best design practices for single-shot FPP. Each paper employs different data representations, some normalize depth to [0,1] [ 5 ] , others use raw millimeter values [ 9 ] , while some convert to meters [ 15 ] , without justifying these choices or evaluating alternatives. Similarly, loss functions vary arbitrarily across studies: L1 [ 25 ] , L2 [ 9 ] , cross-entropy [ 13 ] , SSIM [ 23 ] , or hybrid combinations [ 33 ] , with no systematic comparison of their effectiveness. This inconsistency makes it difficult to determine whether performance differences stem from architectural innovations, data preprocessing choices, or simply hyperparameter tuning. Hence, there is a need for comprehensive ablation studies to isolate the impact of each design decision. This will help the community to clearly establish whether observed errors in prediction are fundamental constraints of single-shot reconstruction or merely suboptimal configurations. To address these issues, we build on VIRTUS-FPP [ 7 ] , which introduced the first physics-based virtual FPP system with end-to-end camera-projector modeling in NVIDIA Isaac Sim, to present a systematic machine learning benchmarking framework. Our contributions include: • First open-source, photorealistic synthetic dataset for FPP, comprising 15,600 fringe images and 300 depth reconstructions across 50 diverse objects with perfect ground truth geometry at 1.5-2.1 m measurement range • Comprehensive three-phase ablation study establishing optimal learning configurations: – Phase 1 - Normalization : Individual normalization achieves 9.1 \times improvement over raw depth (16.20 mm vs 148.07 mm object MAE) by decoupling shape from scale – Phase 1b - Background ablation : Counterintuitive finding that removing background fringes degrades performance 2.8-7.3 \times , proving background provides essential spatial phase reference – Phase 2 - Loss functions : Hybrid L1 loss ( \alpha =0.7) achieves 10% improvement (14.54 mm object MAE), while masked losses catastrophically fail due to scale drift – Phase 3 - Architectures : UNet outperforms Hformer (52%), ResUNet (60%), and Pix2Pix (91%) with optimal configuration • Empirical evidence that reconstruction errors (14.54-27.73 mm object MAE, representing 18-35% of objects with an 80 mm depth range) expose fundamental limitations of direct fringe-to-depth mapping without explicit phase information: networks learn coarse shape priors rather than precise geometry. This causes a 1.9 \times performance gap across diverse architectures confirming information deficit, not model design, as the primary constraint • Standardized benchmarking protocols and open-source dataset enabling systematic comparison and advancement of learning-based FPP approaches 2 Virtual Fringe Projection Profilometry The VIRTUS-FPP [ 7 ] used for benchmarking, is built in NVIDIA Isaac Sim, integrating OptiX ray tracing for or photorealistic rendering, PhysX for physics, and Universal Scene Description (USD) for 3D composition. This section is structured as follows: Section 2.1 discusses the configuration of the FPP system used for data acquisition and Section 2.2 elaborates the calibration process of the constructed virtual system. 2.1 System Configuration The virtual system consists of a calibrated camera-projector pair (Table 1 ). The camera uses Isaac Sim’s pinhole primitive (960 \times 960 resolution, 50 cm focal length), while the projector is modeled using a rectangular light source (0.625 m \times 0.5 m, 40 nits) with texture projection. The projector is positioned