Look Ma, No Ground Truth! G round- T ruth- F ree Tuning of Structure from Motion and Visual SLAM

Alejandro Fontan, Javier Civera, Tobias Fischer, Michael Milford, Queensland University of Technology, Universidad de Zaragoza · 2024

Read Look Ma, No Ground Truth! G round- T ruth- F ree Tuning of Structure from Motion and Visual SLAM by Alejandro Fontan, Javier Civera, Tobias Fischer, Michael Milford, Queensland University of Technology, Universidad de Zaragoza (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01116.txt.

Introduction

Map builds from shots need a true map to be tuned, and true maps are dear, slow, or nil in most fields. Alejandro Fontan, Javier Civera, Tobias Fischer, and Michael Milford cut that cord in a 2024 study. Their grade needs no true map: it probes how firm a build's ware stands when the shots are stirred with a hair of noise.

Firm wares earn high grades; frail wares fail as the stir grows. The grades track the true map grades of the old tests with care. Tune by the new grade, and builds set their own dials with no map in hand. The team pitch the grade for fields where no true map can be made.

Catalog record

CREATORAlejandro Fontan, Javier Civera, Tobias Fischer, Michael Milford, Queensland University of Technology, Universidad de Zaragoza
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Look Ma, No Ground Truth! G round- T ruth- F ree Tuning of Structure from Motion and Visual SLAM Alejandro Fontan, Javier Civera, Tobias Fischer, Michael Milford, Queensland University of Technology, Universidad de Zaragoza Abstract Evaluation is critical to both developing and tuning Structure from Motion (SfM) and Visual SLAM (VSLAM) systems, but is universally reliant on high-quality geometric ground truth – a resource that is not only costly and time-intensive but, in many cases, entirely unobtainable. This dependency on ground truth restricts SfM and SLAM applications across diverse environments and limits scalability to real-world scenarios. In this work, we propose a novel ground-truth-free ( GTF ) evaluation methodology that eliminates the need for geometric ground truth, instead using sensitivity estimation via sampling from both original and noisy versions of input images. Our approach shows strong correlation with traditional ground-truth-based benchmarks and supports GTF hyperparameter tuning. Removing the need for ground truth opens up new opportunities to leverage a much larger number of dataset sources, and for self-supervised and online tuning, with the potential for a data-driven breakthrough analogous to what has occurred in generative AI. 1 Introduction ATE \log_{10} Max. BA \boldsymbol{e_{r}} GTF ATE Figure 1 : Illustration of ground-truth-free tuning in GLOMAP . The green line fits Absolute Trajectory Error (ATE) results of GLOMAP as we vary one of its hyperparameters, specifically the maximum reprojection error for inliers in the Bundle Adjustment (Max. BA e_{r} ) in radians. Note that, while the Max. BA e_{r} default value in GLOMAP is 10^{-2} , leading to an ATE of 1.3 mm , the optimal one for this particular sequence is \simeq 10^{-3} , for which the ATE improvement is \simeq 40\% , reaching 0.8 mm . Now look at our proposed GTF ATE curve in pink , which without ground truth , is able to mimic the relative GLOMAP performance for different values of the hyperparameter, and hence also discerning its optimal setup. SVO [ 30 , 31 ] LSD / DSO / D3VO [ 24 , 25 , 103 , 104 ] ORB-SLAM [ 66 , 65 , 14 ] Colmap / Glomap [ 79 , 68 ] Tartan-VO [ 98 ] Droid-SLAM [ 90 ] NICE / NICER SLAM [ 113 , 114 ] PVO [ 105 ] AnyFeature [ 29 ] MonoGS [ 59 ] DPV-SLAM [ 91 , 56 ] ICL-NUIM [ 40 ] ATE ATE - - - - - - - - ATE Replica [ 86 ] - - - - - - Map. - - Pho. - VKITTI [ 32 , 12 ] - - - - - - - ATE - - - TartanAir [ 97 ] - - - - ATE ATE - - - - ATE Drunkards [ 74 ] - - - - - - - - - - - TUM-RGBD [ 87 ] ATE ATE ATE - - ATE ATE ATE ATE ATE ATE 7-Scenes [ 84 , 37 ] - - - - - - ATE - - - - KITTI [ 35 ] - t_{r} t_{r} - t_{r} - - ATE ATE - t_{r} EuRoC [ 11 ] ATE ATE ATE - ATE ATE - - ATE - ATE ScanNet [ 17 , 106 ] - - - - - - ATE - - - - ETH3D [ 81 , 82 ] - - ATE ATE - ATE - - ATE - - Rosario [ 69 ] - - ATE - - - - - - - - MADMAX [ 60 ] - - ATE - - - - - Qua. - - Lamar [ 77 ] - - - ATE - - - - - - - Minimal Texture [ 28 ] - - ATE - - - - - Qua. - - 4Seasons [ 100 , 101 ] - - RPE - - - - - - - RPE TUM Mono [ 25 ] - e_{a} e_{a} - - - - - Qua. - - HAMLYN [ 73 ] - - - - - - - - Qua. - - Table 1 : Benchmarking Metrics in SfM and VSLAM ‡ . Despite substantial efforts towards diversity, current benchmarks still rely heavily on small, curated datasets, limiting the adaptability of localization pipelines to real-world scenarios. Datasets are listed in descending order as synthetic, real with ground truth, and real with pseudo-ground truth. Metrics : Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) [ 87 ] , translational and rotational errors ( t_{r} ) [ 35 ] , alignment error ( e_{a} ) [ 25 ] , qualitative graph metrics (Qua.) [ 29 ] , reconstruction metrics (Map.) [ 88 ] , and photometric rendering metrics (Pho.) [ 99 , 109 , 59 ] . ‡ Due to the intrinsic complexity of benchmarking SfM/VSLAM [ 110 ] , this table provides only a high-level overview. Please, refer to the respective publications for the full details. Despite significant advances over the past decades, localization and 3D reconstruction from the images of a single moving camera still holds great potential for various downstream tasks, including, among others, view synthesis [ 92 ] and robotics [ 70 ] . A critical long-term goal in this domain is achieving data scalability, which would unlock new applications and significantly enhance the performance of current systems. However, while cameras are inexpensive and easy to deploy, and access to vast video data is increasingly feasible [ 22 , 2 , 83 , 38 ] , the field of visual localization—encompassing methods like Structure from Motion (SfM) and Visual SLAM (VSLAM)—has yet to achieve the same scalability and robustness breakthroughs seen in fields such as natural language processing [ 93 ] or generative AI [ 75 ] . A major obstacle in advancing localization pipelines is the complexity of benchmarking tasks, such as hyperparameter optimization during development or performance comparison against existing solutions [ 110 ] . Accurate benchmarking requires objective evaluation against ground truth data, which serves as a crucial reference for assessing system performance [ 3 , 5 ] . Moreover, real-world datasets are indispensable for meaningful benchmarking, as simulated data, while valuable for controlled experimentation, often falls short to capture the intricate complexities of real-world scenarios, including varying material properties, fine-grained structures, and dynamic reflections [ 86 , 97 , 74 ] . Unlike tasks such as object detection, tracking, or image segmentation—where ground truth is derived from large, human-annotated datasets [ 20 , 55 , 16 , 17 , 106 ] —localization pipelines require highly precise global positioning data. Outdoors, this typically involves sophisticated systems like RTK-GPS [ 35 , 100 , 1 , 101 ] , while urban and indoor settings demand expensive and complex measurement setups [ 87 , 11 , 82 , 108 , 41 ] . These challenges make the acquisition of ground truth data for localization resource-intensive and technically demanding [ 58 , 60 ] . In some specialized domains, the difficulty of obtaining reliable ground truth is even more pronounced. Fields such as medical robotics [ 50 , 73 , 62 ] , extra-planetary exploration [ 60 , 36 ] , and underwater robotics [ 44 , 78 ] operate in environments where

Read the complete text