Generating Synthetic Stereo Datasets using 3D Gaussian Splatting and Expert Knowledge Transfer

Filip Slezák, Magnus Kaufmann Gjerde, Joakim Bruslund Haurum, Ivan Nikolov, Morten S. Laursen, Thomas B. Moeslund · 2025

Read Generating Synthetic Stereo Datasets using 3D Gaussian Splatting and Expert Knowledge Transfer by Filip Slezák, Magnus Kaufmann Gjerde, Joakim Bruslund Haurum, Ivan Nikolov, Morten S. Laursen, Thomas B. Moeslund (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04908.txt.

Introduction

Filip Slezák and co-authors make fake twin-view image sets with a fast 3D scene method in a 2025 paper. Twin-view sets train depth tools, but real sets are dear to build and grade. Their pipe builds scenes from soft specks in 3D, a route that runs far faster than the slow glow-field route.

Depth can come from the built scene's own shape or from a strong depth model's calls. Raw scene shape proves rough and flawed, and those flaws pass to trained tools. Depth model calls are far cleaner, and tools trained on them score well on fresh tests.

Catalog record

CREATORFilip Slezák, Magnus Kaufmann Gjerde, Joakim Bruslund Haurum, Ivan Nikolov, Morten S. Laursen, Thomas B. Moeslund
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04908

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generating Synthetic Stereo Datasets using 3D Gaussian Splatting and Expert Knowledge Transfer Filip Slezák, Magnus Kaufmann Gjerde, Joakim Bruslund Haurum, Ivan Nikolov, Morten S. Laursen, Thomas B. Moeslund Abstract In this paper, we introduce a 3D Gaussian Splatting (3DGS)-based pipeline for stereo dataset generation, offering an efficient alternative to Neural Radiance Fields (NeRF)-based methods. To obtain useful geometry estimates, we explore utilizing the reconstructed geometry from the explicit 3D representations as well as depth estimates from the FoundationStereo model in an expert knowledge transfer setup. We find that when fine-tuning stereo models on 3DGS-generated datasets, we demonstrate competitive performance in zero-shot generalization benchmarks. When using the reconstructed geometry directly, we observe that it is often noisy and contains artifacts, which propagate noise to the trained model. In contrast, we find that the disparity estimates from FoundationStereo are cleaner and consequently result in a better performance on the zero-shot generalization benchmarks. Our method highlights the potential for low-cost, high-fidelity dataset creation and fast fine-tuning for deep stereo models. Moreover, we also reveal that while the latest Gaussian Splatting based methods have achieved superior performance on established benchmarks, their robustness falls short in challenging in-the-wild settings warranting further exploration. 1 Introduction Recovering the 3D structure of a scene captured from images is a widely researched problem that has exploded in popularity through the recent advances in monocular depth estimation as well as novel view synthesis approaches such as Neural Radiance Fields (NeRFs) Mildenhall et al. 2020 and Gaussian Splatting Kerbl et al. 2023 . Initial work in the field required the use of stereo image pairs and, through traditional algorithms like semi-global matching (SGM) Hirschmuller 2005 , achieved remarkable performance using well-designed heuristics. This resulted in accurate disparity maps but still had several shortcomings due to the common challenges of stereo matching. These are, namely, textureless regions and occlusions - both of which make it very challenging to find reliable correspondences, leading to errors or a lack of prediction. With the introduction of large-scale datasets with accurate disparity labels, such as Mayer et al. 2015 , deep learning-based disparity estimation methods were made possible. However, obtaining per-pixel labels is very challenging, typically addressed by using a simulated environment where accurate ground truth geometry can be directly extracted from the explicit scene representation. While the geometry labels are highly accurate, the realism gap between rendered and real-world images makes these methods perform suboptimal when applied on real-world samples. Recent advancements include the addition of vision foundation models pre-trained for monocular depth estimation Hu et al. 2024 ; Yang et al. 2024 and powerful context networks, such as Dinov2 Oquab et al. 2023 , which have greatly improved the networks’ reasoning ability in the ill-posed regions. This motivated the use of monocular depth estimators such as DepthAnythingV2 Yang et al. 2024 and Metric3Dv2 Hu et al. 2024 to produce high-quality depth estimates of real images. Orthogonally, Tosi et al . Tosi et al. 2023 instead simply collect data via mobile phone recordings and apply the recent advances in the field of Novel View Synthesis in order to obtain realistic images and cheap geometry labels. However, NeRFs, as used by e.g . Tosi et al . , have a considerable shortcoming. The implicit representation of the scene fails to produce accurate, fully dense geometry, requiring a lot of filtering, which makes the disparity maps very sparse. As a consequence, a complicated training procedure requires photometric losses apart from the common L1 disparity loss to compensate for the lack of density. Such a training setup suffers from training instability, and the results are not replicable as outlined in greater detail in our experimentation section and observing the replication efforts of the academic community. Figure 1 : Expert Knowledge Transfer using FoundationStereo. Overview of the proposed expert knowledge transfer setup where stereo pairs rendered with Gaussian Splatting are supplied to FoundationStereo Wen et al. 2025 , which produces high-quality pseudo depth estimates. The process starts with the 3D Gaussian Splatting (3DGS) pipeline Kerbl et al. 2023 , where COLMAP Schönberger et al. 2016 Schönberger and Frahm 2016 is used to initialize the optimization of 3DGS. Stereo image pairs are rendered after the scene has been fitted with 3DGS. As the underlying geometry of 3DGS is poorly reconstructed, the stereo pairs are instead sent to FoundationStereo, producing ground-truth disparity images. The generated synthetic stereo dataset is then used to train a lightweight stereo network, RAFT-Stereo Lipson et al. 2021 (not depicted in this figure). Instead, we investigate the newly introduced 3D Gaussian Splatting Kerbl et al. 2023 based methods, which explicitly model the scene using Gaussians that can be converted to meshes Huang et al. 2024 ; Chen et al. 2024 ; Yu et al. 2024 . As these methods offer highly realistic images and seemingly accurate surface reconstruction, we hypothesize that they might be the missing link to the easy acquisition of stereo-matching datasets. Furthermore, our choice of 3DGS based methods is further motivated by their excellent rendering speed compared to NeRF based methods of comparable reconstruction quality. Concurrently, we are inspired by the recent development within monocular and stereo vision foundation models, and investigate whether using depth estimates from a large stereo vision foundation model Wen et al. 2025 outperforms the performance obtained with the reconstructed geometry, as shown in Figure 1 . Using a fine-tuning training setup, we evaluate the effect of the different 3D Gaussian Splatting-based synthetic datasets, and compare directly with NeRF-Stereo. We do so by performing zero-shot evaluation on a set of previously unseen datasets commonly used within the stereo depth estimation field. Our contributions are as follows: • We qualitatively and empirically find that the rendered disparity from 3DGS and the reconstructed mesh of PGSR are insufficient for fine-tuning a RAFT-Stereo network. • We propose a knowledge transfer setup where predicted depth estimates from FoundationStereo are used as ground truth for 3DGS stereo renderings. • We find that fine-tuning a RAFT-Stereo network using our proposed expert knowledge transfer setup consistently outperforms other methods, including the state-of-the-art NeRF-Stereo method. 2 Related Works 2.1 Disparity Estimation Algorithms Large-scale disparity map estimation became possible with the development of SGM algorithms Hirschmuller 2005 , which optimized the calculated disparity along preset directional paths, instead of purely global optimization, which can be costly, or local methods that tended to leave artifacts around edges of objects. With the rise of deep learning models, architectures like PSM-Net Chang and Chen 2018 and models built on top of it Huang et al. 2021 , using features extracted by ResNet and a pyramidal structure which utilizes features from different scales. Later on, the work presented in RAFT-Stereo Teed and Deng 2021 and later on in RAFT-3D Lipson et al. 2021 models the disparity and full 3D scene flow between consecutive frames and iteratively optimizes them using GRU layers. Taking the iterative optimization idea and expanding on it through the use of Geometry Encoding Volume is the IGEV-Stereo Xu et al. 2023 . It incorporates more local details through the iterative optimization of the disparity maps to capture smaller 3D surface features. One problem that persists in both RAFT and IGEV models is the ambiguity in smooth and featureless regions, which

Read the complete text