FoundationStereo: Zero-Shot Stereo Matching

Bowen Wen Matthew Trepte Joseph Aribido Jan Kautz · 2025

Read FoundationStereo: Zero-Shot Stereo Matching by Bowen Wen Matthew Trepte Joseph Aribido Jan Kautz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09898.txt.

Introduction

Bowen Wen and co-authors build a base model in a 2025 paper for depth from two eyes. Twin-view depth tools ace their home tests after fine tunes, yet fail cold on new ground. Base models in kin fields pass cold tests with ease, which set the bar.

Their build trains on a vast fake set of twin photos, rich and true to light, with bad pairs culled by a self-check. A side tune borrows the rich eye of one-view models for the twin task. The end tool calls depth on new scenes with no tune. Robot and car teams gain a depth eye that travels.

Catalog record

CREATORBowen Wen Matthew Trepte Joseph Aribido Jan Kautz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FoundationStereo: Zero-Shot Stereo Matching Bowen Wen Matthew Trepte Joseph Aribido Jan Kautz Abstract Tremendous progress has been made in deep stereo matching to excel on benchmark datasets through per-domain fine-tuning. However, achieving strong zero-shot generalization — a hallmark of foundation models in other computer vision tasks — remains challenging for stereo matching. We introduce FoundationStereo, a foundation model for stereo depth estimation designed to achieve strong zero-shot generalization. To this end, we first construct a large-scale (1M stereo pairs) synthetic training dataset featuring large diversity and high photorealism, followed by an automatic self-curation pipeline to remove ambiguous samples. We then design a number of network architecture components to enhance scalability, including a side-tuning feature backbone that adapts rich monocular priors from vision foundation models to mitigate the sim-to-real gap, and long-range context reasoning for effective cost volume filtering. Together, these components lead to strong robustness and accuracy across domains, establishing a new standard in zero-shot stereo depth estimation. Project page: https://nvlabs.github.io/FoundationStereo/ Figure 1 : Zero-shot prediction on in-the-wild images. Our method generalizes to diverse scenarios (indoor / outdoor), objects of challenging properties (textureless / reflective / translucent / thin-structured), complex illuminations (shadow / strong exposure), various viewing perspectives and sensing ranges. 1 Introduction Since the advent of the first stereo matching algorithm nearly half a century ago [ 42 ] , we have come a long way. Recent stereo algorithms can achieve amazing results, almost saturating the most challenging benchmarks—thanks to the proliferation of training datasets and advances in deep neural network architectures. Yet, fine-tuning on the dataset of the target domain is still the method of choice to get competitive results. Given the zero-shot generalization ability shown on other problems within computer vision via the scaling law [ 32 , 46 , 78 , 79 ] , what prevents stereo matching algorithms from achieving a similar level of generalization? Leading stereo networks [ 73 , 80 , 53 , 41 , 54 , 11 ] construct cost volumes from the unary features and leverage 3D CNNs for cost filtering. Refinement-based methods [ 36 , 86 , 60 , 21 , 34 , 27 , 67 , 14 ] iteratively refine the disparity map based on recurrent modules such as Gated Recurrent Units (GRU). Despite their success on public benchmarks under per-domain fine-tuning setup, however, they struggle to gather non-local information to effectively scale to larger datasets. Other methods [ 35 , 68 ] explore transformer architectures for unary feature extraction, while lacking the specialized structure afforded by cost volumes and iterative refinement to achieve high accuracy. Such limitations have, to date, hindered the development of a stereo network that generalizes well to other domains. While it is true that cross-domain generalization has been explored by some prior works [ 82 , 84 , 17 , 10 , 49 , 37 ] , such approaches have not achieved results that are competitive with those obtained by fine-tuning on the target domain, either due to insufficient structure in the network architecture, impoverished training data, or both. These networks are generally experimented on Scene Flow [ 43 ] , a rather small dataset with only 40K annotated training image pairs. As a result, none of these methods can be used as an off-the-shelf solution, as opposed to the strong generalizability of vision foundation models that have emerged in other tasks. To address these limitations, we propose FoundationStereo, a large foundation model for stereo depth estimation that achieves strong zero-shot generalization without per-domain fine-tuning. We train the network on a large-scale (1M image pairs) high-fidelity synthetic training dataset with high diversity and photorealism. An automatic self-curation pipeline is developed to eliminate the ambiguous samples that are inevitably introduced during the domain randomized data generation process, improving both the dataset quality and model robustness over iterate updates. To mitigate the sim-to-real gap, we propose a side-tuning feature backbone that adapts internet-scale rich priors from DepthAnythingV2 [ 79 ] that is trained on real monocular images to the stereo setup. To effectively leverage these rich monocular priors embedded into the 4D cost volume, we then propose an Attentive Hybrid Cost Volume (AHCF) module, consisting of 3D Axial-Planar Convolution (APC) filtering that decouples standard 3D convolution into two separate spatial- and disparity-oriented 3D convolutions, enhancing the receptive fields for volume feature aggregation; and a Disparity Transformer (DT) that performs self-attention over the entire disparity space within the cost volume, providing long range context for global reasoning. Together, these innovations significantly enhance the representation, leading to better disparity initialization, as well as more powerful features for the subsequent iterative refinement process. Our contributions can be summarized as follows: \bullet We present FoundationStereo, a zero-shot generalizable stereo matching model that achieves comparable or even more favorable results to prior works fine-tuned on a target domain; it also significantly outperforms existing methods when applied to in-the-wild data. \bullet We create a large-scale (1M) high-fidelity synthetic dataset for stereo learning with high diversity and photorealism; and a self-curation pipeline to ensure that bad samples are pruned. \bullet To harness internet-scale knowledge containing rich semantic and geometric priors, we propose a Side-Tuning Adapter (STA) that adapts the ViT-based monocular depth estimation model [ 79 ] to the stereo setup. \bullet We develop Attentive Hybrid Cost Filtering (AHCF), which includes an hourglass module with 3D Axial-Planar Convolution (APC), and a Disparity Transformer (DT) module that performs full self-attention over the disparity dimension. 2 Related Work Deep Stereo Matching. Recent advances in stereo matching have been driven by deep learning, significantly enhancing accuracy and generalization. Cost volume aggregation methods construct cost volumes from unary features and perform 3D CNN for volume filtering [ 73 , 80 , 53 , 41 , 54 , 11 ] , though the high memory consumption prevents direct application to high resolution images. Iterative refinement methods, inspired by RAFT [ 57 ] , bypasses the costly 4D volume construction and filtering by recurrently refining the disparity [ 36 , 86 , 60 , 21 , 34 , 27 , 67 , 14 ] . While they generalize well to various disparity range, the recurrent updates are often time-consuming, and lack long-range context reasoning. Recent works [ 71 , 72 ] thus combine the strengths of cost filtering and iterative refinement. With the tremendous progress made by vision transformers, another line of research [ 35 , 23 , 68 ] introduces transformer architecture to stereo matching, particularly in the unary feature extraction stage. Despite their success on per-domain fine-tuning setup, zero-shot generalization still remains challenging. To tackle this problem, [ 82 , 84 , 17 , 10 , 49 , 37 ] explore learning domain-invariant features for cross-domain generalization, with a focus on training on Scene Flow [ 43 ] dataset. Concurrent work [ 3 ] achieves remarkable zero-shot generalization with monocular prior enhanced correlation volumes. However, the strong generalizability of vision foundation models emerged in other tasks that is supported by scaling law has yet to be fully realized in stereo matching for practical applications. Properties Sintel [ 6 ] Sceneflow [ 43 ] CREStereo [ 34 ] IRS [ 64 ] TartanAir [ 66 ] FallingThings [ 61 ] UnrealStereo4K [ 59 ] Spring [ 44 ]

Read the complete text