CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding

Lihuang Fang, Xiao Hu, Yuchen Zou, Hong Zhang, Life Fellow, IEEE · 2025

Read CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding by Lihuang Fang, Xiao Hu, Yuchen Zou, Hong Zhang, Life Fellow, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22119.txt.

Introduction

Lihuang Fang, Xiao Hu, Yuchen Zou, and Hong Zhang improve stereo depth estimation, the task of turning two camera views into a distance map. Learned stereo systems shine on the datasets they were tuned for but lack the easy generalization that foundation models show elsewhere. They fail predictably where matching is ambiguous: hidden surfaces and textureless walls.

Their framework, CogStereo, lends the matcher a sense of the whole scene. Depth features from a single-image model, which carry holistic layout understanding, are embedded into the refinement stage as a spatial cognition prior. A dual-conditional corrector weighs pixel-level uncertainty together with those cognition features to fix mismatches coherently. Across six benchmarks and real-world footage, the system reached leading accuracy and, notably, kept its quality when carried to scenes unlike its training data.

Catalog record

CREATORLihuang Fang, Xiao Hu, Yuchen Zou, Hong Zhang, Life Fellow, IEEE
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22119

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding Lihuang Fang, Xiao Hu, Yuchen Zou, Hong Zhang, Life Fellow, IEEE Abstract Deep stereo matching has advanced significantly on benchmark datasets through fine-tuning but falls short of the zero-shot generalization seen in foundation models in other vision tasks. We introduce CogStereo, a novel framework that addresses challenging regions, such as occlusions or weak textures, without relying on dataset-specific priors. CogStereo embeds implicit spatial cognition into the refinement process by using monocular depth features as priors, capturing holistic scene understanding beyond local correspondences. This approach ensures structurally coherent disparity estimation, even in areas where geometry alone is inadequate. CogStereo employs a dual-conditional refinement mechanism that combines pixel-wise uncertainty with cognition-guided features for consistent global correction of mismatches. Extensive experiments on Scene Flow, KITTI, Middlebury, ETH3D, EuRoc, and real-world demonstrate that CogStereo not only achieves state-of-the-art results but also excels in cross-domain generalization, shifting stereo vision towards a cognition-driven approach. I Introduction Stereo matching, essential for estimating depth from binocular images, remains a core challenge in robotics and computer vision [ 1 , 2 , 3 , 4 , 5 , 6 ] . Despite advancements with synthetic datasets and powerful neural architectures, robust performance in varied real-world environments is elusive. Challenges arise in ill-posed regions like occlusions and weak textures, where pixel correspondence is unreliable. Current state-of-the-art (SOTA) methods often rely on domain-specific tuning with techniques such as cost volumes [ 7 ] , recurrent refinements [ 8 , 9 ] , and transformer-based reasoning [ 10 ] , limiting their generalization to ill-posed regions. In contrast, foundation models for other vision tasks have shown strong zero-shot generalization [ 11 ] . Models pretrained for classification [ 12 , 13 ] , segmentation [ 14 ] , and monocular depth estimation [ 15 ] have demonstrated robust performance on diverse data without domain adaptation. This raises the question: Can stereo matching adopt a foundation-model approach for improved generalization to ill-posed regions? This is crucial for applications like autonomous driving [ 16 ] and robotics [ 1 ] , where consistent performance over diverse regions is vital. Fig. 1: Illustration of the motivation behind CogStereo. While modern stereo networks achieve highly accurate disparity estimation in most regions, they remain vulnerable to mismatches in textureless, reflective, or occluded areas. CogStereo addresses these challenges by embedding implicit SC, enabling globally consistent and robust disparity estimation. A key shortcoming in stereo systems is their reliance on local geometric correspondence, often failing in difficult regions. In contrast, the depth foundation model captures object-level geometry and global scene understanding [ 15 ] –a capability we term spatial cognition (SC) , similar to human perception (as shown in Figure 1 ). This understanding encompasses depth ordering, shape consistency, and semantic priors (e.g., flat roads, upright humans), allowing monocular models to maintain coherence. Previous attempts to merge monocular cues with stereo [ 17 ] have explicit depth maps , which are limited by local errors and lack global consistency. To address this, we propose CogStereo , a framework transcending geometric correspondences by embedding implicit SC. Instead of explicit depth predictions, CogStereo utilizes feature representations from depth foundation model as SC priors. To refine disparities, it introduces a Dual-Condition Refinement mechanism, which adjusts disparity based on (i) pixel-wise uncertainty identifiers for unreliable regions and (ii) spatial cognition features for semantic and geometric consistency. This ensures globally coherent disparity maps, even in challenging conditions, as Figure 1 shows how SC aids in achieving robust disparities. Overall, our contributions are summarized as follows: • We propose CogStereo , embedding implicit spatial cognition into stereo matching, leveraging monocular depth features as priors for improved understanding and accuracy. • We introduce a Dual-Condition Refinement mechanism to integrate uncertainty priors with implicit SC features, enhancing disparity correction in ambiguous regions and preventing metric drift caused by implicit optimization. • Extensive benchmarks and real-world experiments demonstrate CogStereo’s SOTA results and robust zero-shot generalization, marking a shift toward cognitively informed stereo matching. II Related Work II-A Learning based Stereo Matching Early deep stereo methods [ 18 , 19 , 20 ] built 3D cost volumes from binocular features and used 3D convolutions for disparity regression. Later works introduced recurrent refinement [ 21 , 18 ] and transformer-based methods [ 22 ] for long-range pixel correspondences, while NMRF-Stereo [ 23 ] used neural MRFs for improved accuracy. Despite these advances, all rely on geometric matching and still exhibit poor generalization in occlusions, weak textures, or repetitive patterns. Recursive stereo methods like RAFT-Stereo [ 18 ] and its variants improve structure and information integration [ 20 , 7 , 24 ] , but still struggle with weak texture or reflective regions. Even large-scale pre-training (FoundationStereo [ 25 ] ) shows high zero-shot errors in complex scenes( Figure 2 ), revealing a cognitive bottleneck in pure stereo paradigms. In contrast, CogStereo incorporates spatial cognition from depth foundation model, directly addressing the matching challenge. II-B Monocular Depth Estimation Monocular depth estimation has advanced rapidly with deep learning [ 26 , 27 , 28 , 29 ] . Early CNN-based regression methods suffered from scale ambiguity and domain gaps, while MiDaS [ 30 ] reframed the task as relative scene understanding, enabling zero-shot inference. Depth Anything (DA) [ 31 ] and DAv2 [ 15 ] further improved robustness through semantic alignment and pseudo-label distillation, and diffusion-based approaches [ 32 , 33 ] leverage generative priors to enforce structural consistency. With large-scale pretraining and foundation models such as DINOv2 [ 34 ] , modern monocular depth models can capture relative depth ordering, object geometry, and global layout. Crucially, they preserve geometric and semantic consistency even in textureless or occluded regions, a property we call SC , reflecting holistic scene understanding and strong zero-shot generalization. This emerging property provides the key inspiration [ 35 ] for our proposed CogStereo, which aims to embed SC into stereo matching for robust and globally consistent disparity estimation. III CogStereo III-A Preliminaries Given a rectified image pair I_{l} and I_{r} , stereo matching estimates a dense horizontal displacement field f_{h} , mapping each pixel (u,v) in I_{l} to (u+f_{h},v) in I_{r} . A method like RAFT-Stereo [ 18 ] constructs a cost volume by correlating features from the left and right images: \displaystyle C(i,j,k)=\frac{F(I_{l})\cdot F(I_{r})}{\sqrt{D}}, (1) where F(\cdot) denotes feature maps, D is the feature dimension, and C(i,j,k) measures the similarity between pixel (i,j) in I_{l} and displacement k in I_{r} . Disparity is then obtained by iterative refinement over the cost volume. While effective in textured regions, these matching and refinement methods often fail in areas such as textureless surfaces, reflections, repetitive patterns, and occlusions where correspondences are ambiguous. The underlying limitation is the absence of SC: reasoning is restricted to local feature similarity without object- or scene-level awareness. Fig. 2: Zero-shot prediction on EuRoC, demonstrating CogStereo’s generalization to practical scenarios with challenging properties like weak texture, appearance ambiguities, reflectance, translucency and occlusion. Fig. 3: Overview of Our CogStereo Framework. The CogStereo framework enhances stereo matching accuracy through two stages. Stage 1 pretrains the network to generate an uncertainty map from the cost volume. Stage 2 uses the UG-SCA module to refine disparities, leveraging SC to correct errors

Read the complete text