MLG-Stereo: ViT Based Stereo Matching with Multi-Stage Local-Global Enhancement

Haoyu Zhang, Jingyi Zhou, Peng Ye, Jiakang Yuan, Lin Zhang · 2026

Read MLG-Stereo: ViT Based Stereo Matching with Multi-Stage Local-Global Enhancement by Haoyu Zhang, Jingyi Zhou, Peng Ye, Jiakang Yuan, Lin Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20393.txt.

Introduction

Zhang and six colleagues present MLG-Stereo, a vision Transformer pipeline for stereo matching that pushes global modeling past the encoder stage. A multi-granularity feature network balances global context with local geometric detail, handling images of any resolution and bridging train-inference scale gaps. A local-global cost volume captures locally correlated and globally aware matching clues. A local-global guided recurrent unit then refines disparity step by step under global guidance, with top results on Middlebury, KITTI-2015, and KITTI-2012.

Catalog record

CREATORHaoyu Zhang, Jingyi Zhou, Peng Ye, Jiakang Yuan, Lin Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57722
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20393

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MLG-Stereo: ViT Based Stereo Matching with Multi-Stage Local-Global Enhancement Thanks: Resubmitted on March 26, 2026. Revised on February 15, 2026. Initially submitted on April 13, 2025. This work is supported by National Key Research and Development Program of China (No. 2022ZD0160101), Shanghai Natural Science Foundation (No. 23ZR1402900), Shanghai Science and Technology Commission Explorer Program Project (24TS1401300), Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103). The computations in this research were performed using the CFFF platform of Fudan University. (Haoyu Zhang and Jingyi Zhou contributed equally to this work) (Corresponding author: Peng Ye and Tao Chen.) Thanks: Haoyu Zhang, Jingyi Zhou, Jiakang Yuan, Lin Zhang and Tao Chen are with the Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]) Thanks: Peng Ye is with the Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China, and also with Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China, and also with the Chinese University of Hong Kong, Hong Kong (e-mail: [email protected]) Thanks: Feng Xu is with the Key Laboratory for Information Science of Electromagnetic Waves, Ministry of Education, Fudan University, Shanghai 200433, China, and also with the College of Future Information Technology, Fudan University, Shanghai 200433, China (e-mail: [email protected]) PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE Haoyu Zhang, Jingyi Zhou, Peng Ye, Jiakang Yuan, Lin Zhang Abstract With the development of deep learning, ViT-based stereo matching methods have made significant progress due to their remarkable robustness and zero-shot ability. However, due to the limitations of ViTs in handling resolution sensitivity and their relative neglect of local information, the ability of ViT-based methods to predict details and handle arbitrary-resolution images is still weaker than that of CNN-based methods. To address these shortcomings, we propose MLG-Stereo, a systematic pipeline-level design that extends global modeling beyond the encoder stage. First, we propose a Multi-Granularity Feature Network to effectively balance global context and local geometric information, enabling comprehensive feature extraction from images of arbitrary resolution and bridging the gap between training and inference scales. Then, a Local-Global Cost Volume is constructed to capture both locally-correlated and global-aware matching information. Finally, a Local-Global Guided Recurrent Unit is introduced to iteratively optimize the disparity locally under the guidance of global information. Extensive experiments are conducted on multiple benchmark datasets, demonstrating that our MLG-Stereo exhibits highly competitive performance on the Middlebury and KITTI-2015 benchmarks compared to contemporaneous leading methods, and achieves outstanding results in the KITTI-2012 dataset. I Introduction Stereo matching seeks to estimate dense pixel-wise correspondences—specifically, disparity—between rectified image pairs captured by horizontally displaced cameras. Due to its high accuracy in depth estimation enabled by camera calibration, stereo matching has established itself as a cornerstone task in computer vision and 3D reconstruction. Its practical relevance spans diverse real-world domains such as robotics, autonomous driving, and augmented reality [ 15 , 57 , 2 , 37 , 35 , 13 ] , where accurate scene depth perception is indispensable for environment modeling, intelligent decision-making, and immersive human-computer interactions. With the development of deep learning, Convolutional Neural Networks (CNNs) have been applied to stereo matching, significantly improving accuracy. Early works mainly consist of aggregation-based methods [ 24 , 9 , 19 ] , as shown in the upper part of Fig. 1 (a), where CNNs acted as encoders to extract features and 3D convolutions were used to construct a 3D cost volume, followed by regression to predict disparity. Since RAFT-Stereo [ 29 ] introduced the idea of iterative optimization into the stereo matching task, many researchers have devised various methods [ 60 , 61 , 48 , 45 , 63 ] , as shown in the lower part of Fig. 1 (a). These methods typically use a multi-scale CNN encoder that includes a feature network and a context network, followed by initializing disparity using regression or zero disparity initialization, and decoding residual disparity using Gated Recurrent Unit (GRU) or Long Short-Term Memory (LSTM) based blocks. In general, these CNN-based methods not only support arbitrary resolution input but also bring stronger local detail prediction ability through local perception. However, these methods have poor zero-shot capabilities due to the limited general feature extraction ability of CNN networks, and their prediction errors are prone to occur in some difficult areas (such as repeated textures and weak textures) due to the lack of a global receptive field. Fig. 1: Overview of existing typical stereo matching frameworks. (a) CNN-based frameworks. (b) ViT-based frameworks. (c) Our MLG-Stereo framework which builds on the ViT-based method and improves it with multi-stage local-global enhancement. This design allows our method to have the advantages of both CNN-based and ViT-based methods. Inspired by the powerful global context extraction capability of Vision Transformers (ViTs), ViT-based methods have been explored in recent years. Some researchers adopt a hybrid approach by combining CNNs and ViTs in a sequential manner, where CNNs serve as feature encoders and ViTs are used for matching, such as STTR [ 27 ] and CSTR [ 18 ] , as shown in the upper part of Fig. 1 (b). Some researchers, on the other hand, utilize the powerful feature extraction capability of ViT to directly extract image features or enhance the features extracted by CNNs, as shown in the lower part of Fig. 1 (b) [ 51 , 30 ] . These ViT-based methods rely on the attention mechanism to make more robust global receptive field image predictions and can leverage Vision Foundation Models (VFMs) as the backbone to achieve stronger zero-shot capabilities. However, ViTs are highly sensitive to resolution, resulting in degraded performance on ultra-high-resolution images. Moreover, their limited ability to capture fine local details often results in detail loss, as observed in the results of VITAStereo [ 30 ] and our ablation experiments. In addition to the encoders, current decoder architectures also face challenges in fully leveraging the global matching information extracted by the encoder. Due to the computational burden of the cost volume, existing methods typically feed only local window cost volume information into the iterative optimization process. The underutilization of global matching information in existing methods makes the optimization process prone to local optima and often requires more iterations to achieve satisfactory stereo matching results. To address these challenges, we propose MLG-Stereo, a unified pipeline for ViT-based stereo matching. We observe that while CNN-based stereo matching inherently excels at preserving local details, its global modeling capacity is constrained by local perception fields. Conversely, ViT-based methods demonstrate robust global context understanding but face bottlenecks in arbitrary-resolution adaptation and fine detail recovery. Consequently, the core design philosophy of our work is to amplify the potential of ViT-based stereo matching by overcoming their inherent limitations, which systematically couples global geometric guidance with local iterative refinement. Specifically, we first propose a Multi-Granularity Feature Network (MGFN) that employs a shared-weight hierarchical architecure to overcome single-scale constraints, bridging pretrained global semantics with local fine-grained geometric details. Building on this, we introduce the Local-Global Cost Volume (LGCV), leveraging latent token compression to achieve bidirectional global cost aggregation within hardware-affordable limits. Finally, we design a Local-Global Guided Recurrent Unit (LGRU) that genuinely injects global cost priors into iterative updates, rectifying the optimization direction in ill-posed

Read the complete text