Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties

Jisoo Jeong Hong Cai Jamie Menjay Lin Fatih Porikli, Qualcomm AI Research {jisojeon, hongcai, jmlin, fporikli}@qti.qualcomm.com · 2025

Read Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties by Jisoo Jeong Hong Cai Jamie Menjay Lin Fatih Porikli, Qualcomm AI Research {jisojeon, hongcai, jmlin, fporikli}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00324.txt.

Introduction

Cameras and AI systems judge depth and motion by matching pixels between two views, but some spots are far harder to match than others. Jisoo Jeong and co-authors improve that training in a 2025 paper from Qualcomm AI Research. Standard training treats every pixel the same, which wastes effort on easy spots and starves the hard ones.

The team maps how unsure the model is at each pixel and uses that map in two new training losses. One loss pushes the model to spend more effort on hard pixels. The other steers it away from spots hidden in one view, where no true match exists. Tests on both motion and depth tasks showed clear gains from the paired losses.

Catalog record

CREATORJisoo Jeong Hong Cai Jamie Menjay Lin Fatih Porikli, Qualcomm AI Research {jisojeon, hongcai, jmlin, fporikli}@qti.qualcomm.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00324

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties Jisoo Jeong Hong Cai Jamie Menjay Lin Fatih Porikli, Qualcomm AI Research {jisojeon, hongcai, jmlin, fporikli}@qti.qualcomm.com Abstract Conventional training for optical flow and stereo depth models typically employs a uniform loss function across all pixels. However, this one-size-fits-all approach often overlooks the significant variations in learning difficulty among individual pixels and contextual regions. This paper investigates the uncertainty-based confidence maps which capture these spatially varying learning difficulties and introduces tailored solutions to address them. We first present the Difficulty Balancing (DB) loss, which utilizes an error-based confidence measure to encourage the network to focus more on challenging pixels and regions. Moreover, we identify that some difficult pixels and regions are affected by occlusions, resulting from the inherently ill-posed matching problem in the absence of real correspondences. To address this, we propose the Occlusion Avoiding (OA) loss, designed to guide the network into cycle consistency-based confident regions, where feature matching is more reliable. By combining the DB and OA losses, we effectively manage various types of challenging pixels and regions during training. Experiments on both optical flow and stereo depth tasks consistently demonstrate significant performance improvements when applying our proposed combination of the DB and OA losses. 1 Introduction † † footnotetext: \dagger Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Feature matching serves as a core technique in the realm of computer vision, supporting various tasks and applications. Optical flow, which captures 2D pixel-wise displacements via feature matching, enables applications ranging from object tracking Kale et al. 2015 , action recognition Lee et al. 2018 ; Cai et al. 2019 , video compression Wu et al. 2018 ; Lu et al. 2019 , and video frame interpolation Kong et al. 2022 ; Li et al. 2023 ; Jeong et al. 2024 . Similarly, rectified stereo depth estimation, discerning disparities between stereo images through feature matching, supports extensive applications including autonomous driving, extended reality, and mixed reality. When training a network to predict optical flow Teed and Deng 2020 ; Jiang et al. 2021 ; Huang et al. 2022 or stereo depth Lipson et al. 2021 ; Li et al. 2022 ; Karaev et al. 2023 , it has been a standard practice in various models to use the same loss function across all valid pixels (Fig 1 top). In our study, however, the difficulty for the model to learn fine-grained correspondence could vary across pixels due to various factors including contexts, non-rigid motions, and lighting conditions. Moreover, learning for occlusion regions can involve additional challenges Zhao et al. 2020 . Figure 1 : (a) Most existing methods (e.g., Teed and Deng 2020 ; Lipson et al. 2021 ) predominately treat training loss on each pixel equally for optical flow and Stereo depth. (b) SCIFlow Lin et al. 2024 utilizes a Regression Focal Loss, which focuses more on low-confident samples, for training optical flow models. (c) Our proposed approach more comprehensively considers two sources of learning difficulties in training, i.e., model confidence and occlusion. In this paper, we propose a novel, effective training approach for optical flow and stereo depth models, leveraging uncertainty understanding to infer the learning difficulties for the model due to both pixel-wise contents and occlusions. In particular, we enable the model to learn by its confidence and to leverage contextual insight to mitigate the challenge of non-matching pixels. More specifically, we first utilize a Difficulty Balancing (DB) loss, which imposes larger weights on pixels of lower prediction confidence. Our DB proposal is an extension from the Regression Focal Loss (RFL) originally introduced in Lin et al. 2024 for optical flow estimation. In this paper, our DB loss further improves over the RFL with optimal hyperparameters and we further extend the application of DB to stereo depth (Fig 1 bottom). Moreover, we introduce the Occlusion Avoiding (OA) loss, which mitigates the loss in regions where pixel-wise feature matching may not be feasible. In contrast to some prior works Jonschkowski et al. 2020 ; Stone et al. 2021 that completely discard non-matching areas during training, we continue to compute a minimum loss for such occluded areas, as it is still necessary to predict the motion for occlusion regions (from all object) in the dense prediction. In this paper, we utilize forward and backward consistency to derive occlusion information and infer matchable areas, and make the network concentrate its learning in the matchable regions during training. Finally, we combine the DB and OA losses to address pixel-wise learning difficulties in training. Our proposal is model agnostic in nature. We apply our methods to several leading networks in optical flow and stereo depth, RAFT Teed and Deng 2020 , FlowFormer Huang et al. 2022 , and RAFT-Stereo Lipson et al. 2021 . We empirically validate the effects of the DB and OA losses, and demonstrate how their proper combination yields further benefits in model accuracy. In summary, our main contributions are as follows: • We observe pixel-wise variations in learning difficulties and hypothesize the sources for such behavior in optical flow and stereo depth estimation. • First, we introduce the Difficulty Balancing (DB) loss to incorporate model confidence, which is inspired by and improves upon SCIFlow Lin et al. 2024 . We find the optimal hyperparameters for confidence map calibration, as well as weighting for optical flow and stereo estimation tasks. • We further propose the Occlusion Avoiding (OA) loss, which infers the matching reliability from the cycle (forward-backward) consistency and mitigates the weights of the regions that are less likely to be matchable accordingly. • Finally, we demonstrate options to use both DB and OA losses simultaneously with an optimal combination, which provides uncertainty awareness in training and leads to improved model accuracy for optical flow and stereo depth. 2 Related Work In optical flow estimation, consider two consecutive video frames, I_{0} and I_{1} . We denote the optical flow from I_{0} to I_{1} as f_{0\rightarrow 1} . In (rectified) stereo depth estimation, we consider two stereo images, I_{L} and I_{R} . We denote the stereo depth estimation output as d_{L\rightarrow R} 2.0.1 Optical Flow The architecture of RAFT Teed and Deng 2020 showed remarkable performance, and many subsequent studies Huang et al. 2022 ; Zhang et al. 2021 ; Jiang et al. 2021 followed this baseline RAFT architecture. They extract features from two images and build a 4D correlation volume. And then, they iteratively regress to predict the optical flow output using ConvGRU blocks with the correlation volume. Such iteration-based prediction methods compute the loss for each optical flow prediction. l^{i}=||(f_{gt}-f_{1\rightarrow 2}^{i})||_{1} (1) where f_{gt} and f_{1\rightarrow 2}^{i} are the optical flow ground truth and prediction for the i -th iteration, respectively. The loss is accumulated over iterations as follows. L_{total}=\sum_{i=1}^{N}\gamma^{N-i}\cdot l^{i} (2) Stereo Depth Estimation: RAFT-stereo Lipson et al. 2021 follows the RAFT architecture and achievs competitive quality in stereo depth estimation. It extracts features for the stereo images, builds 3D correlation volume, and iteratively updates the disparity (namely, the scaled inverse depth). The model is trained similarly to Eq. 1 and 2 with the disparity predictions over iterations against the ground truth. Figure 2 : Overview of our method. Optical Flows ( f_{1\rightarrow

Read the complete text