Gaussian Belief Propagation Network for Depth Completion

Jie Tang, Pingping Xie & Jian Li, Ping Tan · 2026

Read Gaussian Belief Propagation Network for Depth Completion by Jie Tang, Pingping Xie & Jian Li, Ping Tan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21291.txt.

Introduction

Jie Tang, Pingping Xie, Jian Li, and Ping Tan work on filling in depth maps for scenes. A camera image plus a few depth readings must be turned into a full map of distances. Deep learning leads the field, but sparse and uneven depth inputs are hard for networks to use well.

The authors build a network based on belief propagation with Gaussian messages. The design spreads the few known depths across the image in a principled way. It aims to hold up even when readings are very sparse. Better depth maps support robots, self-driving cars, and tools that blend digital content with real rooms.

Catalog record

CREATORJie Tang, Pingping Xie & Jian Li, Ping Tan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21291

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gaussian Belief Propagation Network for Depth Completion Jie Tang, Pingping Xie & Jian Li, Ping Tan Abstract Depth completion aims to predict a dense depth map from a color image with sparse depth measurements. Although deep learning methods have achieved state-of-the-art (SOTA), effectively handling the sparse and irregular nature of input depth data in deep networks remains a significant challenge, often limiting performance, especially under high sparsity. To overcome this limitation, we introduce the Gaussian Belief Propagation Network (GBPN), a novel hybrid framework synergistically integrating deep learning with probabilistic graphical models for end-to-end depth completion. Specifically, a scene-specific Markov Random Field (MRF) is dynamically constructed by the Graphical Model Construction Network (GMCN), and then inferred via Gaussian Belief Propagation (GBP) to yield the dense depth distribution. Crucially, the GMCN learns to construct not only the data-dependent potentials of MRF but also its structure by predicting adaptive non-local edges, enabling the capture of complex, long-range spatial dependencies. Furthermore, we enhance GBP with a serial & parallel message passing scheme, designed for effective information propagation, particularly from sparse measurements. Extensive experiments demonstrate that GBPN achieves SOTA performance on the NYUv2 and KITTI benchmarks. Evaluations across varying sparsity levels, sparsity patterns, and datasets highlight GBPN’s superior performance, notable robustness, and generalizable capability. 1 Introduction Dense depth estimation is critical for various computer vision and robotics tasks. While dedicated sensors or methods provide sparse, irregular depth measurements ( 37 ) , acquiring dense depth directly is often challenging or costly. Depth completion (DC) bridges this gap through inferring a dense depth map from sparse depth guided by a synchronized color image. Sparse depth provides essential scale and absolute constraints, while the high-resolution color image offers rich structural and semantic information necessary for propagating depth and filling missing regions. Consequently, DC is a vital technique attracting increasing interest for enhancing downstream applications like 3D object detection ( 49 ) , novel view synthesis ( 34 ) , and robotic manipulation ( 23 ) . Traditional DC methods mostly relied on hand-crafted pipelines ( 21 ) or fixed graphical models like Markov Random Fields (MRF) ( 10 ) . Although offering some robustness, their rigid, pre-defined priors struggled to capture complex geometry and fine details. Recently, learning-based methods, primarily deep neural networks, have achieved impressive accuracy improvements by directly regressing dense depth. However, effectively processing sparse, irregular input within standard deep architectures remains a significant challenge ( 43 ; 18 ; 41 ) , often degrading performance and robustness, particularly under high sparsity. In this paper, we introduce the Gaussian Belief Propagation Network (GBPN), a novel hybrid framework that synergistically combines deep learning’s representational power with the structured inference of probabilistic graphical models. Unlike methods that directly regress depth or refine depth with limited propagation range, GBPN trains a deep network (the Graphical Model Construction Network, GMCN) to dynamically construct a global scene-specific MRF over dense depth variables. Dense depth is then efficiently inferred via Gaussian Belief Propagation (GBP) on the learned MRF. This formulation offers several key advantages. Firstly, learning to construct a scene-specific MRF overcomes the limitations of fixed, hand-crafted models, enabling adaptation to diverse geometries. Secondly, sparse depth measurements are naturally integrated as principled data terms within the globally consistent MRF framework, inherently addressing input sparsity and irregularity by propagating depth across the entire image. Finally, GBPN yields a depth distribution, providing valuable confidence estimates for risk-aware downstream tasks, such as planning ( 2 ) . To realize this, our GMCN infers not only potentials of the MRF but also its structure by predicting non-local edges, allowing the model to adaptively capture complex, long-range spatial dependencies guided by image content. We also propose a novel serial & parallel message passing scheme for GBP to enhance information flow, particularly from sparse measurements to distant unmeasured pixels. The entire GBPN is trained end-to-end using a probability-based loss function leveraging the estimated mean and precision from GBP, promoting the learning of reliable depth predictions along with an estimate of their confidence. We validate the efficacy of GBPN through extensive experiments on two leading DC benchmarks: NYUv2 for indoor scenes and KITTI for outdoor scenes. Our method achieves state-of-the-art performance on these datasets at the time of submission. Comprehensive ablation studies demonstrate the effectiveness of each component within GBPN, including the dynamic MRF construction, propagation scheme, and probability-based loss function, etc . Furthermore, evaluations across varying sparsity levels, sparsity patterns, and datasets highlight GBPN’s superior performance, notable robustness, and generalizable capability. Code and trained models will be available at omitted for blind review . 2 Related Work Research on depth completion (DC) has undergone a significant evolution, transitioning from traditional hand-crafted techniques to data-driven learning methods. Notably, concepts and principles from traditional approaches have significantly influenced the design of recent learning-based methods. This section briefly reviews these two lines of work and the combination of their respective strengths. Traditional Hand-crafted Techniques. Early DC approaches largely relied on explicitly defined priors and assumptions about scene geometry and texture, often borrowing techniques from traditional image processing and inpainting. These methods constructed hand-crafted pipelines to infer dense depth from sparse measurements. For example, 20 proposed joint bilateral filtering guided by color information to upsample low-resolution depth maps. 21 employed a sequence of classical image processing operators, such as dilation and hole filling, for depth map densification. 54 leveraged local surface smoothness assumptions to estimate depth by computing surface normals in a spherical coordinate system. 15 reconstructed dense disparity from sparse measurements by minimizing an energy function formulated by Compressive Sensing. 10 modeled depth completion as a multi-resolution Markov Random Field (MRF) with simple smoothness potentials, and then solved using conjugate gradient optimization. 3 developed a global optimization approach to reconstruct dense depth modeled by MRF. While demonstrating effectiveness in certain scenarios, these hand-crafted methods often struggle to capture complex geometric structures and fine-grained details. Data-driven Learning Methods. In contrast, learning-based methods typically formulate DC as an end-to-end regression task, taking color images and sparse depth maps as input and directly predicting the target dense depth map. A primary focus in this line of work has been on effectively handling the sparse input data and fusing information from different modalities (color and depth). For explicitly processing sparse data, pioneering work by 43 introduced sparsity-invariant convolutions, designed to handle missing data by maintaining validity masks. 18 integrated the sparsity-invariant convolutions into an encoder-decoder architecture. 11 extended this concept by propagating a continuous confidence measure. However, general-purpose network architectures employing standard convolutional layers, often combined with sophisticated multi-modal fusion strategies, have frequently demonstrated strong performance. For instance, 42 proposed to learn content-dependent and spatially-variant kernels to guide the fusion of color and depth features. 52 leveraged Transformer architectures to capture long-range dependencies for feature extraction in depth completion. 4 used 2D convolutions for image features and continuous convolutions for 3D point cloud features, followed by fusion. While achieving impressive results and learning complex mappings from data, purely data-driven methods often exhibit limited generalization performance, particularly when faced with data of different sparsity levels. Combination of Models and Learning. More recently, a significant trend has emerged towards approaches that combine the strengths of both traditional modeling and data-driven learning. These methods aim to leverage

Read the complete text